Re: Killing 'Stuck' Processors without restarting NiFi

On my phone right now, can't provide much details, but there will be a way to terminate processors in the next version: https://issues.apache.org/jira/browse/NIFI-4895 Pierre Le mar. 3 avr. 2018 à 03:03, Joseph Niemiec a écrit : > Yes that's correct. Is there a way to

Re: Killing 'Stuck' Processors without restarting NiFi

Yes that's correct. Is there a way to identify the thread pool name of the stuck ZK process with the dump command? Then it would be possible to use the Java attach API to find and stop the thread with an agent.

Re: Killing 'Stuck' Processors without restarting NiFi

Hey Joseph, I don’t have a sure shot fix but I’m willing to bet this is the same issue we all experience using any zookeeper based system. Phoenix for example. In that the real problem is the JVM hangs up trying to communicate with zookeeper more than the actual underlying system. Is your

Re: Permission to access the API

Just to follow up. For the most part, the permission required is the same as the resource being requested. In this case, it is the /flow (view the flow) resource. Status/Stats for a component are available for anyone with permission to view the flow. 'View the component" will expose configuration

Re: Permission to access the API

Hi Mike, System Diagnostic is a completely different view in the UI when going into Summary and then System Diagnostics (bottom right of the summary view). In this case, your colleague wants to access all the information of the workflow. IIRC he'd need access to the flow + "view the component"

Permission to access the API

Colleague of mine is trying to poll this API: /nifi-api/flow/process-groups/root/status?recursive=true The instance is protected with SSL and LDAP auth. The user account I'm trying has "view system diagnostics," but it still gets a 403. Any ideas? Thanks, Mike

Re: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Mohit, I see. I think this is an issue because the Avro Writer expects that the data must be in the proper schema, or else it will throw an Exception when trying to write the data. To address this, we should update ValidateRecord to support a different Record Writer to use for valid data vs.

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Mark, Error:- ValidateRecord[id=5a9c3616-ab7c-17c1--e6c2fc5d] ValidateRecord[id=5a9c3616-ab7c-17c1--e6c2fc5d] failed to process due to org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value mohit of type class java.lang.String

Re: ConvertCSVToAvro taking a lot of time when passing single record as an input.

What is the error that you're seeing? On Apr 2, 2018, at 10:22 AM, Mohit > wrote: Hi Mark, I tried the ValidateRecord processor, it is converting the flowfile if it is valid. But If the records are not valid, it is passing

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Hi Mark, I tried the ValidateRecord processor, it is converting the flowfile if it is valid. But If the records are not valid, it is passing to the invalid relationship. Instead it keeps on throwing bulletins keeping the flowfile in the queue. Any suggestion? Mohit From: Mark

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Mark, Agree with you. I’ll try the ValidateRecord Processor, if it serve the purpose. Thanks, Mohit From: Mark Payne Sent: 02 April 2018 19:02 To: users@nifi.apache.org Subject: Re: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Re: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Mohit, You can certainly dial back that number of Concurrent Tasks. Setting that to something like 10 is a pretty big number. Setting it to a thousand means that you'll likely starve out other processors that are waiting on a thread and will generally perform a lot worse because you have 1,000

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Hi Mark, The main intention to use such flow is to track bad records. The records which doesn’t get converted should be tracked somewhere. For that purpose I’m using Split-Merge approach. Meanwhile, I’m able to improve the performance by increasing the ‘Concurrent Tasks’ to 1000. Now

Re: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Mohit, I agree that 45-50 records per second is quite slow. I'm not very familiar with the implementation of ConvertCSVToAvro, but it may well be that it must perform some sort of initialization for each FlowFile that it receives, which would explain why it's fast for a single incoming

ConvertCSVToAvro taking a lot of time when passing single record as an input.

Hi, I'm trying to capture bad records from ConvertCSVToAvro processor. For that, I'm using two SplitText processors in a row to create chunks and then each record per flow file. My flow is - ListFile -> FetchFile -> SplitText(1 records) -> SplitText(1 record) -> ConvertCSVToAvro ->