Re: Help needed in debugging why one of the nodes in a cluster is not making any progress ..

[Bryan Bende]

Hello,

I think the behavior you saw with the queue going to 0 is expected
behavior... when you are looking at the UI it is showing the aggregated
view of all the nodes in the cluster, so if one node has flow files in the
queue and that nodes goes down while you are in the UI in another node,
that number of flow files would no longer be visible in the UI.

How many flow files are in the queue before ExecuteStreamCommand when you
see it not making any progress?

There was a bug in 1.0 where if the # of flow files in a queue was evenly
splittable by the swap size, then those flow files would be swapped out and
never swapped back in and would be sitting there.

The JIRA was: https://issues.apache.org/jira/browse/NIFI-2754

Additionally you could try upgrading to the recently released 1.1 release
to see if the same behavior occurs.

-Bryan


On Thu, Dec 1, 2016 at 3:39 PM, A.B. Srinivasan 
wrote:

> Folks,
>
> I have a NiFi 1.0 deployed in a non-secure cluster across 3 nodes.
>
> I have a flow pipeline that reads from a Kafka topic using ConsumeKafka
> and kicks off an ExecuteStreamCommand mediated job based on attributes
> included in the notification message.
>
> What I observe is that jobs are  being kicked off and they complete
> successfully on 2 of the nodes. The 3rd node however never seems to make
> progress on any of the jobs scheduled on it.
> I do see the node receiving the notification messages (based on PutRiemann
> events posted when message is received by ConsumeKafka) but thereafter
> there is no progress at all. The consequence is that the queue in front of
> the ExecuteStreamCommand processor keeps growing whenever a job is
> scheduled on the 'stuck' node.
>
> I don't see anything obvious to me in the nifi-app logs on any of the
> nodes that helps me get insight into what is afoot. I figured that some
> state is out-of-sync on the stuck node and decided to restart it. When that
> node went down, the queue in front of the ExecuteStreamCommand immediately
> went to 0 (I happened to be watching using the UI on one of the other
> nodes). When that node came back up, the queue is restored to the value it
> had prior to the restart.
>
> I am looking for debugging hints / ideas to help get insight into what is
> really going on.
>
> Thanks,
> A.B.
>

Help needed in debugging why one of the nodes in a cluster is not making any progress ..

[A.B. Srinivasan]

Folks,

I have a NiFi 1.0 deployed in a non-secure cluster across 3 nodes.

I have a flow pipeline that reads from a Kafka topic using ConsumeKafka and
kicks off an ExecuteStreamCommand mediated job based on attributes included
in the notification message.

What I observe is that jobs are  being kicked off and they complete
successfully on 2 of the nodes. The 3rd node however never seems to make
progress on any of the jobs scheduled on it.
I do see the node receiving the notification messages (based on PutRiemann
events posted when message is received by ConsumeKafka) but thereafter
there is no progress at all. The consequence is that the queue in front of
the ExecuteStreamCommand processor keeps growing whenever a job is
scheduled on the 'stuck' node.

I don't see anything obvious to me in the nifi-app logs on any of the nodes
that helps me get insight into what is afoot. I figured that some state is
out-of-sync on the stuck node and decided to restart it. When that node
went down, the queue in front of the ExecuteStreamCommand immediately went
to 0 (I happened to be watching using the UI on one of the other nodes).
When that node came back up, the queue is restored to the value it had
prior to the restart.

I am looking for debugging hints / ideas to help get insight into what is
really going on.

Thanks,
A.B.

Re: Flow File Stuck for no reason

[Bryan Bende]

Manish,

Ok what I am saying is that NIFI-1922 was simply trying to add one
dependency into the HDFS processors to make the Azure file system
implementations available:


 org.apache.hadoop
 hadoop-azure
 2.7.2+


In NiFi 1.1 this is no longer necessary because the HDFS processors have a
property where you can point to additional JARS.

I created a directory, lets say /opt/wasb/jars where I put:

azure-storage-2.0.0.jar
commons-codec-1.6.jar
commons-lang3-3.3.2.jar
commons-logging-1.1.1.jar
guava-11.0.2.jar
hadoop-azure-2.7.3.jar
httpclient-4.2.5.jar
httpcore-4.2.4.jar
jackson-core-2.2.3.jar
jsr305-1.3.9.jar
slf4j-api-1.7.5.jar

Then using NiFi 1.1 I created a PutHDFS processor and set the "Additional
Classpath Resources" property to point "/opt/wasb/jars", thus adding all
those JARs to the classpath.

I was able to talk to a WASB filesystem using this approach, as well as an
ADLS filesystem (required a different set of classpath resources).

It might be better trying to run NiFi 1.1 and using the above approach to
see if the problem persists. Right now it is really hard to say since you
are running a custom processor.

Thanks,

Bryan


On Thu, Dec 1, 2016 at 12:35 PM, Manish G  wrote:

> Bryan,
>
> It's for uploading data on HDInsight cluster and as per the patch in
> Nifi-1922.
>
> Regards,
> Manish
>
>
> On Thu, Dec 1, 2016 at 12:16 PM, Bryan Bende  wrote:
>
>> Manish,
>>
>> Was the reason for your custom version of PutHDFS just to include the
>> Azure libraries? or was there other functionality you needed to introduce?
>>
>> The reason I'm asking is because in the Apache NiFi 1.1 release (just
>> released) we added an "Additional Classpath Resources" property to all HDFS
>> processors, and I have been able to use that to connect to an Azure WASB
>> filesystem.
>>
>> I can provide the list of JARs I added to the classpath to make it work
>> if that would help.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Thu, Dec 1, 2016 at 12:11 PM, Manish G  wrote:
>>
>>> Hi Joe,
>>>
>>> I am not using anything from Nifi 1.0 and its all 0.7. What indicates
>>> the use of 1.0?
>>>
>>> Regards,
>>> Manish
>>>
>>> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt  wrote:
>>>
 Manish

 It also appears from the log output you provided so far that this is a
 combination of nifi 0.7 and 1.0 parts.  We do not recommend doing this
 and it is not likely to work.

 Please try the flow from a single release line.  Apache NiFi 1.1.0 is
 available for use now.

 Thanks
 Joe

 On Thu, Dec 1, 2016 at 9:02 AM, Manish G 
 wrote:
 > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I
 see
 > the issue again.
 >
 > Thanks,
 > Manish
 >
 > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt  wrote:
 >>
 >> Manish
 >>
 >> When it is in the stuck state can you please run
 >> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
 >> would aid us in narrowing in on a possible cause.
 >>
 >> Thanks
 >> Joe
 >>
 >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G 
 wrote:
 >> > Hi Joe,
 >> >
 >> > Here is what I can see in the App Log:
 >> >
 >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
 >> > testing.nifi.processor.hdfs.testingPutHDFS ""
 >> > org.apache.hadoop.fs.azure.AzureException:
 >> > java.util.NoSuchElementException:
 >> > An error occurred while enumerating the result, check the original
 >> > exception
 >> > for details.
 >> > at
 >> >
 >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
 veMetadata(AzureNativeFileSystemStore.java:1930)
 >> > ~[hadoop-azure-2.7.2.jar:na]
 >> > at
 >> >
 >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat
 us(NativeAzureFileSystem.java:1592)
 >> > ~[hadoop-azure-2.7.2.jar:na]
 >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
 >> > ~[hadoop-common-2.7.2.jar:na]
 >> > at
 >> >
 >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing
 PutHDFS.java:260)
 >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
 >> > at
 >> >
 >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
 ctProcessor.java:27)
 >> > [nifi-api-0.7.0.jar:0.7.0]
 >> > at
 >> >
 >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
 tandardProcessorNode.java:1054)
 >> > [nifi-framework-core-0.7.0.jar:0.7.0]
 >> > at
 >> >
 >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
 .call(ContinuallyRunProcessorTask.java:136)
 >> > [nifi-framework-core-0.7.0.jar:0.7.0]
 >> > at
 >> >
 >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
 

Re: Flow File Stuck for no reason

[Manish G]

Bryan,

It's for uploading data on HDInsight cluster and as per the patch in
Nifi-1922.

Regards,
Manish


On Thu, Dec 1, 2016 at 12:16 PM, Bryan Bende  wrote:

> Manish,
>
> Was the reason for your custom version of PutHDFS just to include the
> Azure libraries? or was there other functionality you needed to introduce?
>
> The reason I'm asking is because in the Apache NiFi 1.1 release (just
> released) we added an "Additional Classpath Resources" property to all HDFS
> processors, and I have been able to use that to connect to an Azure WASB
> filesystem.
>
> I can provide the list of JARs I added to the classpath to make it work if
> that would help.
>
> Thanks,
>
> Bryan
>
>
> On Thu, Dec 1, 2016 at 12:11 PM, Manish G  wrote:
>
>> Hi Joe,
>>
>> I am not using anything from Nifi 1.0 and its all 0.7. What indicates the
>> use of 1.0?
>>
>> Regards,
>> Manish
>>
>> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt  wrote:
>>
>>> Manish
>>>
>>> It also appears from the log output you provided so far that this is a
>>> combination of nifi 0.7 and 1.0 parts.  We do not recommend doing this
>>> and it is not likely to work.
>>>
>>> Please try the flow from a single release line.  Apache NiFi 1.1.0 is
>>> available for use now.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G  wrote:
>>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I
>>> see
>>> > the issue again.
>>> >
>>> > Thanks,
>>> > Manish
>>> >
>>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt  wrote:
>>> >>
>>> >> Manish
>>> >>
>>> >> When it is in the stuck state can you please run
>>> >> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>>> >> would aid us in narrowing in on a possible cause.
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G 
>>> wrote:
>>> >> > Hi Joe,
>>> >> >
>>> >> > Here is what I can see in the App Log:
>>> >> >
>>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS ""
>>> >> > org.apache.hadoop.fs.azure.AzureException:
>>> >> > java.util.NoSuchElementException:
>>> >> > An error occurred while enumerating the result, check the original
>>> >> > exception
>>> >> > for details.
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>>> veMetadata(AzureNativeFileSystemStore.java:1930)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat
>>> us(NativeAzureFileSystem.java:1592)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>>> >> > ~[hadoop-common-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing
>>> PutHDFS.java:260)
>>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
>>> ctProcessor.java:27)
>>> >> > [nifi-api-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>>> tandardProcessorNode.java:1054)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>>> .call(ContinuallyRunProcessorTask.java:136)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>>> .call(ContinuallyRunProcessorTask.java:47)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>>> gent$1.run(TimerDrivenSchedulingAgent.java:127)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>>> >> > [na:1.8.0_101]
>>> >> > at
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.access$301(Unknown
>>> >> > Source) [na:1.8.0_101]
>>> >> > at
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.run(Unknown
>>> >> > Source) [na:1.8.0_101]
>>> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>> Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>> Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>>> >> > Caused by: java.util.NoSuchElementException: An error occurred
>>> while
>>> >> > enumerating the result, check the original exception for details.
>>> >> > at
>>> >> >
>>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>>> xt(LazySegmentedIterator.java:113)
>>> >> > 

Re: Flow File Stuck for no reason

[Joe Witt]

manish

my apologies now I see what you mean.  It is 0.7.0 nifi and you have a
custom nar that is 1.0.0 which is totally fine.

Bryan's response though is a good one to follow up on.

Thanks
joe

On Thu, Dec 1, 2016 at 9:16 AM, Bryan Bende  wrote:
> Manish,
>
> Was the reason for your custom version of PutHDFS just to include the Azure
> libraries? or was there other functionality you needed to introduce?
>
> The reason I'm asking is because in the Apache NiFi 1.1 release (just
> released) we added an "Additional Classpath Resources" property to all HDFS
> processors, and I have been able to use that to connect to an Azure WASB
> filesystem.
>
> I can provide the list of JARs I added to the classpath to make it work if
> that would help.
>
> Thanks,
>
> Bryan
>
>
> On Thu, Dec 1, 2016 at 12:11 PM, Manish G  wrote:
>>
>> Hi Joe,
>>
>> I am not using anything from Nifi 1.0 and its all 0.7. What indicates the
>> use of 1.0?
>>
>> Regards,
>> Manish
>>
>> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt  wrote:
>>>
>>> Manish
>>>
>>> It also appears from the log output you provided so far that this is a
>>> combination of nifi 0.7 and 1.0 parts.  We do not recommend doing this
>>> and it is not likely to work.
>>>
>>> Please try the flow from a single release line.  Apache NiFi 1.1.0 is
>>> available for use now.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G  wrote:
>>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I
>>> > see
>>> > the issue again.
>>> >
>>> > Thanks,
>>> > Manish
>>> >
>>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt  wrote:
>>> >>
>>> >> Manish
>>> >>
>>> >> When it is in the stuck state can you please run
>>> >> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>>> >> would aid us in narrowing in on a possible cause.
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G 
>>> >> wrote:
>>> >> > Hi Joe,
>>> >> >
>>> >> > Here is what I can see in the App Log:
>>> >> >
>>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS ""
>>> >> > org.apache.hadoop.fs.azure.AzureException:
>>> >> > java.util.NoSuchElementException:
>>> >> > An error occurred while enumerating the result, check the original
>>> >> > exception
>>> >> > for details.
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
>>> >> > ~[hadoop-azure-2.7.2.jar:na]
>>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>>> >> > ~[hadoop-common-2.7.2.jar:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
>>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> >> > [nifi-api-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
>>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> >> > Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>>> >> > [na:1.8.0_101]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>>> >> > Source) [na:1.8.0_101]
>>> >> > at
>>> >> >
>>> >> >
>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> >> > Source) [na:1.8.0_101]
>>> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>> >> > Source)
>>> >> > [na:1.8.0_101]
>>> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>>> >> > Caused by: java.util.NoSuchElementException: An error occurred while
>>> >> > enumerating the result, check the original exception for details.
>>> >> > at
>>> 

Re: Flow File Stuck for no reason

[Bryan Bende]

Manish,

Was the reason for your custom version of PutHDFS just to include the Azure
libraries? or was there other functionality you needed to introduce?

The reason I'm asking is because in the Apache NiFi 1.1 release (just
released) we added an "Additional Classpath Resources" property to all HDFS
processors, and I have been able to use that to connect to an Azure WASB
filesystem.

I can provide the list of JARs I added to the classpath to make it work if
that would help.

Thanks,

Bryan


On Thu, Dec 1, 2016 at 12:11 PM, Manish G  wrote:

> Hi Joe,
>
> I am not using anything from Nifi 1.0 and its all 0.7. What indicates the
> use of 1.0?
>
> Regards,
> Manish
>
> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt  wrote:
>
>> Manish
>>
>> It also appears from the log output you provided so far that this is a
>> combination of nifi 0.7 and 1.0 parts.  We do not recommend doing this
>> and it is not likely to work.
>>
>> Please try the flow from a single release line.  Apache NiFi 1.1.0 is
>> available for use now.
>>
>> Thanks
>> Joe
>>
>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G  wrote:
>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I
>> see
>> > the issue again.
>> >
>> > Thanks,
>> > Manish
>> >
>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt  wrote:
>> >>
>> >> Manish
>> >>
>> >> When it is in the stuck state can you please run
>> >> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>> >> would aid us in narrowing in on a possible cause.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G 
>> wrote:
>> >> > Hi Joe,
>> >> >
>> >> > Here is what I can see in the App Log:
>> >> >
>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>> >> > testing.nifi.processor.hdfs.testingPutHDFS ""
>> >> > org.apache.hadoop.fs.azure.AzureException:
>> >> > java.util.NoSuchElementException:
>> >> > An error occurred while enumerating the result, check the original
>> >> > exception
>> >> > for details.
>> >> > at
>> >> >
>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>> veMetadata(AzureNativeFileSystemStore.java:1930)
>> >> > ~[hadoop-azure-2.7.2.jar:na]
>> >> > at
>> >> >
>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat
>> us(NativeAzureFileSystem.java:1592)
>> >> > ~[hadoop-azure-2.7.2.jar:na]
>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>> >> > ~[hadoop-common-2.7.2.jar:na]
>> >> > at
>> >> >
>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing
>> PutHDFS.java:260)
>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>> >> > at
>> >> >
>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
>> ctProcessor.java:27)
>> >> > [nifi-api-0.7.0.jar:0.7.0]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>> tandardProcessorNode.java:1054)
>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:136)
>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:47)
>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>> gent$1.run(TimerDrivenSchedulingAgent.java:127)
>> >> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source)
>> >> > [na:1.8.0_101]
>> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>> >> > [na:1.8.0_101]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(Unknown
>> >> > Source) [na:1.8.0_101]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(Unknown
>> >> > Source) [na:1.8.0_101]
>> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> >> > [na:1.8.0_101]
>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> Source)
>> >> > [na:1.8.0_101]
>> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>> >> > Caused by: java.util.NoSuchElementException: An error occurred while
>> >> > enumerating the result, check the original exception for details.
>> >> > at
>> >> >
>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe
>> xt(LazySegmentedIterator.java:113)
>> >> > ~[azure-storage-2.0.0.jar:na]
>> >> > at
>> >> >
>> >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIter
>> ator.hasNext(StorageInterfaceImpl.java:128)
>> >> > ~[hadoop-azure-2.7.2.jar:na]
>> >> > at
>> >> >
>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie
>> veMetadata(AzureNativeFileSystemStore.java:1909)
>> >> > 

Re: Flow File Stuck for no reason

[Joe Witt]

Manish

It also appears from the log output you provided so far that this is a
combination of nifi 0.7 and 1.0 parts.  We do not recommend doing this
and it is not likely to work.

Please try the flow from a single release line.  Apache NiFi 1.1.0 is
available for use now.

Thanks
Joe

On Thu, Dec 1, 2016 at 9:02 AM, Manish G  wrote:
> Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see
> the issue again.
>
> Thanks,
> Manish
>
> On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt  wrote:
>>
>> Manish
>>
>> When it is in the stuck state can you please run
>> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>> would aid us in narrowing in on a possible cause.
>>
>> Thanks
>> Joe
>>
>> On Thu, Dec 1, 2016 at 8:44 AM, Manish G  wrote:
>> > Hi Joe,
>> >
>> > Here is what I can see in the App Log:
>> >
>> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>> > testing.nifi.processor.hdfs.testingPutHDFS ""
>> > org.apache.hadoop.fs.azure.AzureException:
>> > java.util.NoSuchElementException:
>> > An error occurred while enumerating the result, check the original
>> > exception
>> > for details.
>> > at
>> >
>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at
>> >
>> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>> > ~[hadoop-common-2.7.2.jar:na]
>> > at
>> >
>> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
>> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>> > at
>> >
>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> > [nifi-api-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at
>> >
>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>> > [na:1.8.0_101]
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>> > Source) [na:1.8.0_101]
>> > at
>> >
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> > Source) [na:1.8.0_101]
>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> > [na:1.8.0_101]
>> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>> > Caused by: java.util.NoSuchElementException: An error occurred while
>> > enumerating the result, check the original exception for details.
>> > at
>> >
>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > at
>> >
>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
>> > ~[hadoop-azure-2.7.2.jar:na]
>> > ... 15 common frames omitted
>> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>> > at
>> >
>> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
>> > ~[azure-storage-2.0.0.jar:na]
>> > at
>> >
>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
>> > ~[azure-storage-2.0.0.jar:na]
>> > ... 17 common frames omitted
>> > Caused by: java.lang.NullPointerException: null
>> > at
>> >
>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
>> > ~[azure-storage-2.0.0.jar:na]
>> > ... 18 common frames omitted
>> >
>> > Regards,
>> > Manish
>> >
>> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt  wrote:
>> >>
>> >> Manish
>> >>
>> >> Please produce and share the 

Re: Flow File Stuck for no reason

[Manish G]

Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see
the issue again.

Thanks,
Manish

On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt  wrote:

> Manish
>
> When it is in the stuck state can you please run
> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
> would aid us in narrowing in on a possible cause.
>
> Thanks
> Joe
>
> On Thu, Dec 1, 2016 at 8:44 AM, Manish G  wrote:
> > Hi Joe,
> >
> > Here is what I can see in the App Log:
> >
> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
> > testing.nifi.processor.hdfs.testingPutHDFS ""
> > org.apache.hadoop.fs.azure.AzureException: java.util.
> NoSuchElementException:
> > An error occurred while enumerating the result, check the original
> exception
> > for details.
> > at
> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(
> AzureNativeFileSystemStore.java:1930)
> > ~[hadoop-azure-2.7.2.jar:na]
> > at
> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(
> NativeAzureFileSystem.java:1592)
> > ~[hadoop-azure-2.7.2.jar:na]
> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> > ~[hadoop-common-2.7.2.jar:na]
> > at
> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(
> testingPutHDFS.java:260)
> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
> > at
> > org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:27)
> > [nifi-api-0.7.0.jar:0.7.0]
> > at
> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1054)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:136)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:47)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > at
> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(
> TimerDrivenSchedulingAgent.java:127)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> > [na:1.8.0_101]
> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> > [na:1.8.0_101]
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(Unknown
> > Source) [na:1.8.0_101]
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(Unknown
> > Source) [na:1.8.0_101]
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > [na:1.8.0_101]
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > [na:1.8.0_101]
> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
> > Caused by: java.util.NoSuchElementException: An error occurred while
> > enumerating the result, check the original exception for details.
> > at
> > com.microsoft.azure.storage.core.LazySegmentedIterator.
> hasNext(LazySegmentedIterator.java:113)
> > ~[azure-storage-2.0.0.jar:na]
> > at
> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$
> WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
> > ~[hadoop-azure-2.7.2.jar:na]
> > at
> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(
> AzureNativeFileSystemStore.java:1909)
> > ~[hadoop-azure-2.7.2.jar:na]
> > ... 15 common frames omitted
> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
> > at
> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(
> StorageException.java:202)
> > ~[azure-storage-2.0.0.jar:na]
> > at
> > com.microsoft.azure.storage.StorageException.translateException(
> StorageException.java:172)
> > ~[azure-storage-2.0.0.jar:na]
> > at
> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(
> ExecutionEngine.java:273)
> > ~[azure-storage-2.0.0.jar:na]
> > at
> > com.microsoft.azure.storage.core.LazySegmentedIterator.
> hasNext(LazySegmentedIterator.java:109)
> > ~[azure-storage-2.0.0.jar:na]
> > ... 17 common frames omitted
> > Caused by: java.lang.NullPointerException: null
> > at
> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(
> ExecutionEngine.java:181)
> > ~[azure-storage-2.0.0.jar:na]
> > ... 18 common frames omitted
> >
> > Regards,
> > Manish
> >
> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt  wrote:
> >>
> >> Manish
> >>
> >> Please produce and share the thread dump I mentioned.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Dec 1, 2016 7:23 AM, "Manish G"  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I don't know why, but this is happening now more frequently. Where
> should
> >>> I look into to find the root cause?
> >>>
> >>> Thanks,
> >>> Manish
> >>>
> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G 
> wrote:
> 
>  Hi Joe,
> 
>  Thanks for the quick reply. Yes, the processor keeps running on a
> single
>  

Re: Flow File Stuck for no reason

[Joe Witt]

Manish

When it is in the stuck state can you please run
bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
would aid us in narrowing in on a possible cause.

Thanks
Joe

On Thu, Dec 1, 2016 at 8:44 AM, Manish G  wrote:
> Hi Joe,
>
> Here is what I can see in the App Log:
>
> 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
> testing.nifi.processor.hdfs.testingPutHDFS ""
> org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException:
> An error occurred while enumerating the result, check the original exception
> for details.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
> ~[hadoop-azure-2.7.2.jar:na]
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
> ~[hadoop-azure-2.7.2.jar:na]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> ~[hadoop-common-2.7.2.jar:na]
> at
> testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
> ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source) [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source) [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.8.0_101]
> at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
> Caused by: java.util.NoSuchElementException: An error occurred while
> enumerating the result, check the original exception for details.
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
> ~[azure-storage-2.0.0.jar:na]
> at
> org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
> ~[hadoop-azure-2.7.2.jar:na]
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
> ~[hadoop-azure-2.7.2.jar:na]
> ... 15 common frames omitted
> Caused by: com.microsoft.azure.storage.StorageException: Forbidden
> at
> com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
> ~[azure-storage-2.0.0.jar:na]
> ... 17 common frames omitted
> Caused by: java.lang.NullPointerException: null
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
> ~[azure-storage-2.0.0.jar:na]
> ... 18 common frames omitted
>
> Regards,
> Manish
>
> On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt  wrote:
>>
>> Manish
>>
>> Please produce and share the thread dump I mentioned.
>>
>> Thanks
>> Joe
>>
>> On Dec 1, 2016 7:23 AM, "Manish G"  wrote:
>>>
>>> Hi,
>>>
>>> I don't know why, but this is happening now more frequently. Where should
>>> I look into to find the root cause?
>>>
>>> Thanks,
>>> Manish
>>>
>>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G  wrote:

 Hi Joe,

 Thanks for the quick reply. Yes, the processor keeps running on a single
 thread (even after stopping). And the number remains there even after
 stopping.
 Today, it happened on my customized putHDFS processor. Only thing
 different in this processor is - I have added an additional attribute that
 tells if the processor created the directory while loading the file on 
 HDFS.
 I don't think this should be the issue though.

 Regards,
 Manish


 On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt  

Re: Flow File Stuck for no reason

[Manish G]

Hi Joe,

Here is what I can see in the App Log:

2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
testing.nifi.processor.hdfs.testingPutHDFS ""
org.apache.hadoop.fs.azure.AzureException:
java.util.NoSuchElementException: An error occurred while enumerating the
result, check the original exception for details.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
~[hadoop-azure-2.7.2.jar:na]
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
~[hadoop-azure-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
~[hadoop-common-2.7.2.jar:na]
at
testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-0.7.0.jar:0.7.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
[nifi-framework-core-0.7.0.jar:0.7.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-0.7.0.jar:0.7.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-0.7.0.jar:0.7.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-0.7.0.jar:0.7.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source) [na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.8.0_101]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
Caused by: java.util.NoSuchElementException: An error occurred while
enumerating the result, check the original exception for details.
at
com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
~[azure-storage-2.0.0.jar:na]
at
org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
~[hadoop-azure-2.7.2.jar:na]
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
~[hadoop-azure-2.7.2.jar:na]
... 15 common frames omitted
Caused by: com.microsoft.azure.storage.StorageException: Forbidden
at
com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
~[azure-storage-2.0.0.jar:na]
at
com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
~[azure-storage-2.0.0.jar:na]
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
~[azure-storage-2.0.0.jar:na]
at
com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
~[azure-storage-2.0.0.jar:na]
... 17 common frames omitted
Caused by: java.lang.NullPointerException: null
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
~[azure-storage-2.0.0.jar:na]
... 18 common frames omitted

Regards,
Manish

On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt  wrote:

> Manish
>
> Please produce and share the thread dump I mentioned.
>
> Thanks
> Joe
>
> On Dec 1, 2016 7:23 AM, "Manish G"  wrote:
>
>> Hi,
>>
>> I don't know why, but this is happening now more frequently. Where should
>> I look into to find the root cause?
>>
>> Thanks,
>> Manish
>>
>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G  wrote:
>>
>>> Hi Joe,
>>>
>>> Thanks for the quick reply. Yes, the processor keeps running on a single
>>> thread (even after stopping). And the number remains there even after
>>> stopping.
>>> Today, it happened on my customized putHDFS processor. Only thing
>>> different in this processor is - I have added an additional attribute that
>>> tells if the processor created the directory while loading the file on
>>> HDFS. I don't think this should be the issue though.
>>>
>>> Regards,
>>> Manish
>>>
>>>
>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt  wrote:
>>>
 Manish

 When it is stuck do you see a number in the top right corner of the
 processor?  When you stop it does the number remain?  That number is
 telling you how many threads are still executing.  Which processor are
 we talking about?  When it is in the stuck state can you please run
 bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
 would aid us in narrowing in on a possible cause.

 Thanks

Re: Flow File Stuck for no reason

[Joe Witt]

Manish

Please produce and share the thread dump I mentioned.

Thanks
Joe

On Dec 1, 2016 7:23 AM, "Manish G"  wrote:

> Hi,
>
> I don't know why, but this is happening now more frequently. Where should
> I look into to find the root cause?
>
> Thanks,
> Manish
>
> On Wed, Nov 30, 2016 at 9:20 PM, Manish G  wrote:
>
>> Hi Joe,
>>
>> Thanks for the quick reply. Yes, the processor keeps running on a single
>> thread (even after stopping). And the number remains there even after
>> stopping.
>> Today, it happened on my customized putHDFS processor. Only thing
>> different in this processor is - I have added an additional attribute that
>> tells if the processor created the directory while loading the file on
>> HDFS. I don't think this should be the issue though.
>>
>> Regards,
>> Manish
>>
>>
>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt  wrote:
>>
>>> Manish
>>>
>>> When it is stuck do you see a number in the top right corner of the
>>> processor?  When you stop it does the number remain?  That number is
>>> telling you how many threads are still executing.  Which processor are
>>> we talking about?  When it is in the stuck state can you please run
>>> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>>> would aid us in narrowing in on a possible cause.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G 
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have noticed that sometime a flow file gets stuck on a processor for
>>> a
>>> > very long time for no reason and then I can not even stop the
>>> processor to
>>> > look at the flow flow file from queue. If I click on stop, then
>>> processor
>>> > goes into a state where I cannot start/stop the processor.
>>> >
>>> > On restarting the NiFi, the file gets processed successfully and
>>> routed to
>>> > success queue. I checked in App log, but everything seems to be normal
>>> for
>>> > the flow file. I don't see anything mysterious in provenance too
>>> (except
>>> > that queue time is in hours).
>>> >
>>> > Has anyone else faced a similar issue? What else should I check to
>>> identify
>>> > the root cause for this?
>>> >
>>> > Thanks,
>>> > Manish
>>>
>>
>>
>>
>> --
>>
>>
>> *With Warm Regards,*
>> *Manish*
>>
>
>
>
> --
>
>
> *With Warm Regards,*
> *Manish*
>

Re: Flow File Stuck for no reason

[Manish G]

Hi,

I don't know why, but this is happening now more frequently. Where should I
look into to find the root cause?

Thanks,
Manish

On Wed, Nov 30, 2016 at 9:20 PM, Manish G  wrote:

> Hi Joe,
>
> Thanks for the quick reply. Yes, the processor keeps running on a single
> thread (even after stopping). And the number remains there even after
> stopping.
> Today, it happened on my customized putHDFS processor. Only thing
> different in this processor is - I have added an additional attribute that
> tells if the processor created the directory while loading the file on
> HDFS. I don't think this should be the issue though.
>
> Regards,
> Manish
>
>
> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt  wrote:
>
>> Manish
>>
>> When it is stuck do you see a number in the top right corner of the
>> processor?  When you stop it does the number remain?  That number is
>> telling you how many threads are still executing.  Which processor are
>> we talking about?  When it is in the stuck state can you please run
>> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>> would aid us in narrowing in on a possible cause.
>>
>> Thanks
>> Joe
>>
>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G  wrote:
>> >
>> > Hi,
>> >
>> > I have noticed that sometime a flow file gets stuck on a processor for a
>> > very long time for no reason and then I can not even stop the processor
>> to
>> > look at the flow flow file from queue. If I click on stop, then
>> processor
>> > goes into a state where I cannot start/stop the processor.
>> >
>> > On restarting the NiFi, the file gets processed successfully and routed
>> to
>> > success queue. I checked in App log, but everything seems to be normal
>> for
>> > the flow file. I don't see anything mysterious in provenance too (except
>> > that queue time is in hours).
>> >
>> > Has anyone else faced a similar issue? What else should I check to
>> identify
>> > the root cause for this?
>> >
>> > Thanks,
>> > Manish
>>
>
>
>
> --
>
>
> *With Warm Regards,*
> *Manish*
>



-- 


*With Warm Regards,*
*Manish*