Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see the issue again.
Thanks, Manish On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <[email protected]> wrote: > Manish > > When it is in the stuck state can you please run > bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that > would aid us in narrowing in on a possible cause. > > Thanks > Joe > > On Thu, Dec 1, 2016 at 8:44 AM, Manish G <[email protected]> wrote: > > Hi Joe, > > > > Here is what I can see in the App Log: > > > > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] > > testing.nifi.processor.hdfs.testingPutHDFS "" > > org.apache.hadoop.fs.azure.AzureException: java.util. > NoSuchElementException: > > An error occurred while enumerating the result, check the original > exception > > for details. > > at > > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata( > AzureNativeFileSystemStore.java:1930) > > ~[hadoop-azure-2.7.2.jar:na] > > at > > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus( > NativeAzureFileSystem.java:1592) > > ~[hadoop-azure-2.7.2.jar:na] > > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) > > ~[hadoop-common-2.7.2.jar:na] > > at > > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger( > testingPutHDFS.java:260) > > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] > > at > > org.apache.nifi.processor.AbstractProcessor.onTrigger( > AbstractProcessor.java:27) > > [nifi-api-0.7.0.jar:0.7.0] > > at > > org.apache.nifi.controller.StandardProcessorNode.onTrigger( > StandardProcessorNode.java:1054) > > [nifi-framework-core-0.7.0.jar:0.7.0] > > at > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call( > ContinuallyRunProcessorTask.java:136) > > [nifi-framework-core-0.7.0.jar:0.7.0] > > at > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call( > ContinuallyRunProcessorTask.java:47) > > [nifi-framework-core-0.7.0.jar:0.7.0] > > at > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run( > TimerDrivenSchedulingAgent.java:127) > > [nifi-framework-core-0.7.0.jar:0.7.0] > > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > > [na:1.8.0_101] > > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > > [na:1.8.0_101] > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$301(Unknown > > Source) [na:1.8.0_101] > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(Unknown > > Source) [na:1.8.0_101] > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > [na:1.8.0_101] > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > [na:1.8.0_101] > > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] > > Caused by: java.util.NoSuchElementException: An error occurred while > > enumerating the result, check the original exception for details. > > at > > com.microsoft.azure.storage.core.LazySegmentedIterator. > hasNext(LazySegmentedIterator.java:113) > > ~[azure-storage-2.0.0.jar:na] > > at > > org.apache.hadoop.fs.azure.StorageInterfaceImpl$ > WrappingIterator.hasNext(StorageInterfaceImpl.java:128) > > ~[hadoop-azure-2.7.2.jar:na] > > at > > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata( > AzureNativeFileSystemStore.java:1909) > > ~[hadoop-azure-2.7.2.jar:na] > > ... 15 common frames omitted > > Caused by: com.microsoft.azure.storage.StorageException: Forbidden > > at > > com.microsoft.azure.storage.StorageException.translateFromHttpStatus( > StorageException.java:202) > > ~[azure-storage-2.0.0.jar:na] > > at > > com.microsoft.azure.storage.StorageException.translateException( > StorageException.java:172) > > ~[azure-storage-2.0.0.jar:na] > > at > > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry( > ExecutionEngine.java:273) > > ~[azure-storage-2.0.0.jar:na] > > at > > com.microsoft.azure.storage.core.LazySegmentedIterator. > hasNext(LazySegmentedIterator.java:109) > > ~[azure-storage-2.0.0.jar:na] > > ... 17 common frames omitted > > Caused by: java.lang.NullPointerException: null > > at > > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry( > ExecutionEngine.java:181) > > ~[azure-storage-2.0.0.jar:na] > > ... 18 common frames omitted > > > > Regards, > > Manish > > > > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <[email protected]> wrote: > >> > >> Manish > >> > >> Please produce and share the thread dump I mentioned. > >> > >> Thanks > >> Joe > >> > >> On Dec 1, 2016 7:23 AM, "Manish G" <[email protected]> wrote: > >>> > >>> Hi, > >>> > >>> I don't know why, but this is happening now more frequently. Where > should > >>> I look into to find the root cause? > >>> > >>> Thanks, > >>> Manish > >>> > >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <[email protected]> > wrote: > >>>> > >>>> Hi Joe, > >>>> > >>>> Thanks for the quick reply. Yes, the processor keeps running on a > single > >>>> thread (even after stopping). And the number remains there even after > >>>> stopping. > >>>> Today, it happened on my customized putHDFS processor. Only thing > >>>> different in this processor is - I have added an additional attribute > that > >>>> tells if the processor created the directory while loading the file > on HDFS. > >>>> I don't think this should be the issue though. > >>>> > >>>> Regards, > >>>> Manish > >>>> > >>>> > >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <[email protected]> wrote: > >>>>> > >>>>> Manish > >>>>> > >>>>> When it is stuck do you see a number in the top right corner of the > >>>>> processor? When you stop it does the number remain? That number is > >>>>> telling you how many threads are still executing. Which processor > are > >>>>> we talking about? When it is in the stuck state can you please run > >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that > >>>>> would aid us in narrowing in on a possible cause. > >>>>> > >>>>> Thanks > >>>>> Joe > >>>>> > >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <[email protected]> > >>>>> wrote: > >>>>> > > >>>>> > Hi, > >>>>> > > >>>>> > I have noticed that sometime a flow file gets stuck on a processor > >>>>> > for a > >>>>> > very long time for no reason and then I can not even stop the > >>>>> > processor to > >>>>> > look at the flow flow file from queue. If I click on stop, then > >>>>> > processor > >>>>> > goes into a state where I cannot start/stop the processor. > >>>>> > > >>>>> > On restarting the NiFi, the file gets processed successfully and > >>>>> > routed to > >>>>> > success queue. I checked in App log, but everything seems to be > >>>>> > normal for > >>>>> > the flow file. I don't see anything mysterious in provenance too > >>>>> > (except > >>>>> > that queue time is in hours). > >>>>> > > >>>>> > Has anyone else faced a similar issue? What else should I check to > >>>>> > identify > >>>>> > the root cause for this? > >>>>> > > >>>>> > Thanks, > >>>>> > Manish > >>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> > >>>> > >>>> With Warm Regards, > >>>> Manish > >>> > >>> > >>> > >>> > >>> -- > >>> > >>> > >>> With Warm Regards, > >>> Manish > > > > > > > > > > -- > > > > > > With Warm Regards, > > Manish > -- *With Warm Regards,* *Manish*
