Hi Joe, Here is what I can see in the App Log:
2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] testing.nifi.processor.hdfs.testingPutHDFS "" org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details. at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930) ~[hadoop-azure-2.7.2.jar:na] at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592) ~[hadoop-azure-2.7.2.jar:na] at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) ~[hadoop-common-2.7.2.jar:na] at testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260) ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-0.7.0.jar:0.7.0] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) [nifi-framework-core-0.7.0.jar:0.7.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0.jar:0.7.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0.jar:0.7.0] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) [nifi-framework-core-0.7.0.jar:0.7.0] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_101] at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_101] at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] Caused by: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details. at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113) ~[azure-storage-2.0.0.jar:na] at org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128) ~[hadoop-azure-2.7.2.jar:na] at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909) ~[hadoop-azure-2.7.2.jar:na] ... 15 common frames omitted Caused by: com.microsoft.azure.storage.StorageException: Forbidden at com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202) ~[azure-storage-2.0.0.jar:na] at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172) ~[azure-storage-2.0.0.jar:na] at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273) ~[azure-storage-2.0.0.jar:na] at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109) ~[azure-storage-2.0.0.jar:na] ... 17 common frames omitted Caused by: java.lang.NullPointerException: null at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181) ~[azure-storage-2.0.0.jar:na] ... 18 common frames omitted Regards, Manish On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <joe.w...@gmail.com> wrote: > Manish > > Please produce and share the thread dump I mentioned. > > Thanks > Joe > > On Dec 1, 2016 7:23 AM, "Manish G" <manish.gg...@gmail.com> wrote: > >> Hi, >> >> I don't know why, but this is happening now more frequently. Where should >> I look into to find the root cause? >> >> Thanks, >> Manish >> >> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <manish.gg...@gmail.com> wrote: >> >>> Hi Joe, >>> >>> Thanks for the quick reply. Yes, the processor keeps running on a single >>> thread (even after stopping). And the number remains there even after >>> stopping. >>> Today, it happened on my customized putHDFS processor. Only thing >>> different in this processor is - I have added an additional attribute that >>> tells if the processor created the directory while loading the file on >>> HDFS. I don't think this should be the issue though. >>> >>> Regards, >>> Manish >>> >>> >>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <joe.w...@gmail.com> wrote: >>> >>>> Manish >>>> >>>> When it is stuck do you see a number in the top right corner of the >>>> processor? When you stop it does the number remain? That number is >>>> telling you how many threads are still executing. Which processor are >>>> we talking about? When it is in the stuck state can you please run >>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >>>> would aid us in narrowing in on a possible cause. >>>> >>>> Thanks >>>> Joe >>>> >>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <manish.gg...@gmail.com> >>>> wrote: >>>> > >>>> > Hi, >>>> > >>>> > I have noticed that sometime a flow file gets stuck on a processor >>>> for a >>>> > very long time for no reason and then I can not even stop the >>>> processor to >>>> > look at the flow flow file from queue. If I click on stop, then >>>> processor >>>> > goes into a state where I cannot start/stop the processor. >>>> > >>>> > On restarting the NiFi, the file gets processed successfully and >>>> routed to >>>> > success queue. I checked in App log, but everything seems to be >>>> normal for >>>> > the flow file. I don't see anything mysterious in provenance too >>>> (except >>>> > that queue time is in hours). >>>> > >>>> > Has anyone else faced a similar issue? What else should I check to >>>> identify >>>> > the root cause for this? >>>> > >>>> > Thanks, >>>> > Manish >>>> >>> >>> >>> >>> -- >>> >>> >>> *With Warm Regards,* >>> *Manish* >>> >> >> >> >> -- >> >> >> *With Warm Regards,* >> *Manish* >> > -- *With Warm Regards,* *Manish*
