Manish It also appears from the log output you provided so far that this is a combination of nifi 0.7 and 1.0 parts. We do not recommend doing this and it is not likely to work.
Please try the flow from a single release line. Apache NiFi 1.1.0 is available for use now. Thanks Joe On Thu, Dec 1, 2016 at 9:02 AM, Manish G <manish.gg...@gmail.com> wrote: > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see > the issue again. > > Thanks, > Manish > > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <joe.w...@gmail.com> wrote: >> >> Manish >> >> When it is in the stuck state can you please run >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >> would aid us in narrowing in on a possible cause. >> >> Thanks >> Joe >> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <manish.gg...@gmail.com> wrote: >> > Hi Joe, >> > >> > Here is what I can see in the App Log: >> > >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] >> > testing.nifi.processor.hdfs.testingPutHDFS "" >> > org.apache.hadoop.fs.azure.AzureException: >> > java.util.NoSuchElementException: >> > An error occurred while enumerating the result, check the original >> > exception >> > for details. >> > at >> > >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930) >> > ~[hadoop-azure-2.7.2.jar:na] >> > at >> > >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592) >> > ~[hadoop-azure-2.7.2.jar:na] >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) >> > ~[hadoop-common-2.7.2.jar:na] >> > at >> > >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260) >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] >> > at >> > >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> > [nifi-api-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) >> > [na:1.8.0_101] >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) >> > [na:1.8.0_101] >> > at >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown >> > Source) [na:1.8.0_101] >> > at >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown >> > Source) [na:1.8.0_101] >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> > [na:1.8.0_101] >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> > [na:1.8.0_101] >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] >> > Caused by: java.util.NoSuchElementException: An error occurred while >> > enumerating the result, check the original exception for details. >> > at >> > >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113) >> > ~[azure-storage-2.0.0.jar:na] >> > at >> > >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128) >> > ~[hadoop-azure-2.7.2.jar:na] >> > at >> > >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909) >> > ~[hadoop-azure-2.7.2.jar:na] >> > ... 15 common frames omitted >> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden >> > at >> > >> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202) >> > ~[azure-storage-2.0.0.jar:na] >> > at >> > >> > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172) >> > ~[azure-storage-2.0.0.jar:na] >> > at >> > >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273) >> > ~[azure-storage-2.0.0.jar:na] >> > at >> > >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109) >> > ~[azure-storage-2.0.0.jar:na] >> > ... 17 common frames omitted >> > Caused by: java.lang.NullPointerException: null >> > at >> > >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181) >> > ~[azure-storage-2.0.0.jar:na] >> > ... 18 common frames omitted >> > >> > Regards, >> > Manish >> > >> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <joe.w...@gmail.com> wrote: >> >> >> >> Manish >> >> >> >> Please produce and share the thread dump I mentioned. >> >> >> >> Thanks >> >> Joe >> >> >> >> On Dec 1, 2016 7:23 AM, "Manish G" <manish.gg...@gmail.com> wrote: >> >>> >> >>> Hi, >> >>> >> >>> I don't know why, but this is happening now more frequently. Where >> >>> should >> >>> I look into to find the root cause? >> >>> >> >>> Thanks, >> >>> Manish >> >>> >> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <manish.gg...@gmail.com> >> >>> wrote: >> >>>> >> >>>> Hi Joe, >> >>>> >> >>>> Thanks for the quick reply. Yes, the processor keeps running on a >> >>>> single >> >>>> thread (even after stopping). And the number remains there even after >> >>>> stopping. >> >>>> Today, it happened on my customized putHDFS processor. Only thing >> >>>> different in this processor is - I have added an additional attribute >> >>>> that >> >>>> tells if the processor created the directory while loading the file >> >>>> on HDFS. >> >>>> I don't think this should be the issue though. >> >>>> >> >>>> Regards, >> >>>> Manish >> >>>> >> >>>> >> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >>>>> >> >>>>> Manish >> >>>>> >> >>>>> When it is stuck do you see a number in the top right corner of the >> >>>>> processor? When you stop it does the number remain? That number is >> >>>>> telling you how many threads are still executing. Which processor >> >>>>> are >> >>>>> we talking about? When it is in the stuck state can you please run >> >>>>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >> >>>>> would aid us in narrowing in on a possible cause. >> >>>>> >> >>>>> Thanks >> >>>>> Joe >> >>>>> >> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <manish.gg...@gmail.com> >> >>>>> wrote: >> >>>>> > >> >>>>> > Hi, >> >>>>> > >> >>>>> > I have noticed that sometime a flow file gets stuck on a processor >> >>>>> > for a >> >>>>> > very long time for no reason and then I can not even stop the >> >>>>> > processor to >> >>>>> > look at the flow flow file from queue. If I click on stop, then >> >>>>> > processor >> >>>>> > goes into a state where I cannot start/stop the processor. >> >>>>> > >> >>>>> > On restarting the NiFi, the file gets processed successfully and >> >>>>> > routed to >> >>>>> > success queue. I checked in App log, but everything seems to be >> >>>>> > normal for >> >>>>> > the flow file. I don't see anything mysterious in provenance too >> >>>>> > (except >> >>>>> > that queue time is in hours). >> >>>>> > >> >>>>> > Has anyone else faced a similar issue? What else should I check to >> >>>>> > identify >> >>>>> > the root cause for this? >> >>>>> > >> >>>>> > Thanks, >> >>>>> > Manish >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> >> >>>> >> >>>> With Warm Regards, >> >>>> Manish >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> >> >>> >> >>> With Warm Regards, >> >>> Manish >> > >> > >> > >> > >> > -- >> > >> > >> > With Warm Regards, >> > Manish > > > > > -- > > > With Warm Regards, > Manish
