Manish

When it is in the stuck state can you please run
bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
would aid us in narrowing in on a possible cause.

Thanks
Joe

On Thu, Dec 1, 2016 at 8:44 AM, Manish G <manish.gg...@gmail.com> wrote:
> Hi Joe,
>
> Here is what I can see in the App Log:
>
> 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
> testing.nifi.processor.hdfs.testingPutHDFS ""
> org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException:
> An error occurred while enumerating the result, check the original exception
> for details.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
> ~[hadoop-azure-2.7.2.jar:na]
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
> ~[hadoop-azure-2.7.2.jar:na]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> ~[hadoop-common-2.7.2.jar:na]
> at
> testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
> ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source) [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source) [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.8.0_101]
> at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
> Caused by: java.util.NoSuchElementException: An error occurred while
> enumerating the result, check the original exception for details.
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
> ~[azure-storage-2.0.0.jar:na]
> at
> org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
> ~[hadoop-azure-2.7.2.jar:na]
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
> ~[hadoop-azure-2.7.2.jar:na]
> ... 15 common frames omitted
> Caused by: com.microsoft.azure.storage.StorageException: Forbidden
> at
> com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
> ~[azure-storage-2.0.0.jar:na]
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
> ~[azure-storage-2.0.0.jar:na]
> ... 17 common frames omitted
> Caused by: java.lang.NullPointerException: null
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
> ~[azure-storage-2.0.0.jar:na]
> ... 18 common frames omitted
>
> Regards,
> Manish
>
> On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <joe.w...@gmail.com> wrote:
>>
>> Manish
>>
>> Please produce and share the thread dump I mentioned.
>>
>> Thanks
>> Joe
>>
>> On Dec 1, 2016 7:23 AM, "Manish G" <manish.gg...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I don't know why, but this is happening now more frequently. Where should
>>> I look into to find the root cause?
>>>
>>> Thanks,
>>> Manish
>>>
>>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <manish.gg...@gmail.com> wrote:
>>>>
>>>> Hi Joe,
>>>>
>>>> Thanks for the quick reply. Yes, the processor keeps running on a single
>>>> thread (even after stopping). And the number remains there even after
>>>> stopping.
>>>> Today, it happened on my customized putHDFS processor. Only thing
>>>> different in this processor is - I have added an additional attribute that
>>>> tells if the processor created the directory while loading the file on 
>>>> HDFS.
>>>> I don't think this should be the issue though.
>>>>
>>>> Regards,
>>>> Manish
>>>>
>>>>
>>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <joe.w...@gmail.com> wrote:
>>>>>
>>>>> Manish
>>>>>
>>>>> When it is stuck do you see a number in the top right corner of the
>>>>> processor?  When you stop it does the number remain?  That number is
>>>>> telling you how many threads are still executing.  Which processor are
>>>>> we talking about?  When it is in the stuck state can you please run
>>>>> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>>>>> would aid us in narrowing in on a possible cause.
>>>>>
>>>>> Thanks
>>>>> Joe
>>>>>
>>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <manish.gg...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > I have noticed that sometime a flow file gets stuck on a processor
>>>>> > for a
>>>>> > very long time for no reason and then I can not even stop the
>>>>> > processor to
>>>>> > look at the flow flow file from queue. If I click on stop, then
>>>>> > processor
>>>>> > goes into a state where I cannot start/stop the processor.
>>>>> >
>>>>> > On restarting the NiFi, the file gets processed successfully and
>>>>> > routed to
>>>>> > success queue. I checked in App log, but everything seems to be
>>>>> > normal for
>>>>> > the flow file. I don't see anything mysterious in provenance too
>>>>> > (except
>>>>> > that queue time is in hours).
>>>>> >
>>>>> > Has anyone else faced a similar issue? What else should I check to
>>>>> > identify
>>>>> > the root cause for this?
>>>>> >
>>>>> > Thanks,
>>>>> > Manish
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> With Warm Regards,
>>>> Manish
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> With Warm Regards,
>>> Manish
>
>
>
>
> --
>
>
> With Warm Regards,
> Manish

Reply via email to