Re: Flow File Stuck for no reason
Hello This part of the stack dump is the only thing that appears out of place to me. it appears to be attempting to read from the socket or stuck there. Perhaps you can use a more recent client library for that custom HD processor? "Timer-Driven Process Thread-8" Id=78 RUNNABLE (in native code) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read1(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - waiting on java.io.BufferedInputStream@7b23ac8 at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source) at sun.net.www.http.HttpClient.parseHTTP(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) - waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) - waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911 at java.net.HttpURLConnection.getResponseCode(Unknown Source) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:124) at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109) at org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128) at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.conditionalRedoFolderRename(NativeAzureFileSystem.java:1638) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1603) at custom.nifi.processor.hdfs.PutHDFSHDCustom.onTrigger(PutHDFSHDCustom.java:247) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@3b501d78 Thanks Joe On Tue, Dec 6, 2016 at 9:05 AM, Manish G wrote: > Hi, > > The issue happened again today. I restarted the NiFi and it started working > again. Below are the rows from bootstrap.log after nifi.sh dump. > Could this be because of some other process like diagnostic monitor on Azure > storage blob? > > Thanks, > Manish > > > 2016-12-06 08:41:47,079 INFO [main] org.apache.nifi.bootstrap.RunNiFi " > "Attach Listener" Id=6 RUNNABLE > > "AzureNativeFilesystemStore-UploadBandwidthUpdater" Id=94 TIMED_WAITING on > null > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.fs.azure.metrics.BandwidthGaugeUpdater$UploadBandwidthUpdater.run(BandwidthGaugeUpdater.java:266) > at java.lang.Thread.run(Unknown Source) > > "Cleanup Archive for default" Id=54 RUNNABLE > at sun.nio.fs.WindowsNativeDispatcher.FindClose(Native Method) > at sun.nio.fs.WindowsDirectoryStream.close(Unknown Source) > at sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(Unknown Source) > at sun.nio.fs.WindowsFileSystemProvider.checkAccess(Unknown Source) > at java.nio.file.Files.exists(Unknown Source) > at > org.apache.nifi.controller.repository.FileSystemRepository.destroyExpiredArchives(FileSystemRepository.java:1307) > at > org.apache.nifi.controller.repository.FileSystemRepository.access$1800(FileSystemRepository.java:84) > at > org.apache.nifi.controller.repository.FileSystemRepository$DestroyExpiredArchiveClaims.run(FileSystemRepository.java:1546) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) > at java.util.co
Re: Flow File Stuck for no reason
Manish, Ok what I am saying is that NIFI-1922 was simply trying to add one dependency into the HDFS processors to make the Azure file system implementations available: org.apache.hadoop hadoop-azure 2.7.2+ In NiFi 1.1 this is no longer necessary because the HDFS processors have a property where you can point to additional JARS. I created a directory, lets say /opt/wasb/jars where I put: azure-storage-2.0.0.jar commons-codec-1.6.jar commons-lang3-3.3.2.jar commons-logging-1.1.1.jar guava-11.0.2.jar hadoop-azure-2.7.3.jar httpclient-4.2.5.jar httpcore-4.2.4.jar jackson-core-2.2.3.jar jsr305-1.3.9.jar slf4j-api-1.7.5.jar Then using NiFi 1.1 I created a PutHDFS processor and set the "Additional Classpath Resources" property to point "/opt/wasb/jars", thus adding all those JARs to the classpath. I was able to talk to a WASB filesystem using this approach, as well as an ADLS filesystem (required a different set of classpath resources). It might be better trying to run NiFi 1.1 and using the above approach to see if the problem persists. Right now it is really hard to say since you are running a custom processor. Thanks, Bryan On Thu, Dec 1, 2016 at 12:35 PM, Manish G wrote: > Bryan, > > It's for uploading data on HDInsight cluster and as per the patch in > Nifi-1922. > > Regards, > Manish > > > On Thu, Dec 1, 2016 at 12:16 PM, Bryan Bende wrote: > >> Manish, >> >> Was the reason for your custom version of PutHDFS just to include the >> Azure libraries? or was there other functionality you needed to introduce? >> >> The reason I'm asking is because in the Apache NiFi 1.1 release (just >> released) we added an "Additional Classpath Resources" property to all HDFS >> processors, and I have been able to use that to connect to an Azure WASB >> filesystem. >> >> I can provide the list of JARs I added to the classpath to make it work >> if that would help. >> >> Thanks, >> >> Bryan >> >> >> On Thu, Dec 1, 2016 at 12:11 PM, Manish G wrote: >> >>> Hi Joe, >>> >>> I am not using anything from Nifi 1.0 and its all 0.7. What indicates >>> the use of 1.0? >>> >>> Regards, >>> Manish >>> >>> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt wrote: >>> Manish It also appears from the log output you provided so far that this is a combination of nifi 0.7 and 1.0 parts. We do not recommend doing this and it is not likely to work. Please try the flow from a single release line. Apache NiFi 1.1.0 is available for use now. Thanks Joe On Thu, Dec 1, 2016 at 9:02 AM, Manish G wrote: > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see > the issue again. > > Thanks, > Manish > > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt wrote: >> >> Manish >> >> When it is in the stuck state can you please run >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >> would aid us in narrowing in on a possible cause. >> >> Thanks >> Joe >> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G wrote: >> > Hi Joe, >> > >> > Here is what I can see in the App Log: >> > >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] >> > testing.nifi.processor.hdfs.testingPutHDFS "" >> > org.apache.hadoop.fs.azure.AzureException: >> > java.util.NoSuchElementException: >> > An error occurred while enumerating the result, check the original >> > exception >> > for details. >> > at >> > >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie veMetadata(AzureNativeFileSystemStore.java:1930) >> > ~[hadoop-azure-2.7.2.jar:na] >> > at >> > >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat us(NativeAzureFileSystem.java:1592) >> > ~[hadoop-azure-2.7.2.jar:na] >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) >> > ~[hadoop-common-2.7.2.jar:na] >> > at >> > >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing PutHDFS.java:260) >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] >> > at >> > >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra ctProcessor.java:27) >> > [nifi-api-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S tandardProcessorNode.java:1054) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask .call(ContinuallyRunProcessorTask.java:136) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask .call(ContinuallyRunProcessorTask.java:47) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.scheduling.Time
Re: Flow File Stuck for no reason
Bryan, It's for uploading data on HDInsight cluster and as per the patch in Nifi-1922. Regards, Manish On Thu, Dec 1, 2016 at 12:16 PM, Bryan Bende wrote: > Manish, > > Was the reason for your custom version of PutHDFS just to include the > Azure libraries? or was there other functionality you needed to introduce? > > The reason I'm asking is because in the Apache NiFi 1.1 release (just > released) we added an "Additional Classpath Resources" property to all HDFS > processors, and I have been able to use that to connect to an Azure WASB > filesystem. > > I can provide the list of JARs I added to the classpath to make it work if > that would help. > > Thanks, > > Bryan > > > On Thu, Dec 1, 2016 at 12:11 PM, Manish G wrote: > >> Hi Joe, >> >> I am not using anything from Nifi 1.0 and its all 0.7. What indicates the >> use of 1.0? >> >> Regards, >> Manish >> >> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt wrote: >> >>> Manish >>> >>> It also appears from the log output you provided so far that this is a >>> combination of nifi 0.7 and 1.0 parts. We do not recommend doing this >>> and it is not likely to work. >>> >>> Please try the flow from a single release line. Apache NiFi 1.1.0 is >>> available for use now. >>> >>> Thanks >>> Joe >>> >>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G wrote: >>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I >>> see >>> > the issue again. >>> > >>> > Thanks, >>> > Manish >>> > >>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt wrote: >>> >> >>> >> Manish >>> >> >>> >> When it is in the stuck state can you please run >>> >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >>> >> would aid us in narrowing in on a possible cause. >>> >> >>> >> Thanks >>> >> Joe >>> >> >>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G >>> wrote: >>> >> > Hi Joe, >>> >> > >>> >> > Here is what I can see in the App Log: >>> >> > >>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] >>> >> > testing.nifi.processor.hdfs.testingPutHDFS "" >>> >> > org.apache.hadoop.fs.azure.AzureException: >>> >> > java.util.NoSuchElementException: >>> >> > An error occurred while enumerating the result, check the original >>> >> > exception >>> >> > for details. >>> >> > at >>> >> > >>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie >>> veMetadata(AzureNativeFileSystemStore.java:1930) >>> >> > ~[hadoop-azure-2.7.2.jar:na] >>> >> > at >>> >> > >>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat >>> us(NativeAzureFileSystem.java:1592) >>> >> > ~[hadoop-azure-2.7.2.jar:na] >>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) >>> >> > ~[hadoop-common-2.7.2.jar:na] >>> >> > at >>> >> > >>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing >>> PutHDFS.java:260) >>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] >>> >> > at >>> >> > >>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra >>> ctProcessor.java:27) >>> >> > [nifi-api-0.7.0.jar:0.7.0] >>> >> > at >>> >> > >>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S >>> tandardProcessorNode.java:1054) >>> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >>> >> > at >>> >> > >>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >>> .call(ContinuallyRunProcessorTask.java:136) >>> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >>> >> > at >>> >> > >>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >>> .call(ContinuallyRunProcessorTask.java:47) >>> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >>> >> > at >>> >> > >>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA >>> gent$1.run(TimerDrivenSchedulingAgent.java:127) >>> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >>> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>> Source) >>> >> > [na:1.8.0_101] >>> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) >>> >> > [na:1.8.0_101] >>> >> > at >>> >> > >>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >>> tureTask.access$301(Unknown >>> >> > Source) [na:1.8.0_101] >>> >> > at >>> >> > >>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >>> tureTask.run(Unknown >>> >> > Source) [na:1.8.0_101] >>> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>> Source) >>> >> > [na:1.8.0_101] >>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>> Source) >>> >> > [na:1.8.0_101] >>> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] >>> >> > Caused by: java.util.NoSuchElementException: An error occurred >>> while >>> >> > enumerating the result, check the original exception for details. >>> >> > at >>> >> > >>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe >>> xt(LazySegmentedIterator.java:113) >>> >> > ~[azure-storage-2.0.0.jar:na] >>> >> > at >>> >> > >>> >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIter >>> ator.hasNext(Storag
Re: Flow File Stuck for no reason
manish my apologies now I see what you mean. It is 0.7.0 nifi and you have a custom nar that is 1.0.0 which is totally fine. Bryan's response though is a good one to follow up on. Thanks joe On Thu, Dec 1, 2016 at 9:16 AM, Bryan Bende wrote: > Manish, > > Was the reason for your custom version of PutHDFS just to include the Azure > libraries? or was there other functionality you needed to introduce? > > The reason I'm asking is because in the Apache NiFi 1.1 release (just > released) we added an "Additional Classpath Resources" property to all HDFS > processors, and I have been able to use that to connect to an Azure WASB > filesystem. > > I can provide the list of JARs I added to the classpath to make it work if > that would help. > > Thanks, > > Bryan > > > On Thu, Dec 1, 2016 at 12:11 PM, Manish G wrote: >> >> Hi Joe, >> >> I am not using anything from Nifi 1.0 and its all 0.7. What indicates the >> use of 1.0? >> >> Regards, >> Manish >> >> On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt wrote: >>> >>> Manish >>> >>> It also appears from the log output you provided so far that this is a >>> combination of nifi 0.7 and 1.0 parts. We do not recommend doing this >>> and it is not likely to work. >>> >>> Please try the flow from a single release line. Apache NiFi 1.1.0 is >>> available for use now. >>> >>> Thanks >>> Joe >>> >>> On Thu, Dec 1, 2016 at 9:02 AM, Manish G wrote: >>> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I >>> > see >>> > the issue again. >>> > >>> > Thanks, >>> > Manish >>> > >>> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt wrote: >>> >> >>> >> Manish >>> >> >>> >> When it is in the stuck state can you please run >>> >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >>> >> would aid us in narrowing in on a possible cause. >>> >> >>> >> Thanks >>> >> Joe >>> >> >>> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G >>> >> wrote: >>> >> > Hi Joe, >>> >> > >>> >> > Here is what I can see in the App Log: >>> >> > >>> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] >>> >> > testing.nifi.processor.hdfs.testingPutHDFS "" >>> >> > org.apache.hadoop.fs.azure.AzureException: >>> >> > java.util.NoSuchElementException: >>> >> > An error occurred while enumerating the result, check the original >>> >> > exception >>> >> > for details. >>> >> > at >>> >> > >>> >> > >>> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930) >>> >> > ~[hadoop-azure-2.7.2.jar:na] >>> >> > at >>> >> > >>> >> > >>> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592) >>> >> > ~[hadoop-azure-2.7.2.jar:na] >>> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) >>> >> > ~[hadoop-common-2.7.2.jar:na] >>> >> > at >>> >> > >>> >> > >>> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260) >>> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] >>> >> > at >>> >> > >>> >> > >>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>> >> > [nifi-api-0.7.0.jar:0.7.0] >>> >> > at >>> >> > >>> >> > >>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) >>> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >>> >> > at >>> >> > >>> >> > >>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) >>> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >>> >> > at >>> >> > >>> >> > >>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) >>> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >>> >> > at >>> >> > >>> >> > >>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) >>> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >>> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>> >> > Source) >>> >> > [na:1.8.0_101] >>> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) >>> >> > [na:1.8.0_101] >>> >> > at >>> >> > >>> >> > >>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown >>> >> > Source) [na:1.8.0_101] >>> >> > at >>> >> > >>> >> > >>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown >>> >> > Source) [na:1.8.0_101] >>> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >>> >> > [na:1.8.0_101] >>> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>> >> > Source) >>> >> > [na:1.8.0_101] >>> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] >>> >> > Caused by: java.util.NoSuchElementException: An error occurred while >>> >> > enumerating the result, check the original exception for details. >>> >> > at >>> >> > >>> >> > >>> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113) >>> >> > ~[az
Re: Flow File Stuck for no reason
Manish, Was the reason for your custom version of PutHDFS just to include the Azure libraries? or was there other functionality you needed to introduce? The reason I'm asking is because in the Apache NiFi 1.1 release (just released) we added an "Additional Classpath Resources" property to all HDFS processors, and I have been able to use that to connect to an Azure WASB filesystem. I can provide the list of JARs I added to the classpath to make it work if that would help. Thanks, Bryan On Thu, Dec 1, 2016 at 12:11 PM, Manish G wrote: > Hi Joe, > > I am not using anything from Nifi 1.0 and its all 0.7. What indicates the > use of 1.0? > > Regards, > Manish > > On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt wrote: > >> Manish >> >> It also appears from the log output you provided so far that this is a >> combination of nifi 0.7 and 1.0 parts. We do not recommend doing this >> and it is not likely to work. >> >> Please try the flow from a single release line. Apache NiFi 1.1.0 is >> available for use now. >> >> Thanks >> Joe >> >> On Thu, Dec 1, 2016 at 9:02 AM, Manish G wrote: >> > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I >> see >> > the issue again. >> > >> > Thanks, >> > Manish >> > >> > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt wrote: >> >> >> >> Manish >> >> >> >> When it is in the stuck state can you please run >> >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >> >> would aid us in narrowing in on a possible cause. >> >> >> >> Thanks >> >> Joe >> >> >> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G >> wrote: >> >> > Hi Joe, >> >> > >> >> > Here is what I can see in the App Log: >> >> > >> >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] >> >> > testing.nifi.processor.hdfs.testingPutHDFS "" >> >> > org.apache.hadoop.fs.azure.AzureException: >> >> > java.util.NoSuchElementException: >> >> > An error occurred while enumerating the result, check the original >> >> > exception >> >> > for details. >> >> > at >> >> > >> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie >> veMetadata(AzureNativeFileSystemStore.java:1930) >> >> > ~[hadoop-azure-2.7.2.jar:na] >> >> > at >> >> > >> >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStat >> us(NativeAzureFileSystem.java:1592) >> >> > ~[hadoop-azure-2.7.2.jar:na] >> >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) >> >> > ~[hadoop-common-2.7.2.jar:na] >> >> > at >> >> > >> >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testing >> PutHDFS.java:260) >> >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] >> >> > at >> >> > >> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra >> ctProcessor.java:27) >> >> > [nifi-api-0.7.0.jar:0.7.0] >> >> > at >> >> > >> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S >> tandardProcessorNode.java:1054) >> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> >> > at >> >> > >> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >> .call(ContinuallyRunProcessorTask.java:136) >> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> >> > at >> >> > >> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >> .call(ContinuallyRunProcessorTask.java:47) >> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> >> > at >> >> > >> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA >> gent$1.run(TimerDrivenSchedulingAgent.java:127) >> >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >> Source) >> >> > [na:1.8.0_101] >> >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) >> >> > [na:1.8.0_101] >> >> > at >> >> > >> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.access$301(Unknown >> >> > Source) [na:1.8.0_101] >> >> > at >> >> > >> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.run(Unknown >> >> > Source) [na:1.8.0_101] >> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> >> > [na:1.8.0_101] >> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >> Source) >> >> > [na:1.8.0_101] >> >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] >> >> > Caused by: java.util.NoSuchElementException: An error occurred while >> >> > enumerating the result, check the original exception for details. >> >> > at >> >> > >> >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNe >> xt(LazySegmentedIterator.java:113) >> >> > ~[azure-storage-2.0.0.jar:na] >> >> > at >> >> > >> >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIter >> ator.hasNext(StorageInterfaceImpl.java:128) >> >> > ~[hadoop-azure-2.7.2.jar:na] >> >> > at >> >> > >> >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrie >> veMetadata(AzureNativeFileSystemStore.java:1909) >> >> > ~[hadoop-azure-2.7.2.jar:na] >> >> > ... 15 common frames omitted >> >> > Caused by: com.microsoft.azure.storage.Storag
Re: Flow File Stuck for no reason
Hi Joe, I am not using anything from Nifi 1.0 and its all 0.7. What indicates the use of 1.0? Regards, Manish On Thu, Dec 1, 2016 at 12:07 PM, Joe Witt wrote: > Manish > > It also appears from the log output you provided so far that this is a > combination of nifi 0.7 and 1.0 parts. We do not recommend doing this > and it is not likely to work. > > Please try the flow from a single release line. Apache NiFi 1.1.0 is > available for use now. > > Thanks > Joe > > On Thu, Dec 1, 2016 at 9:02 AM, Manish G wrote: > > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see > > the issue again. > > > > Thanks, > > Manish > > > > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt wrote: > >> > >> Manish > >> > >> When it is in the stuck state can you please run > >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that > >> would aid us in narrowing in on a possible cause. > >> > >> Thanks > >> Joe > >> > >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G > wrote: > >> > Hi Joe, > >> > > >> > Here is what I can see in the App Log: > >> > > >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] > >> > testing.nifi.processor.hdfs.testingPutHDFS "" > >> > org.apache.hadoop.fs.azure.AzureException: > >> > java.util.NoSuchElementException: > >> > An error occurred while enumerating the result, check the original > >> > exception > >> > for details. > >> > at > >> > > >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore. > retrieveMetadata(AzureNativeFileSystemStore.java:1930) > >> > ~[hadoop-azure-2.7.2.jar:na] > >> > at > >> > > >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus( > NativeAzureFileSystem.java:1592) > >> > ~[hadoop-azure-2.7.2.jar:na] > >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) > >> > ~[hadoop-common-2.7.2.jar:na] > >> > at > >> > > >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger( > testingPutHDFS.java:260) > >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] > >> > at > >> > > >> > org.apache.nifi.processor.AbstractProcessor.onTrigger( > AbstractProcessor.java:27) > >> > [nifi-api-0.7.0.jar:0.7.0] > >> > at > >> > > >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger( > StandardProcessorNode.java:1054) > >> > [nifi-framework-core-0.7.0.jar:0.7.0] > >> > at > >> > > >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call( > ContinuallyRunProcessorTask.java:136) > >> > [nifi-framework-core-0.7.0.jar:0.7.0] > >> > at > >> > > >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call( > ContinuallyRunProcessorTask.java:47) > >> > [nifi-framework-core-0.7.0.jar:0.7.0] > >> > at > >> > > >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1. > run(TimerDrivenSchedulingAgent.java:127) > >> > [nifi-framework-core-0.7.0.jar:0.7.0] > >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > Source) > >> > [na:1.8.0_101] > >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > >> > [na:1.8.0_101] > >> > at > >> > > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$301(Unknown > >> > Source) [na:1.8.0_101] > >> > at > >> > > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(Unknown > >> > Source) [na:1.8.0_101] > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > >> > [na:1.8.0_101] > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > >> > [na:1.8.0_101] > >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] > >> > Caused by: java.util.NoSuchElementException: An error occurred while > >> > enumerating the result, check the original exception for details. > >> > at > >> > > >> > com.microsoft.azure.storage.core.LazySegmentedIterator. > hasNext(LazySegmentedIterator.java:113) > >> > ~[azure-storage-2.0.0.jar:na] > >> > at > >> > > >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$ > WrappingIterator.hasNext(StorageInterfaceImpl.java:128) > >> > ~[hadoop-azure-2.7.2.jar:na] > >> > at > >> > > >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore. > retrieveMetadata(AzureNativeFileSystemStore.java:1909) > >> > ~[hadoop-azure-2.7.2.jar:na] > >> > ... 15 common frames omitted > >> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden > >> > at > >> > > >> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus( > StorageException.java:202) > >> > ~[azure-storage-2.0.0.jar:na] > >> > at > >> > > >> > com.microsoft.azure.storage.StorageException.translateException( > StorageException.java:172) > >> > ~[azure-storage-2.0.0.jar:na] > >> > at > >> > > >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry( > ExecutionEngine.java:273) > >> > ~[azure-storage-2.0.0.jar:na] > >> > at > >> > > >> > com.microsoft.azure.storage.core.LazySegmentedIterator. > hasNext(LazySegmentedIterator.java:109) > >> > ~[azure-storage-2.0.0.jar:na] > >> > ... 17 common frames omitted > >> >
Re: Flow File Stuck for no reason
Manish It also appears from the log output you provided so far that this is a combination of nifi 0.7 and 1.0 parts. We do not recommend doing this and it is not likely to work. Please try the flow from a single release line. Apache NiFi 1.1.0 is available for use now. Thanks Joe On Thu, Dec 1, 2016 at 9:02 AM, Manish G wrote: > Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see > the issue again. > > Thanks, > Manish > > On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt wrote: >> >> Manish >> >> When it is in the stuck state can you please run >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >> would aid us in narrowing in on a possible cause. >> >> Thanks >> Joe >> >> On Thu, Dec 1, 2016 at 8:44 AM, Manish G wrote: >> > Hi Joe, >> > >> > Here is what I can see in the App Log: >> > >> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] >> > testing.nifi.processor.hdfs.testingPutHDFS "" >> > org.apache.hadoop.fs.azure.AzureException: >> > java.util.NoSuchElementException: >> > An error occurred while enumerating the result, check the original >> > exception >> > for details. >> > at >> > >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930) >> > ~[hadoop-azure-2.7.2.jar:na] >> > at >> > >> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592) >> > ~[hadoop-azure-2.7.2.jar:na] >> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) >> > ~[hadoop-common-2.7.2.jar:na] >> > at >> > >> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260) >> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] >> > at >> > >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> > [nifi-api-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at >> > >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) >> > [nifi-framework-core-0.7.0.jar:0.7.0] >> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) >> > [na:1.8.0_101] >> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) >> > [na:1.8.0_101] >> > at >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown >> > Source) [na:1.8.0_101] >> > at >> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown >> > Source) [na:1.8.0_101] >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) >> > [na:1.8.0_101] >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> > [na:1.8.0_101] >> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] >> > Caused by: java.util.NoSuchElementException: An error occurred while >> > enumerating the result, check the original exception for details. >> > at >> > >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113) >> > ~[azure-storage-2.0.0.jar:na] >> > at >> > >> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128) >> > ~[hadoop-azure-2.7.2.jar:na] >> > at >> > >> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909) >> > ~[hadoop-azure-2.7.2.jar:na] >> > ... 15 common frames omitted >> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden >> > at >> > >> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202) >> > ~[azure-storage-2.0.0.jar:na] >> > at >> > >> > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172) >> > ~[azure-storage-2.0.0.jar:na] >> > at >> > >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273) >> > ~[azure-storage-2.0.0.jar:na] >> > at >> > >> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109) >> > ~[azure-storage-2.0.0.jar:na] >> > ... 17 common frames omitted >> > Caused by: java.lang.NullPointerException: null >> > at >> > >> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181) >> > ~[azure-storage-2.0.0.jar:na] >> > ... 18 common frames omitted >> > >> > Regards, >> > Manish >> > >> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt wrote: >> >> >> >> Manish >> >> >> >> Please produce and share the thread dump I mentioned. >> >> >> >> Thanks >> >> Joe >> >> >> >> On Dec 1, 2016 7:23 AM,
Re: Flow File Stuck for no reason
Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see the issue again. Thanks, Manish On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt wrote: > Manish > > When it is in the stuck state can you please run > bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that > would aid us in narrowing in on a possible cause. > > Thanks > Joe > > On Thu, Dec 1, 2016 at 8:44 AM, Manish G wrote: > > Hi Joe, > > > > Here is what I can see in the App Log: > > > > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] > > testing.nifi.processor.hdfs.testingPutHDFS "" > > org.apache.hadoop.fs.azure.AzureException: java.util. > NoSuchElementException: > > An error occurred while enumerating the result, check the original > exception > > for details. > > at > > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata( > AzureNativeFileSystemStore.java:1930) > > ~[hadoop-azure-2.7.2.jar:na] > > at > > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus( > NativeAzureFileSystem.java:1592) > > ~[hadoop-azure-2.7.2.jar:na] > > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) > > ~[hadoop-common-2.7.2.jar:na] > > at > > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger( > testingPutHDFS.java:260) > > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] > > at > > org.apache.nifi.processor.AbstractProcessor.onTrigger( > AbstractProcessor.java:27) > > [nifi-api-0.7.0.jar:0.7.0] > > at > > org.apache.nifi.controller.StandardProcessorNode.onTrigger( > StandardProcessorNode.java:1054) > > [nifi-framework-core-0.7.0.jar:0.7.0] > > at > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call( > ContinuallyRunProcessorTask.java:136) > > [nifi-framework-core-0.7.0.jar:0.7.0] > > at > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call( > ContinuallyRunProcessorTask.java:47) > > [nifi-framework-core-0.7.0.jar:0.7.0] > > at > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run( > TimerDrivenSchedulingAgent.java:127) > > [nifi-framework-core-0.7.0.jar:0.7.0] > > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > > [na:1.8.0_101] > > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > > [na:1.8.0_101] > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$301(Unknown > > Source) [na:1.8.0_101] > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(Unknown > > Source) [na:1.8.0_101] > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > [na:1.8.0_101] > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > [na:1.8.0_101] > > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] > > Caused by: java.util.NoSuchElementException: An error occurred while > > enumerating the result, check the original exception for details. > > at > > com.microsoft.azure.storage.core.LazySegmentedIterator. > hasNext(LazySegmentedIterator.java:113) > > ~[azure-storage-2.0.0.jar:na] > > at > > org.apache.hadoop.fs.azure.StorageInterfaceImpl$ > WrappingIterator.hasNext(StorageInterfaceImpl.java:128) > > ~[hadoop-azure-2.7.2.jar:na] > > at > > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata( > AzureNativeFileSystemStore.java:1909) > > ~[hadoop-azure-2.7.2.jar:na] > > ... 15 common frames omitted > > Caused by: com.microsoft.azure.storage.StorageException: Forbidden > > at > > com.microsoft.azure.storage.StorageException.translateFromHttpStatus( > StorageException.java:202) > > ~[azure-storage-2.0.0.jar:na] > > at > > com.microsoft.azure.storage.StorageException.translateException( > StorageException.java:172) > > ~[azure-storage-2.0.0.jar:na] > > at > > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry( > ExecutionEngine.java:273) > > ~[azure-storage-2.0.0.jar:na] > > at > > com.microsoft.azure.storage.core.LazySegmentedIterator. > hasNext(LazySegmentedIterator.java:109) > > ~[azure-storage-2.0.0.jar:na] > > ... 17 common frames omitted > > Caused by: java.lang.NullPointerException: null > > at > > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry( > ExecutionEngine.java:181) > > ~[azure-storage-2.0.0.jar:na] > > ... 18 common frames omitted > > > > Regards, > > Manish > > > > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt wrote: > >> > >> Manish > >> > >> Please produce and share the thread dump I mentioned. > >> > >> Thanks > >> Joe > >> > >> On Dec 1, 2016 7:23 AM, "Manish G" wrote: > >>> > >>> Hi, > >>> > >>> I don't know why, but this is happening now more frequently. Where > should > >>> I look into to find the root cause? > >>> > >>> Thanks, > >>> Manish > >>> > >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G > wrote: > > Hi Joe, > > Thanks for the quick reply. Yes, the processor keeps running on a > single > thread (even after stopping). And the number remains there even after > stopping. > Today, it happened on
Re: Flow File Stuck for no reason
Manish When it is in the stuck state can you please run bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that would aid us in narrowing in on a possible cause. Thanks Joe On Thu, Dec 1, 2016 at 8:44 AM, Manish G wrote: > Hi Joe, > > Here is what I can see in the App Log: > > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] > testing.nifi.processor.hdfs.testingPutHDFS "" > org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException: > An error occurred while enumerating the result, check the original exception > for details. > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930) > ~[hadoop-azure-2.7.2.jar:na] > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592) > ~[hadoop-azure-2.7.2.jar:na] > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) > ~[hadoop-common-2.7.2.jar:na] > at > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260) > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > [nifi-api-0.7.0.jar:0.7.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) > [nifi-framework-core-0.7.0.jar:0.7.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-0.7.0.jar:0.7.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-0.7.0.jar:0.7.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) > [nifi-framework-core-0.7.0.jar:0.7.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > [na:1.8.0_101] > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > Source) [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) [na:1.8.0_101] > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > [na:1.8.0_101] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > [na:1.8.0_101] > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] > Caused by: java.util.NoSuchElementException: An error occurred while > enumerating the result, check the original exception for details. > at > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113) > ~[azure-storage-2.0.0.jar:na] > at > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128) > ~[hadoop-azure-2.7.2.jar:na] > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909) > ~[hadoop-azure-2.7.2.jar:na] > ... 15 common frames omitted > Caused by: com.microsoft.azure.storage.StorageException: Forbidden > at > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202) > ~[azure-storage-2.0.0.jar:na] > at > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172) > ~[azure-storage-2.0.0.jar:na] > at > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273) > ~[azure-storage-2.0.0.jar:na] > at > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109) > ~[azure-storage-2.0.0.jar:na] > ... 17 common frames omitted > Caused by: java.lang.NullPointerException: null > at > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181) > ~[azure-storage-2.0.0.jar:na] > ... 18 common frames omitted > > Regards, > Manish > > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt wrote: >> >> Manish >> >> Please produce and share the thread dump I mentioned. >> >> Thanks >> Joe >> >> On Dec 1, 2016 7:23 AM, "Manish G" wrote: >>> >>> Hi, >>> >>> I don't know why, but this is happening now more frequently. Where should >>> I look into to find the root cause? >>> >>> Thanks, >>> Manish >>> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G wrote: Hi Joe, Thanks for the quick reply. Yes, the processor keeps running on a single thread (even after stopping). And the number remains there even after stopping. Today, it happened on my customized putHDFS processor. Only thing different in this processor is - I have added an additional attribute that tells if the processor created the directory while loading the file on HDFS. I don't think this should be the issue though. Regards, Manish On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt wrote: > > Manish > > When it is stuck do you see a number in the top right corner of the >
Re: Flow File Stuck for no reason
Hi Joe, Here is what I can see in the App Log: 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4] testing.nifi.processor.hdfs.testingPutHDFS "" org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details. at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930) ~[hadoop-azure-2.7.2.jar:na] at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592) ~[hadoop-azure-2.7.2.jar:na] at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) ~[hadoop-common-2.7.2.jar:na] at testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260) ~[testing.nifi.processor-1.0.0.nar-unpacked/:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-0.7.0.jar:0.7.0] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) [nifi-framework-core-0.7.0.jar:0.7.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0.jar:0.7.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0.jar:0.7.0] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) [nifi-framework-core-0.7.0.jar:0.7.0] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_101] at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_101] at java.lang.Thread.run(Unknown Source) [na:1.8.0_101] Caused by: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details. at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113) ~[azure-storage-2.0.0.jar:na] at org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128) ~[hadoop-azure-2.7.2.jar:na] at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909) ~[hadoop-azure-2.7.2.jar:na] ... 15 common frames omitted Caused by: com.microsoft.azure.storage.StorageException: Forbidden at com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202) ~[azure-storage-2.0.0.jar:na] at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172) ~[azure-storage-2.0.0.jar:na] at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273) ~[azure-storage-2.0.0.jar:na] at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109) ~[azure-storage-2.0.0.jar:na] ... 17 common frames omitted Caused by: java.lang.NullPointerException: null at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181) ~[azure-storage-2.0.0.jar:na] ... 18 common frames omitted Regards, Manish On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt wrote: > Manish > > Please produce and share the thread dump I mentioned. > > Thanks > Joe > > On Dec 1, 2016 7:23 AM, "Manish G" wrote: > >> Hi, >> >> I don't know why, but this is happening now more frequently. Where should >> I look into to find the root cause? >> >> Thanks, >> Manish >> >> On Wed, Nov 30, 2016 at 9:20 PM, Manish G wrote: >> >>> Hi Joe, >>> >>> Thanks for the quick reply. Yes, the processor keeps running on a single >>> thread (even after stopping). And the number remains there even after >>> stopping. >>> Today, it happened on my customized putHDFS processor. Only thing >>> different in this processor is - I have added an additional attribute that >>> tells if the processor created the directory while loading the file on >>> HDFS. I don't think this should be the issue though. >>> >>> Regards, >>> Manish >>> >>> >>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt wrote: >>> Manish When it is stuck do you see a number in the top right corner of the processor? When you stop it does the number remain? That number is telling you how many threads are still executing. Which processor are we talking about? When it is in the stuck state can you please run bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that would aid us in narrowing in on a possible cause. Thanks Joe On Wed, Nov 30, 2016 at 7:02 PM, Manish G wrote: > > Hi,
Re: Flow File Stuck for no reason
Manish Please produce and share the thread dump I mentioned. Thanks Joe On Dec 1, 2016 7:23 AM, "Manish G" wrote: > Hi, > > I don't know why, but this is happening now more frequently. Where should > I look into to find the root cause? > > Thanks, > Manish > > On Wed, Nov 30, 2016 at 9:20 PM, Manish G wrote: > >> Hi Joe, >> >> Thanks for the quick reply. Yes, the processor keeps running on a single >> thread (even after stopping). And the number remains there even after >> stopping. >> Today, it happened on my customized putHDFS processor. Only thing >> different in this processor is - I have added an additional attribute that >> tells if the processor created the directory while loading the file on >> HDFS. I don't think this should be the issue though. >> >> Regards, >> Manish >> >> >> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt wrote: >> >>> Manish >>> >>> When it is stuck do you see a number in the top right corner of the >>> processor? When you stop it does the number remain? That number is >>> telling you how many threads are still executing. Which processor are >>> we talking about? When it is in the stuck state can you please run >>> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >>> would aid us in narrowing in on a possible cause. >>> >>> Thanks >>> Joe >>> >>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G >>> wrote: >>> > >>> > Hi, >>> > >>> > I have noticed that sometime a flow file gets stuck on a processor for >>> a >>> > very long time for no reason and then I can not even stop the >>> processor to >>> > look at the flow flow file from queue. If I click on stop, then >>> processor >>> > goes into a state where I cannot start/stop the processor. >>> > >>> > On restarting the NiFi, the file gets processed successfully and >>> routed to >>> > success queue. I checked in App log, but everything seems to be normal >>> for >>> > the flow file. I don't see anything mysterious in provenance too >>> (except >>> > that queue time is in hours). >>> > >>> > Has anyone else faced a similar issue? What else should I check to >>> identify >>> > the root cause for this? >>> > >>> > Thanks, >>> > Manish >>> >> >> >> >> -- >> >> >> *With Warm Regards,* >> *Manish* >> > > > > -- > > > *With Warm Regards,* > *Manish* >
Re: Flow File Stuck for no reason
Hi, I don't know why, but this is happening now more frequently. Where should I look into to find the root cause? Thanks, Manish On Wed, Nov 30, 2016 at 9:20 PM, Manish G wrote: > Hi Joe, > > Thanks for the quick reply. Yes, the processor keeps running on a single > thread (even after stopping). And the number remains there even after > stopping. > Today, it happened on my customized putHDFS processor. Only thing > different in this processor is - I have added an additional attribute that > tells if the processor created the directory while loading the file on > HDFS. I don't think this should be the issue though. > > Regards, > Manish > > > On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt wrote: > >> Manish >> >> When it is stuck do you see a number in the top right corner of the >> processor? When you stop it does the number remain? That number is >> telling you how many threads are still executing. Which processor are >> we talking about? When it is in the stuck state can you please run >> bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that >> would aid us in narrowing in on a possible cause. >> >> Thanks >> Joe >> >> On Wed, Nov 30, 2016 at 7:02 PM, Manish G wrote: >> > >> > Hi, >> > >> > I have noticed that sometime a flow file gets stuck on a processor for a >> > very long time for no reason and then I can not even stop the processor >> to >> > look at the flow flow file from queue. If I click on stop, then >> processor >> > goes into a state where I cannot start/stop the processor. >> > >> > On restarting the NiFi, the file gets processed successfully and routed >> to >> > success queue. I checked in App log, but everything seems to be normal >> for >> > the flow file. I don't see anything mysterious in provenance too (except >> > that queue time is in hours). >> > >> > Has anyone else faced a similar issue? What else should I check to >> identify >> > the root cause for this? >> > >> > Thanks, >> > Manish >> > > > > -- > > > *With Warm Regards,* > *Manish* > -- *With Warm Regards,* *Manish*
Re: Flow File Stuck for no reason
Hi Joe, Thanks for the quick reply. Yes, the processor keeps running on a single thread (even after stopping). And the number remains there even after stopping. Today, it happened on my customized putHDFS processor. Only thing different in this processor is - I have added an additional attribute that tells if the processor created the directory while loading the file on HDFS. I don't think this should be the issue though. Regards, Manish On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt wrote: > Manish > > When it is stuck do you see a number in the top right corner of the > processor? When you stop it does the number remain? That number is > telling you how many threads are still executing. Which processor are > we talking about? When it is in the stuck state can you please run > bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that > would aid us in narrowing in on a possible cause. > > Thanks > Joe > > On Wed, Nov 30, 2016 at 7:02 PM, Manish G wrote: > > > > Hi, > > > > I have noticed that sometime a flow file gets stuck on a processor for a > > very long time for no reason and then I can not even stop the processor > to > > look at the flow flow file from queue. If I click on stop, then processor > > goes into a state where I cannot start/stop the processor. > > > > On restarting the NiFi, the file gets processed successfully and routed > to > > success queue. I checked in App log, but everything seems to be normal > for > > the flow file. I don't see anything mysterious in provenance too (except > > that queue time is in hours). > > > > Has anyone else faced a similar issue? What else should I check to > identify > > the root cause for this? > > > > Thanks, > > Manish > -- *With Warm Regards,* *Manish*
Re: Flow File Stuck for no reason
Manish When it is stuck do you see a number in the top right corner of the processor? When you stop it does the number remain? That number is telling you how many threads are still executing. Which processor are we talking about? When it is in the stuck state can you please run bin/nifi.sh dump. If you can then share the nifi-bootstrap.log that would aid us in narrowing in on a possible cause. Thanks Joe On Wed, Nov 30, 2016 at 7:02 PM, Manish G wrote: > > Hi, > > I have noticed that sometime a flow file gets stuck on a processor for a > very long time for no reason and then I can not even stop the processor to > look at the flow flow file from queue. If I click on stop, then processor > goes into a state where I cannot start/stop the processor. > > On restarting the NiFi, the file gets processed successfully and routed to > success queue. I checked in App log, but everything seems to be normal for > the flow file. I don't see anything mysterious in provenance too (except > that queue time is in hours). > > Has anyone else faced a similar issue? What else should I check to identify > the root cause for this? > > Thanks, > Manish
Flow File Stuck for no reason
Hi, I have noticed that sometime a flow file gets stuck on a processor for a very long time for no reason and then I can not even stop the processor to look at the flow flow file from queue. If I click on stop, then processor goes into a state where I cannot start/stop the processor. On restarting the NiFi, the *file gets processed successfully* and routed to success queue. I checked in App log, but everything seems to be normal for the flow file. I don't see anything mysterious in provenance too (except that queue time is in hours). Has anyone else faced a similar issue? What else should I check to identify the root cause for this? Thanks, Manish