I had a similar issue but with ConsumerJMS. On Wed, 26 Jul 2017 at 10:04 Noel Alex Makumuli <[email protected]> wrote:
> Hello Mark, > Thanks for the prompt response. > Yes the folder has the data. > > /data/data_warehouse/data_sources/PCT/PCT_Backfile_non_Asian/nifi/sources$ > find -type f -iname wo\*.xml | wc -l > 157905 > > The GetFile is not showing anything wrong run or attention that something > is missing. > > I duplicated the whole flow and changed the sources a but. It work only > worked for few thousands files and then it just no longer GetFile for > extraction. > > I am sending couple of images of the processor GetFile configuration and > you can advice further. > > Regards, > Noel > > > > On 26 July 2017 at 15:44, Mark Payne <[email protected]> wrote: > >> Noel, >> >> From the logs, it appears that you only have a handful of Processors. The >> only one that is a "source" processor >> is the GetFile. Are you sure that it is pointing to a directory that has >> data in it? Is it possible that you've got >> a typo there so that it isn't finding any data? Can you share the config >> for that processor? >> >> Thanks >> -Mark >> >> >> On Jul 26, 2017, at 7:37 AM, Noel Alex Makumuli <[email protected]> >> wrote: >> >> Hello, >> I am running Apache Nifi 1.3.0 on Ubuntu Server. >> >> The application is running fine and prints >> <code> >> 2017-07-26 14:01:54,131 INFO [main] org.apache.nifi.BootstrapListener >> Successfully initiated communication with Bootstrap >> 2017-07-26 14:01:54,131 INFO [main] org.apache.nifi.NiFi Controller >> initialization took 15369655939 nanoseconds. >> 2017-07-26 14:02:04,934 INFO [NiFi Web Server-50] >> c.s.j.s.i.application.WebApplicationImpl Initiating Jersey application, >> version 'Jersey: 1.19 02/11/2015 03:25 AM' >> 2017-07-26 14:02:30,370 INFO [StandardProcessScheduler Thread-5] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> PutFile[id=dee71437-10c9-115d-38ff-22e157972467] to run with 1 threads >> 2017-07-26 14:02:30,369 INFO [StandardProcessScheduler Thread-4] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> PutFile[id=dee7141f-10c9-115d-dcbf-430c652f8b48] to run with 1 threads >> 2017-07-26 14:02:30,370 INFO [StandardProcessScheduler Thread-3] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> PutFile[id=dee71430-10c9-115d-8a1a-d2e0355524d6] to run with 1 threads >> 2017-07-26 14:02:30,372 INFO [StandardProcessScheduler Thread-8] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> PutFile[id=dee71426-10c9-115d-701b-e65fcd56d485] to run with 1 threads >> 2017-07-26 14:02:30,372 INFO [StandardProcessScheduler Thread-5] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> PutFile[id=dee7141c-10c9-115d-ecc1-4fb0c3d2a5a3] to run with 1 threads >> 2017-07-26 14:02:30,374 INFO [StandardProcessScheduler Thread-4] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> PutFile[id=dee71433-10c9-115d-ff6c-a0160d72c5d4] to run with 1 threads >> 2017-07-26 14:02:30,373 INFO [StandardProcessScheduler Thread-6] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> EvaluateXPath[id=dee71422-10c9-115d-bb94-79ed7852b19b] to run with 1 >> threads >> 2017-07-26 14:02:30,375 INFO [StandardProcessScheduler Thread-8] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> PutFile[id=dee71428-10c9-115d-190e-c24016ed8015] to run with 1 threads >> 2017-07-26 14:02:30,376 INFO [StandardProcessScheduler Thread-3] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> ReplaceText[id=015d100b-704a-1013-58f4-955b7fe48f9c] to run with 1 threads >> 2017-07-26 14:02:30,381 INFO [StandardProcessScheduler Thread-7] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> GetFile[id=015d10c8-dee7-1411-5904-52201a329835] to run with 1 threads >> 2017-07-26 14:02:30,407 INFO [StandardProcessScheduler Thread-1] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> PutSQL[id=74971126-015d-1000-0838-8db7f5065da7] to run with 1 threads >> 2017-07-26 14:02:30,434 INFO [StandardProcessScheduler Thread-2] >> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >> TransformXml[id=791eee75-015d-1000-5c5d-df2bd5a4192b] to run with 1 threads >> 2017-07-26 14:02:30,585 INFO [Flow Service Tasks Thread-1] >> o.a.nifi.controller.StandardFlowService Saved flow controller >> org.apache.nifi.controller.FlowController@10c992ce // Another save >> pending = false >> 2017-07-26 14:03:50,962 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@296a71df checkpointed with 114 >> Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 2 >> milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID >> 351 >> 2017-07-26 14:03:52,616 INFO [pool-10-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> 2017-07-26 14:03:53,064 INFO [pool-10-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705 >> Records and 0 Swap Files in 448 milliseconds (Stop-the-world time = 132 >> milliseconds, Clear Edit Logs time = 53 millis), max Transaction ID >> 1597176945 >> 2017-07-26 14:03:53,065 INFO [pool-10-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 6705 records in 448 milliseconds >> 2017-07-26 14:05:50,980 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@296a71df checkpointed with 114 >> Records and 0 Swap Files in 17 milliseconds (Stop-the-world time = 2 >> milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID >> 351 >> 2017-07-26 14:05:53,065 INFO [pool-10-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> 2017-07-26 14:05:53,477 INFO [pool-10-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705 >> Records and 0 Swap Files in 411 milliseconds (Stop-the-world time = 91 >> milliseconds, Clear Edit Logs time = 63 millis), max Transaction ID >> 1597176945 >> 2017-07-26 14:05:53,477 INFO [pool-10-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 6705 records in 411 milliseconds >> 2017-07-26 14:07:50,998 INFO [Write-Ahead Local State Provider >> Maintenance] org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@296a71df checkpointed with 114 >> Records and 0 Swap Files in 16 milliseconds (Stop-the-world time = 2 >> milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID >> 351 >> 2017-07-26 14:07:53,478 INFO [pool-10-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> 2017-07-26 14:07:53,885 INFO [pool-10-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705 >> Records and 0 Swap Files in 406 milliseconds (Stop-the-world time = 86 >> milliseconds, Clear Edit Logs time = 55 millis), max Transaction ID >> 1597176945 >> 2017-07-26 14:07:53,885 INFO [pool-10-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 6705 records in 406 milliseconds >> </code> >> >> The bootstrap is showing everything is normal only nifi-app.log is >> printing the above logs >> >> While the bootstrap.log is printing like below and nothing out of the >> ordinary or any exceptions. >> <code> >> ce/LOGS org.apache.nifi.NiFi >> 2017-07-26 14:01:34,005 INFO [main] org.apache.nifi.bootstrap.Command >> Launched Apache NiFi with Proc >> ess ID 12143 >> 2017-07-26 14:01:34,705 INFO [NiFi Bootstrap Command Listener] >> org.apache.nifi.bootstrap.RunNiFi Apa >> che NiFi now running and listening for Bootstrap requests on port 32977 >> </code> >> >> I changed couple of few things on the configuration mainly the memory to >> gain performance but nothing further.. I am no longer sure what is >> happening. >> onifi >> I have attached my log file , bootstrap & nifi configuration. >> >> on my configuration files.. >> i disable the # The G1GC is still considered experimental but has proven >> to be very advantageous in providing great >> # performance without significant "stop-the-world" delays. >> #java.arg.13=-XX:+UseG1GC >> >> I saw in one thread people complaining of this. >> >> I have lots of files which need to get processed but nifi is just >> standing there while it should be ingesting the file. >> My last change on the nifi.properties file was:: >> # nifi.flow.configuration.archive.max.storage=500 MB >> nifi.flow.configuration.archive.max.storage=50 GB >> Could this be the culprit..?? Because reverting back does seem to help >> neither..!! >> >> I apologies if I have failed to frame my question correctly. >> >> Please advice. >> -- >> NOEL ALEX MAKUMULI >> >> <bootstrap.txt><nifi.txt><nifi-app.log> >> >> >> > > > -- > NOEL ALEX MAKUMULI > TANZANIA > > Mobile: +255 755 100 700 <+255%20755%20100%20700> ( Active) > Mobile: +358 44 3077 817 ( Active) >
