I didn’t add any test code. This custom controller and processor is working for a small size file.
Thanks Kumiko From: Tony Kurc [mailto:[email protected]] Sent: Tuesday, September 6, 2016 2:32 PM To: [email protected] Cc: Joe Witt <[email protected]>; #Operations Automation and Tools <#[email protected]>; Kevin Verhoeven <[email protected]> Subject: RE: Issue writing file (~50mb) to azure data lake with Nifi I didn't see the test script that worked in the source code - did I miss it, or is it not in the tree? On Sep 6, 2016 3:17 PM, "Kumiko Yada" <[email protected]<mailto:[email protected]>> wrote: Joe, Here is the log (there was no callstack related to this error) and code, https://github.com/kyada1/dl_sdkworkaround/tree/master/nifi-azure-dlstore-bundle. 2016-09-06 12:06:50,508 INFO [NiFi Web Server-19] c.s.j.s.i.application.WebApplicationImpl Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM' 2016-09-06 12:07:00,991 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutFileAzureDLStore[id=00dd95dc-0157-1000-8ab1-2de88c159b55] to run with 1 threads 2016-09-06 12:07:01,545 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@28f414a1<mailto:org.apache.nifi.controller.FlowController@28f414a1> // Another save pending = false 2016-09-06 12:07:01,904 INFO [pool-27-thread-1] c.m.aad.adal4j.AuthenticationAuthority [Correlation ID: 564fb5ec-643b-43e6-ab68-59f259a4843a] Instance discovery was successful 2016-09-06 12:08:05,988 ERROR [Timer-Driven Process Thread-1] n.a.d.processors.PutFileAzureDLStore PutFileAzureDLStore[id=00dd95dc-0157-1000-8ab1-2de88c159b55] File was not created: /kumiko/test/20160906120701022.txt com.microsoft.azure.management.datalake.store.models.AdlsErrorException: Invalid status code 404 2016-09-06 12:08:12,541 INFO [Provenance Maintenance Thread-1] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 618 2016-09-06 12:08:12,838 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (1 records) into single Provenance Log File .\provenance_repository\617.prov in 305 milliseconds 2016-09-06 12:08:12,838 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 1 records 2016-09-06 12:08:21,148 INFO [NiFi Web Server-28] o.a.n.controller.StandardProcessorNode Stopping processor: class nifi.azure.dlstore.processors.PutFileAzureDLStore Thanks Kumiko From: Joe Witt [mailto:[email protected]<mailto:[email protected]>] Sent: Friday, September 2, 2016 2:56 PM To: [email protected]<mailto:[email protected]> Cc: #Operations Automation and Tools <#[email protected]<mailto:[email protected]>>; Kevin Verhoeven <[email protected]<mailto:[email protected]>> Subject: Re: Issue writing file (~50mb) to azure data lake with Nifi Lorenzo Without seeing the code and logs it would be very difficult to help. nifi has no trouble by design writing large files (GBs) to many things including hdfs so the issue is probably in how this client library interacts with the data stream. On Sep 2, 2016 4:19 PM, "Lorenzo Peder" <[email protected]<mailto:[email protected]>> wrote: Hi All, We’ve run into an issue uploading a larger file (~50Mb) into an Azure Data Lake using a custom processor in nifi 0.7-1.0. This custom processor has worked consistently for smaller files, but once encountered with this larger file, it spits http error 404 (file not found). Eventually a minor portion of the file wrote to the data lake. We used fiddler to capture network traffic between Nifi and the Azure data lake while the processor was running and captured http error 204 (no contents). Then we wrote a java sdk script to upload this same file without Nifi into the data lake and it worked successfully. These findings lead us to believe that this issue is occurring within Nifi, if someone could please point us in the right direction in resolving this issue it would be greatly appreciated. Thank you, Lorenzo Peder Operations Analyst, Campaign Operations & Services 425.974.1363<tel:425.974.1363> : Office 425.260.5027<tel:425.260.5027> : Mobile www.ds-iq.com<http://www.ds-iq.com> [cid:[email protected]] Dynamic Shopper Intelligence This e-mail may contain confidential or privileged information. If you are not the intended recipient, please notify the sender immediately and then delete this message.
