Hi Aldrin, After the DDA_Processor
The below image shows that the GetFile Processed 174.6 MB and the DDA_Processor is working on 1 file (the 1 in the upper right of the DDA_Processor box) The below image shows that the DDA_Processor is complete but data did not make it to ConvertJSONtoAvro. No errors are being generated. DDA_Processor takes fixed width data and converts it to JSON. Thanks > On Sep 25, 2015, at 7:30 AM, Aldrin Piri <[email protected]> wrote: > > Jeff, > > With regards to: > > "Anything over, the GetFile and DDA_Processor shows data movement but the no > other downstream processor shows movement." > > Are you referencing downstream processors starting immediately after the > DDA_Processor (ConvertJsonToAvro) or starting immediately after the > ConvertJsonToAvro processor? > > In the case of starting immediately after the DDA Processor, as it is a > custom processor, we would need some additional information as to how this > processor is behaving. In the case of the second condition, if you have some > additional context as to the format of the data that is problematic to what > you are seeing (the effective "schema" of the JSON) would be helpful in > tracking down the issue. > > Thanks! > Aldrin > > On Fri, Sep 25, 2015 at 8:22 AM, Jeff <[email protected] > <mailto:[email protected]>> wrote: > Hi Adam, > > > I have a flow that does the following; > > GetFile > DDA_Processor > ConvertJSONToAvro > UpdateAttribute > PutFile > > My source file has 182897 rows at 1001 bytes per row. If I do any number of > rows under ~15000 an output file is created. Anything over, the GetFile and > DDA_Processor shows data movement but the no other downstream processor shows > movement. > > I confirmed that it is not a data problem by processing a 10,000 row file > successfully, then concatenating 10,000 rows into one file twice. > > Thanks for your insight. > > Jeff > <Mail Attachment.gif> > > >> On Sep 24, 2015, at 8:40 PM, Aldrin Piri <[email protected] >> <mailto:[email protected]>> wrote: >> >> Jeff, >> >> This seems to be a bit different as the processor is showing data as having >> been written and there is a listing of one FlowFile of 381 MB being >> transferred out from the processor. Could you provide additional >> information as to how data is not being sent out in the manner anticipated? >> If you can track the issue down more, let us know. May be helpful to create >> another message to help us track the issues separately as we work through >> them. >> >> Thanks! >> >> Adam, >> >> Found a sizable JSON file to work against and have been doing some initial >> exploration. With the large files, it certainly is a nontrivial process. >> At cursory inspection, a good portion of processing seems to be spent on >> validation. There are some ways to tweak the strictness of this with the >> supporting library, but will have to dive in a bit more. >> >> >> >> On Thu, Sep 24, 2015 at 8:14 PM, Jeff <[email protected] >> <mailto:[email protected]>> wrote: >> >> >> >> I’m having a very similar problem. The process picks up the file, a custom >> processor does it’s thing but no data is sent out. >> >> <unknown.gif> >> >> > >
