Hi Adam,
I have a flow that does the following; GetFile > DDA_Processor > ConvertJSONToAvro > UpdateAttribute > PutFile My source file has 182897 rows at 1001 bytes per row. If I do any number of rows under ~15000 an output file is created. Anything over, the GetFile and DDA_Processor shows data movement but the no other downstream processor shows movement. I confirmed that it is not a data problem by processing a 10,000 row file successfully, then concatenating 10,000 rows into one file twice. Thanks for your insight. Jeff > On Sep 24, 2015, at 8:40 PM, Aldrin Piri <[email protected]> wrote: > > Jeff, > > This seems to be a bit different as the processor is showing data as having > been written and there is a listing of one FlowFile of 381 MB being > transferred out from the processor. Could you provide additional information > as to how data is not being sent out in the manner anticipated? If you can > track the issue down more, let us know. May be helpful to create another > message to help us track the issues separately as we work through them. > > Thanks! > > Adam, > > Found a sizable JSON file to work against and have been doing some initial > exploration. With the large files, it certainly is a nontrivial process. At > cursory inspection, a good portion of processing seems to be spent on > validation. There are some ways to tweak the strictness of this with the > supporting library, but will have to dive in a bit more. > > > > On Thu, Sep 24, 2015 at 8:14 PM, Jeff <[email protected] > <mailto:[email protected]>> wrote: > > > > I’m having a very similar problem. The process picks up the file, a custom > processor does it’s thing but no data is sent out. > > <unknown.gif> > >
