Bryan, If I have the content repo implementation set to org.apache.nifi.controller.repository.VolatileContentRepository, it will stream the content in memory, correct?
On Thu, Oct 27, 2016 at 6:22 AM, Bryan Bende <bbe...@gmail.com> wrote: > Monica, > > Are you asking what does NiFi do when it picks up a large file from the > filesystem using a processor like GetFile? > > If so, it will stream the content of that file into NiFi's content > repository, and create a FlowFile pointing to that content. As far as NiFi > is concerned the content is just bytes at this point and has not been > changed in anyway from the original file. > > The content is not held in memory, and the FlowFile can move through many > processors without ever accessing the content, unless the processor needs > to, and then when accessing the content it is typically done in a streaming > fashion (when possible) to avoid loading the large content into memory. > > There are processors that can then split up the content based on specific > data formats, for example SplitText, SplitJSON, SplitAvro, etc.. but it is > up to the designer of the flow to do that. > > -Bryan > > > On Thu, Oct 27, 2016 at 4:52 AM, Monica Franceschini < > monica.francesch...@eng.it> wrote: > >> Hi, >> I'm figuring out how does Nifi ingest large files: does it split them >> into chunks or is it a massive load?Can you please, explain the behavior? >> Kind regards, >> Monica >> -- >> >> *Monica Franceschini* >> Solution Architecture Manager >> >> *Big Data Competence Center Engineering Group* >> Corso Stati Uniti 23/C, 35127 Padova, Italia >> Tel: +39 049.8283547 >> Fax: +39 049.8692566 >> Twitter: @twittmonique >> www.spagobi.org - www.eng.it <http://www.eng.it/web/eng_en/home> *proud >> SpagoBI supporter and contributor* >> [image: SpagoBI] >> Respect the environment. Please don't print this e-mail unless you >> really need to. >> >> The information transmitted is intended only for the person or entity to >> which it is addressed and may contain confidential and/or privileged >> material. Any review, retransmission, dissemination or other use of, or >> taking of any action in reliance upon, this information by persons or >> entities other than the intended recipient is prohibited. If you received >> this in error, please contact the sender and delete the material from any >> computer. >> > > -- [image: Payoff, Inc.] <http://www.payoff.com/> Jeremy Farbota Software Engineer, Data jfarb...@payoff.com <em...@payoff.com> • (217) 898-8110 <(949)+430-0630> I'm a Storyteller. Discover your Financial Personality! <https://www.payoff.com/quiz> [image: Facebook] <https://www.facebook.com/payoff> [image: Twitter] <https://www.twitter.com/payoff> [image: Linkedin] <https://www.linkedin.com/company/payoff-com>
