I think Jeremy is using Volatile specifically because he does *not* want that data ever persisted to disk for compliance purposes.
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 27, 2016, at 8:59 AM, Joe Witt <[email protected]> wrote: > > I should add that if you're comfortable with that sort of volatile behavior a > better path to consider is to setup a RAM-Disk and just run a persistent > content repository on that. It will survive process restarts, give better > memory/heap behavior (by a lot), but you'll lose data on system restarts. > > Thanks > Joe > > On Thu, Oct 27, 2016 at 11:58 AM, Joe Witt <[email protected] > <mailto:[email protected]>> wrote: > That is correct. > > Thanks > Joe > > On Thu, Oct 27, 2016 at 11:55 AM, Jeremy Farbota <[email protected] > <mailto:[email protected]>> wrote: > Bryan, > > If I have the content repo implementation set to > org.apache.nifi.controller.repository.VolatileContentRepository, it will > stream the content in memory, correct? > > On Thu, Oct 27, 2016 at 6:22 AM, Bryan Bende <[email protected] > <mailto:[email protected]>> wrote: > Monica, > > Are you asking what does NiFi do when it picks up a large file from the > filesystem using a processor like GetFile? > > If so, it will stream the content of that file into NiFi's content > repository, and create a FlowFile pointing to that content. As far as NiFi is > concerned the content is just bytes at this point and has not been changed in > anyway from the original file. > > The content is not held in memory, and the FlowFile can move through many > processors without ever accessing the content, unless the processor needs to, > and then when accessing the content it is typically done in a streaming > fashion (when possible) to avoid loading the large content into memory. > > There are processors that can then split up the content based on specific > data formats, for example SplitText, SplitJSON, SplitAvro, etc.. but it is up > to the designer of the flow to do that. > > -Bryan > > > On Thu, Oct 27, 2016 at 4:52 AM, Monica Franceschini > <[email protected] <mailto:[email protected]>> wrote: > Hi, > I'm figuring out how does Nifi ingest large files: does it split them into > chunks or is it a massive load?Can you please, explain the behavior? > Kind regards, > Monica > -- > Monica Franceschini > Solution Architecture Manager > > Big Data Competence Center > Engineering Group > Corso Stati Uniti 23/C, 35127 Padova, Italia > Tel: +39 049.8283547 <tel:%2B39%20049.8283547> > Fax: +39 049.8692566 <tel:%2B39%20049.8692566> > Twitter: @twittmonique > www.spagobi.org <http://www.spagobi.org/> - www.eng.it > <http://www.eng.it/web/eng_en/home> proud SpagoBI supporter and > contributor > <spagobi.jpg> > <Green.jpg> Respect the environment. Please don't print this e-mail > unless you really need to. > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of, or > taking of any action in reliance upon, this information by persons or > entities other than the intended recipient is prohibited. If you received > this in error, please contact the sender and delete the material from any > computer. > > > > > > -- > <http://www.payoff.com/> > Jeremy Farbota > Software Engineer, Data > [email protected] <mailto:[email protected]> • (217) 898-8110 > <tel:(949)+430-0630> > I'm a Storyteller. Discover your Financial Personality! > <https://www.payoff.com/quiz> > <https://www.facebook.com/payoff> <https://www.twitter.com/payoff> > <https://www.linkedin.com/company/payoff-com> >
signature.asc
Description: Message signed with OpenPGP using GPGMail
