Re: Persisting/loading NIFI flows for Docker deployment

2020-04-13 Thread Andy LoPresto
Thanks for that context and perspective, Kevin. Good luck on your project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 13, 2020, at 6:07 AM, Kevin Telford wrote: > > Hi all - thank you for

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-13 Thread Kevin Telford
Hi all - thank you for all the thoughtful feedback. Regarding my original question, I think the patterns Mike outlined would be good enough. That said, we're not going to move forward using NiFi for the project, and I figured I'd take a step back to explain where we were coming from, as some

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Mike Thomsen
Either way. You can just use "docker cp" to copy the file out of a running instance. On Wed, Apr 8, 2020 at 12:36 PM Kevin Telford wrote: > Thank you Mike. This makes sense. > > How do you get the flow.xml.gz file? Do you have NiFi installed locally on > bare metal or do you develop also in a

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Endre Kovacs
Hi, Why not use NiFi Registry to deploy / version the flows? Best regards, Endre Sent from ProtonMail mobile Original Message On Apr 8, 2020, 4:27 PM, Kevin Telford wrote: > Hi all – I have a two part question. > > I’d like to run NiFi inside a container in order to deploy to

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Otto Fowler
No, that is what I mean, I would guess that you could spin up _n_ number of nifi nodes with registry parameters that tell it what to load etc so you wouldn’t have to manage the flow on disk On April 8, 2020 at 14:21:52, Chris Sampson ( chris.samp...@naimuri.com.invalid) wrote: If you mean

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Chris Sampson
If you mean having a way of telling a new nifi instance about a registry, bucket and flow to load at startup, then I'd say yes that would be good. Or even to be able to directly import the flow json that can currently be exported from registry (or nifi). Is this effectively what NiFi serverless

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Otto Fowler
Can I ask a silly question? Would the ability for nodes to load flows from the registry make this easier? Would that even make sense? On April 8, 2020 at 12:52:19, Chris Sampson ( chris.samp...@naimuri.com.invalid) wrote: When first starting with NiFi (I've always used it in container form),

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Chris Sampson
When first starting with NiFi (I've always used it in container form), I looked at taking a similar approach with the flow.xml.gz, but quickly moved away from that after reading various tales online about it not always being very portable and/or people having issues keeping it in sync (remembering

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Kevin Telford
Thank you Mike. This makes sense. How do you get the flow.xml.gz file? Do you have NiFi installed locally on bare metal or do you develop also in a container? Initially I was thinking of adding a custom volume to facilitate both getting and deploying the flow file. Best, Kevin On 2020/04/08

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Pierre Villard
The NiFi Registry is another option to deploy versioned flows in a running NiFi container. It would require to deploy the container, and make some API calls to deploy the flow. Pierre Le mer. 8 avr. 2020 à 16:38, Mike Thomsen a écrit : > What I've done in the past looks something like this: >

Re: Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Mike Thomsen
What I've done in the past looks something like this: FROM apache/nifi:1.11.4 COPY flow.xml.gz /opt/nifi/nifi-current/conf/flow.xml.gz And that's it. The obvious caveat is that you need to follow good practices with ensuring that your flow and the way you setup the container can replicate the

Persisting/loading NIFI flows for Docker deployment

2020-04-08 Thread Kevin Telford
Hi all – I have a two part question. I’d like to run NiFi inside a container in order to deploy to various environments. As far as I can tell, the flow.xml.gz file is the main “source” if you will, for a NiFi data flow. Q1) Is the flow.xml.gz file the “source” of a NiFi data flow, and if so,