Re: [DISCUSS] Bringing MiNiFi into NiFi
To me this sounds like an excellent idea. A while ago I wrote some processors I wanted to use in conjunction with MiNiFi and as a consequence they needed to be built against NiFi 1.7. See https://dsmr.basjes.nl/#apache-minifi-processors With this transition I expect (like to have) the versions of both to be the same. So that a processor for NiFi version X will work on MiNiFi version X. Niels On Fri, May 22, 2020 at 5:16 PM Matt Burgess wrote: > Andy is correct, I should have been more specific in referring to > MiNiFi strictly as "MiNiFi Java" for this discussion. MiNiFi C++ would > remain its own repo with its own build processes, release cadence, > etc. > > On Thu, May 21, 2020 at 3:59 PM Andy LoPresto > wrote: > > > > Jeff, my understanding is that MiNiFi C++ development will continue > unaffected by this. While the release process may shift slightly as MiNiFi > Java and NiFi are combined, I actually think this will improve both > processes and shouldn’t negatively influence the capability to release > either in a useable format. > > > > Andy LoPresto > > alopre...@apache.org > > alopresto.apa...@gmail.com > > He/Him > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > On May 21, 2020, at 12:22 PM, Jeff Zemerick > wrote: > > > > > > Apologies if this has been discussed previously. I've not kept up on > the > > > Mi/NiFi progress as much lately. > > > > > > When I think of two projects being combined I usually consider the > > > connection between the lifecycles of the two projects. One thing I > always > > > liked about MiNiFi being separate was its ability to evolve outside the > > > NiFi lifecycle. New features, enhancements, and fixes could be > released as > > > needed. It also had its own voting process for releases. But it > certainly > > > had its drawbacks, too. > > > > > > With the two projects being combined, is there any fear of negative > effects > > > to MiNiFi's development being tied to NiFi's release cadence? Will it > be > > > possible to do a MiNiFi release outside of a NiFi release? Or any > desire to? > > > > > > I'm not at all proposing not to merge the projects because there's a > lot to > > > gain as stated in the initial email, but I would like to see MiNiFi be > able > > > to maintain some of that independence and flexibility, if possible. > > > > > > Thanks, > > > Jeff > > > > > > On Thu, May 21, 2020 at 1:54 PM Joe Witt wrote: > > > > > >> Yes please! > > >> > > >> On Thu, May 21, 2020 at 1:53 PM Andy LoPresto > > >> wrote: > > >> > > >>> Matt, > > >>> > > >>> I think this is a great idea, and I think it could also provide a > bridge > > >>> to the reduced kernel build of NiFi with separate extensions that the > > >>> extensions registry will ultimately offer. Once the MiNiFi and NiFi > > >>> assemblies are complete, it should be possible to add a third which > does > > >>> include the UI/API, but does not include the majority of specialized > > >>> processor NARs, etc. > > >>> > > >>> Andy LoPresto > > >>> alopre...@apache.org > > >>> alopresto.apa...@gmail.com > > >>> He/Him > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > >>> > > On May 21, 2020, at 10:49 AM, Pierre Villard < > > >>> pierre.villard...@gmail.com> wrote: > > > > Nothing useful to add other than I know this is going to be a lot of > > >>> work, > > but this is GREAT! > > > > Thanks, > > Pierre > > > > Le jeu. 21 mai 2020 à 19:44, Matt Burgess a > > >>> écrit : > > > > > All, > > > > > > I'm currently working on MINIFI-422, which aims to bring the MiNiFi > > > code into the NiFi codebase and have the MiNiFi product be a > > > specialized assembly of NiFi. Picture two different Maven profiles > > > that create a NiFi assembly or a MiNiFi assembly, each including > the > > > common elements but excluding those things that aren't needed, > such as > > > MiNiFi being "headless" and not including Jetty or the UI. > > > > > > This will be a lot of effort and very invasive to NiFi itself, so I > > > created a MINIFI-422 branch (based on the current master) and > pushed > > > that to the Apache NiFi Github repo. I figure that way we can carve > > > out individual tasks and write PRs against that branch, rather than > > > duplicating code in the meantime and having MiNiFi show up little > by > > > little in the NiFi codebase. > > > > > > My first task to that end is to continue Aldrin's work on > MINIFI-488. > > > I originally had a PR against master for this subtask, but after > > > discussing with Aldrin we thought it would be better to have a > > > separate branch and incrementally add MiNiFi functionality until > we're > > > ready for a big PR to bring it all into NiFi. > > > > > > We still want to keep up with NiFi master, so the branch would be > > > subject to force-pushes, rebases, etc. For that
Re: [DISCUSS] Bringing MiNiFi into NiFi
Andy is correct, I should have been more specific in referring to MiNiFi strictly as "MiNiFi Java" for this discussion. MiNiFi C++ would remain its own repo with its own build processes, release cadence, etc. On Thu, May 21, 2020 at 3:59 PM Andy LoPresto wrote: > > Jeff, my understanding is that MiNiFi C++ development will continue > unaffected by this. While the release process may shift slightly as MiNiFi > Java and NiFi are combined, I actually think this will improve both processes > and shouldn’t negatively influence the capability to release either in a > useable format. > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On May 21, 2020, at 12:22 PM, Jeff Zemerick wrote: > > > > Apologies if this has been discussed previously. I've not kept up on the > > Mi/NiFi progress as much lately. > > > > When I think of two projects being combined I usually consider the > > connection between the lifecycles of the two projects. One thing I always > > liked about MiNiFi being separate was its ability to evolve outside the > > NiFi lifecycle. New features, enhancements, and fixes could be released as > > needed. It also had its own voting process for releases. But it certainly > > had its drawbacks, too. > > > > With the two projects being combined, is there any fear of negative effects > > to MiNiFi's development being tied to NiFi's release cadence? Will it be > > possible to do a MiNiFi release outside of a NiFi release? Or any desire to? > > > > I'm not at all proposing not to merge the projects because there's a lot to > > gain as stated in the initial email, but I would like to see MiNiFi be able > > to maintain some of that independence and flexibility, if possible. > > > > Thanks, > > Jeff > > > > On Thu, May 21, 2020 at 1:54 PM Joe Witt wrote: > > > >> Yes please! > >> > >> On Thu, May 21, 2020 at 1:53 PM Andy LoPresto > >> wrote: > >> > >>> Matt, > >>> > >>> I think this is a great idea, and I think it could also provide a bridge > >>> to the reduced kernel build of NiFi with separate extensions that the > >>> extensions registry will ultimately offer. Once the MiNiFi and NiFi > >>> assemblies are complete, it should be possible to add a third which does > >>> include the UI/API, but does not include the majority of specialized > >>> processor NARs, etc. > >>> > >>> Andy LoPresto > >>> alopre...@apache.org > >>> alopresto.apa...@gmail.com > >>> He/Him > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >>> > On May 21, 2020, at 10:49 AM, Pierre Villard < > >>> pierre.villard...@gmail.com> wrote: > > Nothing useful to add other than I know this is going to be a lot of > >>> work, > but this is GREAT! > > Thanks, > Pierre > > Le jeu. 21 mai 2020 à 19:44, Matt Burgess a > >>> écrit : > > > All, > > > > I'm currently working on MINIFI-422, which aims to bring the MiNiFi > > code into the NiFi codebase and have the MiNiFi product be a > > specialized assembly of NiFi. Picture two different Maven profiles > > that create a NiFi assembly or a MiNiFi assembly, each including the > > common elements but excluding those things that aren't needed, such as > > MiNiFi being "headless" and not including Jetty or the UI. > > > > This will be a lot of effort and very invasive to NiFi itself, so I > > created a MINIFI-422 branch (based on the current master) and pushed > > that to the Apache NiFi Github repo. I figure that way we can carve > > out individual tasks and write PRs against that branch, rather than > > duplicating code in the meantime and having MiNiFi show up little by > > little in the NiFi codebase. > > > > My first task to that end is to continue Aldrin's work on MINIFI-488. > > I originally had a PR against master for this subtask, but after > > discussing with Aldrin we thought it would be better to have a > > separate branch and incrementally add MiNiFi functionality until we're > > ready for a big PR to bring it all into NiFi. > > > > We still want to keep up with NiFi master, so the branch would be > > subject to force-pushes, rebases, etc. For that reason I'd like to ask > > that anyone working on that branch please reach out to me > > (mattyb...@apache.org) so we can coordinate and collaborate, to > >> ensure > > we're not stepping on each other's toes :) > > > > Also happy to discuss alternate approaches or any other questions or > > concerns you may have. > > > > Regards, > > Matt > > > >>> > >>> > >> >
Re: [DISCUSS] Bringing MiNiFi into NiFi
Jeff, my understanding is that MiNiFi C++ development will continue unaffected by this. While the release process may shift slightly as MiNiFi Java and NiFi are combined, I actually think this will improve both processes and shouldn’t negatively influence the capability to release either in a useable format. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 21, 2020, at 12:22 PM, Jeff Zemerick wrote: > > Apologies if this has been discussed previously. I've not kept up on the > Mi/NiFi progress as much lately. > > When I think of two projects being combined I usually consider the > connection between the lifecycles of the two projects. One thing I always > liked about MiNiFi being separate was its ability to evolve outside the > NiFi lifecycle. New features, enhancements, and fixes could be released as > needed. It also had its own voting process for releases. But it certainly > had its drawbacks, too. > > With the two projects being combined, is there any fear of negative effects > to MiNiFi's development being tied to NiFi's release cadence? Will it be > possible to do a MiNiFi release outside of a NiFi release? Or any desire to? > > I'm not at all proposing not to merge the projects because there's a lot to > gain as stated in the initial email, but I would like to see MiNiFi be able > to maintain some of that independence and flexibility, if possible. > > Thanks, > Jeff > > On Thu, May 21, 2020 at 1:54 PM Joe Witt wrote: > >> Yes please! >> >> On Thu, May 21, 2020 at 1:53 PM Andy LoPresto >> wrote: >> >>> Matt, >>> >>> I think this is a great idea, and I think it could also provide a bridge >>> to the reduced kernel build of NiFi with separate extensions that the >>> extensions registry will ultimately offer. Once the MiNiFi and NiFi >>> assemblies are complete, it should be possible to add a third which does >>> include the UI/API, but does not include the majority of specialized >>> processor NARs, etc. >>> >>> Andy LoPresto >>> alopre...@apache.org >>> alopresto.apa...@gmail.com >>> He/Him >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> On May 21, 2020, at 10:49 AM, Pierre Villard < >>> pierre.villard...@gmail.com> wrote: Nothing useful to add other than I know this is going to be a lot of >>> work, but this is GREAT! Thanks, Pierre Le jeu. 21 mai 2020 à 19:44, Matt Burgess a >>> écrit : > All, > > I'm currently working on MINIFI-422, which aims to bring the MiNiFi > code into the NiFi codebase and have the MiNiFi product be a > specialized assembly of NiFi. Picture two different Maven profiles > that create a NiFi assembly or a MiNiFi assembly, each including the > common elements but excluding those things that aren't needed, such as > MiNiFi being "headless" and not including Jetty or the UI. > > This will be a lot of effort and very invasive to NiFi itself, so I > created a MINIFI-422 branch (based on the current master) and pushed > that to the Apache NiFi Github repo. I figure that way we can carve > out individual tasks and write PRs against that branch, rather than > duplicating code in the meantime and having MiNiFi show up little by > little in the NiFi codebase. > > My first task to that end is to continue Aldrin's work on MINIFI-488. > I originally had a PR against master for this subtask, but after > discussing with Aldrin we thought it would be better to have a > separate branch and incrementally add MiNiFi functionality until we're > ready for a big PR to bring it all into NiFi. > > We still want to keep up with NiFi master, so the branch would be > subject to force-pushes, rebases, etc. For that reason I'd like to ask > that anyone working on that branch please reach out to me > (mattyb...@apache.org) so we can coordinate and collaborate, to >> ensure > we're not stepping on each other's toes :) > > Also happy to discuss alternate approaches or any other questions or > concerns you may have. > > Regards, > Matt > >>> >>> >>
Re: [DISCUSS] Bringing MiNiFi into NiFi
Apologies if this has been discussed previously. I've not kept up on the Mi/NiFi progress as much lately. When I think of two projects being combined I usually consider the connection between the lifecycles of the two projects. One thing I always liked about MiNiFi being separate was its ability to evolve outside the NiFi lifecycle. New features, enhancements, and fixes could be released as needed. It also had its own voting process for releases. But it certainly had its drawbacks, too. With the two projects being combined, is there any fear of negative effects to MiNiFi's development being tied to NiFi's release cadence? Will it be possible to do a MiNiFi release outside of a NiFi release? Or any desire to? I'm not at all proposing not to merge the projects because there's a lot to gain as stated in the initial email, but I would like to see MiNiFi be able to maintain some of that independence and flexibility, if possible. Thanks, Jeff On Thu, May 21, 2020 at 1:54 PM Joe Witt wrote: > Yes please! > > On Thu, May 21, 2020 at 1:53 PM Andy LoPresto > wrote: > > > Matt, > > > > I think this is a great idea, and I think it could also provide a bridge > > to the reduced kernel build of NiFi with separate extensions that the > > extensions registry will ultimately offer. Once the MiNiFi and NiFi > > assemblies are complete, it should be possible to add a third which does > > include the UI/API, but does not include the majority of specialized > > processor NARs, etc. > > > > Andy LoPresto > > alopre...@apache.org > > alopresto.apa...@gmail.com > > He/Him > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > On May 21, 2020, at 10:49 AM, Pierre Villard < > > pierre.villard...@gmail.com> wrote: > > > > > > Nothing useful to add other than I know this is going to be a lot of > > work, > > > but this is GREAT! > > > > > > Thanks, > > > Pierre > > > > > > Le jeu. 21 mai 2020 à 19:44, Matt Burgess a > > écrit : > > > > > >> All, > > >> > > >> I'm currently working on MINIFI-422, which aims to bring the MiNiFi > > >> code into the NiFi codebase and have the MiNiFi product be a > > >> specialized assembly of NiFi. Picture two different Maven profiles > > >> that create a NiFi assembly or a MiNiFi assembly, each including the > > >> common elements but excluding those things that aren't needed, such as > > >> MiNiFi being "headless" and not including Jetty or the UI. > > >> > > >> This will be a lot of effort and very invasive to NiFi itself, so I > > >> created a MINIFI-422 branch (based on the current master) and pushed > > >> that to the Apache NiFi Github repo. I figure that way we can carve > > >> out individual tasks and write PRs against that branch, rather than > > >> duplicating code in the meantime and having MiNiFi show up little by > > >> little in the NiFi codebase. > > >> > > >> My first task to that end is to continue Aldrin's work on MINIFI-488. > > >> I originally had a PR against master for this subtask, but after > > >> discussing with Aldrin we thought it would be better to have a > > >> separate branch and incrementally add MiNiFi functionality until we're > > >> ready for a big PR to bring it all into NiFi. > > >> > > >> We still want to keep up with NiFi master, so the branch would be > > >> subject to force-pushes, rebases, etc. For that reason I'd like to ask > > >> that anyone working on that branch please reach out to me > > >> (mattyb...@apache.org) so we can coordinate and collaborate, to > ensure > > >> we're not stepping on each other's toes :) > > >> > > >> Also happy to discuss alternate approaches or any other questions or > > >> concerns you may have. > > >> > > >> Regards, > > >> Matt > > >> > > > > >
Re: [DISCUSS] Bringing MiNiFi into NiFi
Yes please! On Thu, May 21, 2020 at 1:53 PM Andy LoPresto wrote: > Matt, > > I think this is a great idea, and I think it could also provide a bridge > to the reduced kernel build of NiFi with separate extensions that the > extensions registry will ultimately offer. Once the MiNiFi and NiFi > assemblies are complete, it should be possible to add a third which does > include the UI/API, but does not include the majority of specialized > processor NARs, etc. > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On May 21, 2020, at 10:49 AM, Pierre Villard < > pierre.villard...@gmail.com> wrote: > > > > Nothing useful to add other than I know this is going to be a lot of > work, > > but this is GREAT! > > > > Thanks, > > Pierre > > > > Le jeu. 21 mai 2020 à 19:44, Matt Burgess a > écrit : > > > >> All, > >> > >> I'm currently working on MINIFI-422, which aims to bring the MiNiFi > >> code into the NiFi codebase and have the MiNiFi product be a > >> specialized assembly of NiFi. Picture two different Maven profiles > >> that create a NiFi assembly or a MiNiFi assembly, each including the > >> common elements but excluding those things that aren't needed, such as > >> MiNiFi being "headless" and not including Jetty or the UI. > >> > >> This will be a lot of effort and very invasive to NiFi itself, so I > >> created a MINIFI-422 branch (based on the current master) and pushed > >> that to the Apache NiFi Github repo. I figure that way we can carve > >> out individual tasks and write PRs against that branch, rather than > >> duplicating code in the meantime and having MiNiFi show up little by > >> little in the NiFi codebase. > >> > >> My first task to that end is to continue Aldrin's work on MINIFI-488. > >> I originally had a PR against master for this subtask, but after > >> discussing with Aldrin we thought it would be better to have a > >> separate branch and incrementally add MiNiFi functionality until we're > >> ready for a big PR to bring it all into NiFi. > >> > >> We still want to keep up with NiFi master, so the branch would be > >> subject to force-pushes, rebases, etc. For that reason I'd like to ask > >> that anyone working on that branch please reach out to me > >> (mattyb...@apache.org) so we can coordinate and collaborate, to ensure > >> we're not stepping on each other's toes :) > >> > >> Also happy to discuss alternate approaches or any other questions or > >> concerns you may have. > >> > >> Regards, > >> Matt > >> > >
Re: [DISCUSS] Bringing MiNiFi into NiFi
Matt, I think this is a great idea, and I think it could also provide a bridge to the reduced kernel build of NiFi with separate extensions that the extensions registry will ultimately offer. Once the MiNiFi and NiFi assemblies are complete, it should be possible to add a third which does include the UI/API, but does not include the majority of specialized processor NARs, etc. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 21, 2020, at 10:49 AM, Pierre Villard > wrote: > > Nothing useful to add other than I know this is going to be a lot of work, > but this is GREAT! > > Thanks, > Pierre > > Le jeu. 21 mai 2020 à 19:44, Matt Burgess a écrit : > >> All, >> >> I'm currently working on MINIFI-422, which aims to bring the MiNiFi >> code into the NiFi codebase and have the MiNiFi product be a >> specialized assembly of NiFi. Picture two different Maven profiles >> that create a NiFi assembly or a MiNiFi assembly, each including the >> common elements but excluding those things that aren't needed, such as >> MiNiFi being "headless" and not including Jetty or the UI. >> >> This will be a lot of effort and very invasive to NiFi itself, so I >> created a MINIFI-422 branch (based on the current master) and pushed >> that to the Apache NiFi Github repo. I figure that way we can carve >> out individual tasks and write PRs against that branch, rather than >> duplicating code in the meantime and having MiNiFi show up little by >> little in the NiFi codebase. >> >> My first task to that end is to continue Aldrin's work on MINIFI-488. >> I originally had a PR against master for this subtask, but after >> discussing with Aldrin we thought it would be better to have a >> separate branch and incrementally add MiNiFi functionality until we're >> ready for a big PR to bring it all into NiFi. >> >> We still want to keep up with NiFi master, so the branch would be >> subject to force-pushes, rebases, etc. For that reason I'd like to ask >> that anyone working on that branch please reach out to me >> (mattyb...@apache.org) so we can coordinate and collaborate, to ensure >> we're not stepping on each other's toes :) >> >> Also happy to discuss alternate approaches or any other questions or >> concerns you may have. >> >> Regards, >> Matt >>
Re: [DISCUSS] Bringing MiNiFi into NiFi
Nothing useful to add other than I know this is going to be a lot of work, but this is GREAT! Thanks, Pierre Le jeu. 21 mai 2020 à 19:44, Matt Burgess a écrit : > All, > > I'm currently working on MINIFI-422, which aims to bring the MiNiFi > code into the NiFi codebase and have the MiNiFi product be a > specialized assembly of NiFi. Picture two different Maven profiles > that create a NiFi assembly or a MiNiFi assembly, each including the > common elements but excluding those things that aren't needed, such as > MiNiFi being "headless" and not including Jetty or the UI. > > This will be a lot of effort and very invasive to NiFi itself, so I > created a MINIFI-422 branch (based on the current master) and pushed > that to the Apache NiFi Github repo. I figure that way we can carve > out individual tasks and write PRs against that branch, rather than > duplicating code in the meantime and having MiNiFi show up little by > little in the NiFi codebase. > > My first task to that end is to continue Aldrin's work on MINIFI-488. > I originally had a PR against master for this subtask, but after > discussing with Aldrin we thought it would be better to have a > separate branch and incrementally add MiNiFi functionality until we're > ready for a big PR to bring it all into NiFi. > > We still want to keep up with NiFi master, so the branch would be > subject to force-pushes, rebases, etc. For that reason I'd like to ask > that anyone working on that branch please reach out to me > (mattyb...@apache.org) so we can coordinate and collaborate, to ensure > we're not stepping on each other's toes :) > > Also happy to discuss alternate approaches or any other questions or > concerns you may have. > > Regards, > Matt >
[DISCUSS] Bringing MiNiFi into NiFi
All, I'm currently working on MINIFI-422, which aims to bring the MiNiFi code into the NiFi codebase and have the MiNiFi product be a specialized assembly of NiFi. Picture two different Maven profiles that create a NiFi assembly or a MiNiFi assembly, each including the common elements but excluding those things that aren't needed, such as MiNiFi being "headless" and not including Jetty or the UI. This will be a lot of effort and very invasive to NiFi itself, so I created a MINIFI-422 branch (based on the current master) and pushed that to the Apache NiFi Github repo. I figure that way we can carve out individual tasks and write PRs against that branch, rather than duplicating code in the meantime and having MiNiFi show up little by little in the NiFi codebase. My first task to that end is to continue Aldrin's work on MINIFI-488. I originally had a PR against master for this subtask, but after discussing with Aldrin we thought it would be better to have a separate branch and incrementally add MiNiFi functionality until we're ready for a big PR to bring it all into NiFi. We still want to keep up with NiFi master, so the branch would be subject to force-pushes, rebases, etc. For that reason I'd like to ask that anyone working on that branch please reach out to me (mattyb...@apache.org) so we can coordinate and collaborate, to ensure we're not stepping on each other's toes :) Also happy to discuss alternate approaches or any other questions or concerns you may have. Regards, Matt
