Re: [PROPOSAL] NiFi for Incubation
Arvind Thank you for offering to be a mentor. The proposal has been updated. Arvind Prabhakar (StreamSets) arvind at apache dot org One of the comments we have received thus far is a potential concern of whether multiple communities/projects can exist in similar problem areas. Your background in Flume which solves similar problems to NiFi is a good sign that multiple projects and communities can co-exist within the space and even provide complimentary capability. Thanks Joe On Thu, Nov 20, 2014 at 9:20 PM, Joe Witt joe.w...@gmail.com wrote: Josh, Really appreciate it and have updated the proposal. Thanks Joe On Thu, Nov 20, 2014 at 9:35 AM, Josh Elser els...@apache.org wrote: Very exciting stuff! Not presently on IPMC, but if you'd have me, I'd be happy to volunteer as a mentor. If so, I'll submit an application to join the IPMC and we can go from there. - Josh Joe Witt wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. ===
Re: [PROPOSAL] NiFi for Incubation
Very interesting project! I'm happy to help mentoring it. On 20/11/14 03:02, Joe Witt wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === From its inception, NiFi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF clearly provides a mature and effective environment for successful development as is evident across the spectrum of well-known projects. Further, NiFi depends on numerous ASF libraries and projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop, HttpClient, Jakarta and Maven. We also anticipate extensions and dependencies with several more ASF projects, including Accumulo, Avro, Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
Re: [PROPOSAL] NiFi for Incubation
Hi, On Fri, Nov 21, 2014 at 1:52 PM, Joe Witt joe.w...@gmail.com wrote: ... http://wiki.apache.org/incubator/NiFiProposal Note that AFAICS some of the nominated mentors are not members of the Incubator PMC [1] - they cannot be formal incubation mentors then. ASF members can join that PMC just by asking. ASF committers can be elected to that PMC with the support of ASF or Incubator PMC members, subject to an Incubator PMC vote, that can be discussed on the incubator private list. Others can help with mentoring activities of course - and that's a great way of eventually joining this PMC if you're not an ASF member. But they won't be listed as mentors on the proposal or podling status page. -Bertrand [1] http://people.apache.org/committers-by-project.html#incubator-pmc - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] NiFi for Incubation
Bertrand, Understood. I had noticed other proposals listing these folks as mentors but also providing their IPMC status along side. I'll do the same for now and if they join the ipmc i'll update it and if they end up not doing so will remove them. If that is not reasonable then just let me know. Thanks Joe On Fri, Nov 21, 2014 at 7:59 AM, Bertrand Delacretaz bdelacre...@apache.org wrote: Hi, On Fri, Nov 21, 2014 at 1:52 PM, Joe Witt joe.w...@gmail.com wrote: ... http://wiki.apache.org/incubator/NiFiProposal Note that AFAICS some of the nominated mentors are not members of the Incubator PMC [1] - they cannot be formal incubation mentors then. ASF members can join that PMC just by asking. ASF committers can be elected to that PMC with the support of ASF or Incubator PMC members, subject to an Incubator PMC vote, that can be discussed on the incubator private list. Others can help with mentoring activities of course - and that's a great way of eventually joining this PMC if you're not an ASF member. But they won't be listed as mentors on the proposal or podling status page. -Bertrand [1] http://people.apache.org/committers-by-project.html#incubator-pmc - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] NiFi for Incubation
Hi, On Fri, Nov 21, 2014 at 2:23 PM, Joe Witt joe.w...@gmail.com wrote: ...I had noticed other proposals listing these folks as mentors but also providing their IPMC status along side. I'll do the same for now and if they join the ipmc i'll update it and if they end up not doing so will remove them. If that is not reasonable then just let me know That's perfectly fine, thanks! -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] NiFi for Incubation
I've advised Joe to 'asterix' the would-be mentors who are not iPMC yet, so that he can proceed to a vote on the base of the ones who are sooner rather than later, and the stragglers can be formally added to the metadata once they are on the iPMC.
Re: [PROPOSAL] NiFi for Incubation
Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Witt joe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.org general@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over
Re: [PROPOSAL] NiFi for Incubation
++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Wittjoe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.orggeneral@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more
Re: [PROPOSAL] NiFi for Incubation
19, 2014 at 8:11 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Wittjoe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.orggeneral@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent
Re: [PROPOSAL] NiFi for Incubation
. Cheers! Brock On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Wittjoe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.orggeneral@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems
Re: [PROPOSAL] NiFi for Incubation
love projects that start with documentation. :) I don't think the proposal needs to include that the documentation will be posted to the NiFi wiki, since that's an implementation detail. Just say this documentation will be made available via the NiFi project's use of incubator infra. (I'll save detail for the eventual dev@ list, but you should strongly consider not using the wiki to host this documentation.) -Sean On Wed, Nov 19, 2014 at 11:27 PM, Brock Nolandbr...@cloudera.com wrote: Hi Joe, I know you've done a tremendous amount of work to make this happen so I am extremely happy this is *finally* making it's way to the incubator! I look forward to helping in anyway I can. Cheers! Brock On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Wittjoe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.orggeneral@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure
Re: [PROPOSAL] NiFi for Incubation
. We will be rapidly expanding the available documentation to cover things like installation, developer guide, frequently asked questions, best practices, and more. This documentation will be posted to the NiFi wiki at apache.org. I love projects that start with documentation. :) I don't think the proposal needs to include that the documentation will be posted to the NiFi wiki, since that's an implementation detail. Just say this documentation will be made available via the NiFi project's use of incubator infra. (I'll save detail for the eventual dev@ list, but you should strongly consider not using the wiki to host this documentation.) -Sean On Wed, Nov 19, 2014 at 11:27 PM, Brock Nolandbr...@cloudera.com wrote: Hi Joe, I know you've done a tremendous amount of work to make this happen so I am extremely happy this is *finally* making it's way to the incubator! I look forward to helping in anyway I can. Cheers! Brock On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Wittjoe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.orggeneral@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF
Re: [PROPOSAL] NiFi for Incubation
joe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.org general@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === From its inception, NiFi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF
Re: [PROPOSAL] NiFi for Incubation
Sounds exciting. I have a couple of questions: 1. Is there a code grant? I assume so, the proposal states that the project is active since 2006. What I could find [1] doesn't seem to be it. 2. What is the overlap with Apache Camel (if any)? Cheers, Hadrian [1] https://github.com/Nifi On 11/19/2014 09:02 PM, Joe Witt wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === From its inception, NiFi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF clearly provides a mature and effective environment for successful development as is evident across the spectrum of well-known projects. Further, NiFi depends on numerous ASF libraries and projects including;
Re: [PROPOSAL] NiFi for Incubation
Hello Thank you for all the feedback thus far. Sean, Jan I, I've adjusted the proposal for the goals, community, and documentation. Thanks Joe On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey bus...@cloudera.com wrote: I'm really excited to see NiFi come to the incubator; it'd be a great addition to the ASF. A few points in the proposal: == Initial Goals == One of these should be to grow the community outside of the current niche, IMHO. More on this below under orphaned projects * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. I don't think the proposal needs to include the e.g. with sub-projects part. Just noting that your goals in the incubator are to address the need to have different release cycles for core and extensions is sufficient. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. How much, if any, of this community do you expect to engage via the customary project lists once NiFi is established within the ASF? Will the project be able to leverage this established group? === Orphaned Products === Risk of orphaning is minimal. The project user and developer base is substantial, growing, and there is already extensive operational use of NiFi. Given that the established base is internal to the U.S. government, I'd encourage the podling to consider the risk of a bifurcated project should a substantial outside community fail to emerge or if those internal users should fail to engage with the outside community. You cover a related issue in your Homogenous Developers section. But I think building on the Community section of the current state to call this out as an independent issue is worthwhile. possible. This environment includes widely accessible source code repositories, published artifacts, ticket tracking, and extensive documentation. We also encourage contributions and frequent debate and hold regular, collaborative discussions through e-mail, chat rooms, and in-person meet-ups. Do you anticipate any difficulties moving these established communication mechanisms to ASF public lists? === Documentation === At this time there is no NiFi documentation on the web. However, we have extensive documentation included within the application that details usage of the many functions. We will be rapidly expanding the available documentation to cover things like installation, developer guide, frequently asked questions, best practices, and more. This documentation will be posted to the NiFi wiki at apache.org. I love projects that start with documentation. :) I don't think the proposal needs to include that the documentation will be posted to the NiFi wiki, since that's an implementation detail. Just say this documentation will be made available via the NiFi project's use of incubator infra. (I'll save detail for the eventual dev@ list, but you should strongly consider not using the wiki to host this documentation.) -Sean On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland br...@cloudera.com wrote: Hi Joe, I know you've done a tremendous amount of work to make this happen so I am extremely happy this is *finally* making it's way to the incubator! I look forward to helping in anyway I can. Cheers! Brock On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Witt joe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.org general@incubator.apache.org Subject: [PROPOSAL] NiFi
Re: [PROPOSAL] NiFi for Incubation
very, VERY cool! On Nov 19, 2014, at 9:02 PM, Joe Witt joe.w...@gmail.com wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] NiFi for Incubation
Hadrian Yes there is a Software Grant Agreement. NSA's tech transfer folks have already sent that to Apache. Given that we are coming from a closed source environment you won't find much. That is what this proposal is about though as we're working hard to change that. The github link you reference has no relationship to this project. The relationship to Apache Camel will need to be explored further as NiFi is often used in similar problem spaces (integration). Camel is really powerful in its core purpose and has an excellent community and a great deal of maturity. NiFi provides a complete dataflow application with a major focus on the user experience, graphical creation and real-time command and control of those flows. It will be interesting as we progress to see how we can best integrate with projects like Camel and I am looking forward to hearing some of the thoughts and ideas the community comes up. Thanks Joe On Thu, Nov 20, 2014 at 7:45 AM, Hadrian Zbarcea hzbar...@gmail.com wrote: Sounds exciting. I have a couple of questions: 1. Is there a code grant? I assume so, the proposal states that the project is active since 2006. What I could find [1] doesn't seem to be it. 2. What is the overlap with Apache Camel (if any)? Cheers, Hadrian [1] https://github.com/Nifi On 11/19/2014 09:02 PM, Joe Witt wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to
Re: [PROPOSAL] NiFi for Incubation
+1, good stuff... --tim On Wed, Nov 19, 2014 at 9:02 PM, Joe Witt joe.w...@gmail.com wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === From its inception, NiFi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF clearly provides a mature and effective environment for successful development as is evident across the spectrum of well-known projects. Further, NiFi depends on numerous ASF libraries and projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop, HttpClient, Jakarta and Maven. We also anticipate extensions and dependencies with several more ASF projects, including
Re: [PROPOSAL] NiFi for Incubation
Very exciting stuff! Not presently on IPMC, but if you'd have me, I'd be happy to volunteer as a mentor. If so, I'll submit an application to join the IPMC and we can go from there. - Josh Joe Witt wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === From its inception, NiFi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF clearly provides a mature and effective environment for successful development as is evident across the spectrum of well-known projects. Further, NiFi depends on numerous ASF libraries and projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop, HttpClient, Jakarta and Maven. We also anticipate extensions and dependencies with
Re: [PROPOSAL] NiFi for Incubation
On 20 November 2014 14:05, Joe Witt joe.w...@gmail.com wrote: Hadrian Yes there is a Software Grant Agreement. NSA's tech transfer folks have already sent that to Apache. Given that we are coming from a closed source environment you won't find much. That is what this proposal is about though as we're working hard to change that. The github link you reference has no relationship to this project. The relationship to Apache Camel will need to be explored further as NiFi is often used in similar problem spaces (integration). Camel is really powerful in its core purpose and has an excellent community and a great deal of maturity. NiFi provides a complete dataflow application with a major focus on the user experience, graphical creation and real-time command and control of those flows. It will be interesting as we progress to see how we can best integrate with projects like Camel and I am looking forward to hearing some of the thoughts and ideas the community comes up. Thanks for the explanation, but just to be sure, similar/overlapping projects is not a problem per se, the only real concern is if 2 communities can grow. rgds jan i. Thanks Joe On Thu, Nov 20, 2014 at 7:45 AM, Hadrian Zbarcea hzbar...@gmail.com wrote: Sounds exciting. I have a couple of questions: 1. Is there a code grant? I assume so, the proposal states that the project is active since 2006. What I could find [1] doesn't seem to be it. 2. What is the overlap with Apache Camel (if any)? Cheers, Hadrian [1] https://github.com/Nifi On 11/19/2014 09:02 PM, Joe Witt wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a
Re: [PROPOSAL] NiFi for Incubation
Josh, Really appreciate it and have updated the proposal. Thanks Joe On Thu, Nov 20, 2014 at 9:35 AM, Josh Elser els...@apache.org wrote: Very exciting stuff! Not presently on IPMC, but if you'd have me, I'd be happy to volunteer as a mentor. If so, I'll submit an application to join the IPMC and we can go from there. - Josh Joe Witt wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === From its inception, NiFi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF clearly provides a mature and effective environment for successful development as is evident across the
[PROPOSAL] NiFi for Incubation
Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === From its inception, NiFi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF clearly provides a mature and effective environment for successful development as is evident across the spectrum of well-known projects. Further, NiFi depends on numerous ASF libraries and projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop, HttpClient, Jakarta and Maven. We also anticipate extensions and dependencies with several more ASF projects, including Accumulo, Avro, Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others. This existing alignment with Apache and the desired community makes the Apache Incubator a good fit for
Re: [PROPOSAL] NiFi for Incubation
Very excited to see this proposal! Michael Ridley Sent from my mobile. Pardon any spelling errors. On Nov 19, 2014, at 8:02 PM, Joe Witt joe.w...@gmail.com wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === From its inception, NiFi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF clearly provides a mature and effective environment for successful development as is evident across the spectrum of well-known projects. Further, NiFi depends on numerous ASF libraries and projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop, HttpClient, Jakarta and Maven. We also
Re: [PROPOSAL] NiFi for Incubation
This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Witt joe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.org general@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over
Re: [PROPOSAL] NiFi for Incubation
Hi Joe, I know you've done a tremendous amount of work to make this happen so I am extremely happy this is *finally* making it's way to the incubator! I look forward to helping in anyway I can. Cheers! Brock On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Witt joe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.org general@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status
Re: [PROPOSAL] NiFi for Incubation
I'm really excited to see NiFi come to the incubator; it'd be a great addition to the ASF. A few points in the proposal: == Initial Goals == One of these should be to grow the community outside of the current niche, IMHO. More on this below under orphaned projects * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. I don't think the proposal needs to include the e.g. with sub-projects part. Just noting that your goals in the incubator are to address the need to have different release cycles for core and extensions is sufficient. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. How much, if any, of this community do you expect to engage via the customary project lists once NiFi is established within the ASF? Will the project be able to leverage this established group? === Orphaned Products === Risk of orphaning is minimal. The project user and developer base is substantial, growing, and there is already extensive operational use of NiFi. Given that the established base is internal to the U.S. government, I'd encourage the podling to consider the risk of a bifurcated project should a substantial outside community fail to emerge or if those internal users should fail to engage with the outside community. You cover a related issue in your Homogenous Developers section. But I think building on the Community section of the current state to call this out as an independent issue is worthwhile. possible. This environment includes widely accessible source code repositories, published artifacts, ticket tracking, and extensive documentation. We also encourage contributions and frequent debate and hold regular, collaborative discussions through e-mail, chat rooms, and in-person meet-ups. Do you anticipate any difficulties moving these established communication mechanisms to ASF public lists? === Documentation === At this time there is no NiFi documentation on the web. However, we have extensive documentation included within the application that details usage of the many functions. We will be rapidly expanding the available documentation to cover things like installation, developer guide, frequently asked questions, best practices, and more. This documentation will be posted to the NiFi wiki at apache.org. I love projects that start with documentation. :) I don't think the proposal needs to include that the documentation will be posted to the NiFi wiki, since that's an implementation detail. Just say this documentation will be made available via the NiFi project's use of incubator infra. (I'll save detail for the eventual dev@ list, but you should strongly consider not using the wiki to host this documentation.) -Sean On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland br...@cloudera.com wrote: Hi Joe, I know you've done a tremendous amount of work to make this happen so I am extremely happy this is *finally* making it's way to the incubator! I look forward to helping in anyway I can. Cheers! Brock On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Witt joe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.org general@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based
Re: [PROPOSAL] NiFi for Incubation
Hi Joe. I would really like to welcome this project at ASF. I have one slight concern, could you be a bit more specific about the diversity of the initial committers ? I could read the proposal as if they come from one or more very related entities. I would also stresss in initial goals, that the project should get committers from outside of the circles you mention, to ensure diversity in the project. rgds jan i On 20 November 2014 06:27, Brock Noland br...@cloudera.com wrote: Hi Joe, I know you've done a tremendous amount of work to make this happen so I am extremely happy this is *finally* making it's way to the incubator! I look forward to helping in anyway I can. Cheers! Brock On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Joe Witt joe.w...@gmail.com Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Thursday, November 20, 2014 at 3:02 AM To: general@incubator.apache.org general@incubator.apache.org Subject: [PROPOSAL] NiFi for Incubation Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near
