Re: [DISCUSS] roadmap for the next 6-12 months

[Joe Witt]

Team,

I had a lengthy conversation today with Matt Gilman to talk through
specifics of the Authorizer API concept.  The ideas are captured here
https://cwiki.apache.org/confluence/display/NIFI/Support+Authorizer+API.
Definitely would like to hear ideas from people experienced in working
with centralized authority providers.

This will be an important step to making Multi-tenant authorization possible.

Thanks
Joe

On Fri, Jan 8, 2016 at 11:25 AM, Tony Kurc  wrote:
> Joe! Looks amazing! Thanks for pulling this together
> Hello NiFi Community
>
> Please review the proposed roadmap and suggest things to add, remove,
> update, prioritize differently.
>
> Over the past year we have had great contributions that improved the
> quality of NiFi and the community.  As we know there are many JIRAs
> related to improvements or new features.  However, it can be difficult
> for new and existing community members to know when to jump in, how to
> help, or when things they are interested in might be able to happen.
> So among other things we should provide and maintain a documented
> roadmap to help this.
>
> The following suggested timeline follows pretty strictly against our
> previously stated objective of having releases roughly every 6 weeks.
> We do not need to stay strict to this but a time bound (as opposed to
> feature bound) release cycle does help provide a regular cadence to
> follow and helps give contributors a sense of when their efforts might
> end up in a release.
>
> The items below are largely just focused on some of the major muscle
> movements needed at a framework level.  There will of course be many
> many more JIRAs with each releas.
>
> NiFi 0.5.0 (Feb 5)
>
>   State Management
> https://cwiki.apache.org/confluence/display/NIFI/State+Management
>   Kerberos
>
> https://cwiki.apache.org/confluence/display/NIFI/Pluggable+Authentication
>   Interactive Queue
>
> https://cwiki.apache.org/confluence/display/NIFI/Interactive+Queue+Management
>   Scripting Support
> https://issues.apache.org/jira/browse/NIFI-210
>
> NiFi 0.6.0 (Mar 18)
>
>   Deterministic Template Export
> https://issues.apache.org/jira/browse/NIFI-826
>   Schema/Format Editor (JSON to JSON,…?)
> Numerous JIRAs related to format/schema conversion.  An example
> implementation:
>
> https://github.com/fsauer65/NiFi-Extensions/tree/master/nifi-jsontransform-bundle
>
> NiFi 0.7.0 (Apr 29)
>
> NiFi 1.0.0 (Jun 17)
>
>   HA Cluster Management
> https://cwiki.apache.org/confluence/display/NIFI/Clustering+Redesign
>   HA Data
>
> https://cwiki.apache.org/confluence/display/NIFI/High+Availability+Processing
>   Multi-Tenant Authorization
> https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow
>   Improved Authorization API
>
> https://cwiki.apache.org/confluence/display/NIFI/Pluggable+Authentication
> NiFi's current authority provider gathers data then makes a decision.
> We should instead support PDP/PEP model [4].
>   Modernized UX
> https://cwiki.apache.org/confluence/display/NIFI/Redesign+User+Interface
>   Dependent Properties
> https://issues.apache.org/jira/browse/NIFI-1121
>
> NiFi 1.1.0 (Sep?)
>
>   Template/Extension Registry
> https://cwiki.apache.org/confluence/display/NIFI/Extension%2C+Template%2C+Dataset+Registry
>   Variable Registry
> https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>
> The timing of these things is pretty important as far as
> interdependencies and how long they'll take to develop and validate.
> State management in 050 sets up an important way to have cluster-wide
> shared state and introduces zookeeper formally into NiFi which
> obviously will be important for the distributed systems behaviors
> we'll need as we restructure how clustering works and introduce data
> replication for HA data processing.  The 060 and 070 releases are
> intentionally quite light to allow for very focused work on a parallel
> branch of 1.x which probably should start right on the heels of 050.
> The 1.0 release will introduce some very key features, offer a chance
> to remove deprecated classes, ensure all APIs are extremely clearly
> documented for stability and audience, allow us to introduce UI/UX
> changes, and so on.
>
> Now, of course this is simply a proposal and these timings may not
> work out.  But this should serve as a good basis for discussion and
> consensus forming.
>
> As we progress I plan to update the roadmap [1], product requirements
> [2], and feature proposals [3] to reflect these things a bit more
> clearly than they are now.
>
> We should also have a discussion on how long we should be committed to
> supporting the 0.x line and what that means.  We need to document a
> commitment for the community.
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851850
> [2] https://cwiki.apache.org/confluence/display/NIFI/Product+requirements
> [3] 

Re: [DISCUSS] roadmap for the next 6-12 months

[Sean Busbey]

On Fri, Jan 8, 2016 at 8:29 AM, Joe Witt  wrote:
> We should also have a discussion on how long we should be committed to
> supporting the 0.x line and what that means.  We need to document a
> commitment for the community.


Worth a dedicated thread?

Presuming that we're going to use "1.0.0" as the start of a stricter
versioning policy, I think the first issue is deciding how long we'll
support major versions once that happens. If the expectation is that
"pre-1.0" versions involve more latitude as we work out what NiFi
needs to better find a fit for the public good, then I think it's
reasonable for our continuing support period to be less than a
'normal' major version. But we have to know what that looks like
first.

-- 
Sean

Re: [DISCUSS] roadmap for the next 6-12 months

[Tony Kurc]

Joe! Looks amazing! Thanks for pulling this together
Hello NiFi Community

Please review the proposed roadmap and suggest things to add, remove,
update, prioritize differently.

Over the past year we have had great contributions that improved the
quality of NiFi and the community.  As we know there are many JIRAs
related to improvements or new features.  However, it can be difficult
for new and existing community members to know when to jump in, how to
help, or when things they are interested in might be able to happen.
So among other things we should provide and maintain a documented
roadmap to help this.

The following suggested timeline follows pretty strictly against our
previously stated objective of having releases roughly every 6 weeks.
We do not need to stay strict to this but a time bound (as opposed to
feature bound) release cycle does help provide a regular cadence to
follow and helps give contributors a sense of when their efforts might
end up in a release.

The items below are largely just focused on some of the major muscle
movements needed at a framework level.  There will of course be many
many more JIRAs with each releas.

NiFi 0.5.0 (Feb 5)

  State Management
https://cwiki.apache.org/confluence/display/NIFI/State+Management
  Kerberos

https://cwiki.apache.org/confluence/display/NIFI/Pluggable+Authentication
  Interactive Queue

https://cwiki.apache.org/confluence/display/NIFI/Interactive+Queue+Management
  Scripting Support
https://issues.apache.org/jira/browse/NIFI-210

NiFi 0.6.0 (Mar 18)

  Deterministic Template Export
https://issues.apache.org/jira/browse/NIFI-826
  Schema/Format Editor (JSON to JSON,…?)
Numerous JIRAs related to format/schema conversion.  An example
implementation:

https://github.com/fsauer65/NiFi-Extensions/tree/master/nifi-jsontransform-bundle

NiFi 0.7.0 (Apr 29)

NiFi 1.0.0 (Jun 17)

  HA Cluster Management
https://cwiki.apache.org/confluence/display/NIFI/Clustering+Redesign
  HA Data

https://cwiki.apache.org/confluence/display/NIFI/High+Availability+Processing
  Multi-Tenant Authorization
https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow
  Improved Authorization API

https://cwiki.apache.org/confluence/display/NIFI/Pluggable+Authentication
NiFi's current authority provider gathers data then makes a decision.
We should instead support PDP/PEP model [4].
  Modernized UX
https://cwiki.apache.org/confluence/display/NIFI/Redesign+User+Interface
  Dependent Properties
https://issues.apache.org/jira/browse/NIFI-1121

NiFi 1.1.0 (Sep?)

  Template/Extension Registry
https://cwiki.apache.org/confluence/display/NIFI/Extension%2C+Template%2C+Dataset+Registry
  Variable Registry
https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

The timing of these things is pretty important as far as
interdependencies and how long they'll take to develop and validate.
State management in 050 sets up an important way to have cluster-wide
shared state and introduces zookeeper formally into NiFi which
obviously will be important for the distributed systems behaviors
we'll need as we restructure how clustering works and introduce data
replication for HA data processing.  The 060 and 070 releases are
intentionally quite light to allow for very focused work on a parallel
branch of 1.x which probably should start right on the heels of 050.
The 1.0 release will introduce some very key features, offer a chance
to remove deprecated classes, ensure all APIs are extremely clearly
documented for stability and audience, allow us to introduce UI/UX
changes, and so on.

Now, of course this is simply a proposal and these timings may not
work out.  But this should serve as a good basis for discussion and
consensus forming.

As we progress I plan to update the roadmap [1], product requirements
[2], and feature proposals [3] to reflect these things a bit more
clearly than they are now.

We should also have a discussion on how long we should be committed to
supporting the 0.x line and what that means.  We need to document a
commitment for the community.

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851850
[2] https://cwiki.apache.org/confluence/display/NIFI/Product+requirements
[3] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
[4] https://en.wikipedia.org/wiki/Common_Open_Policy_Service

Thanks
Joe