Thanks to everyone who has provided feedback thus far, it is good to see general interest in moving forward with a release focusing on technical debt removal.
It seems like it would be helpful to start gathering removal candidates on a Confluence wiki page, and then turning those into Jira issues. I am open to suggestions here, but it seems like creating several epics related to high level goals would be a good way to organize the work. I can start with creating a Confluence wiki page to capture some of these initial details. A release version 2.0.0 already exists in Jira, so that could be used for tagging planned issues. >From a branching and maintenance perspective, perhaps PMC members could describe how this should work? Should the main branch become the branch planned for 2.0.0 work, and does that require a vote for approval to proceed? Or should a new 2.0 branch be created and new commits applied selectively? Regards, David Handermann On Fri, Jul 23, 2021 at 11:29 AM Matt Burgess <[email protected]> wrote: > Along with the itemized list for ancient components we should look at > updating versions of drivers, SDKs, etc. for external systems such as > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0 > is probably the right time to get things up to date to make them more > useful to more people. > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <[email protected]> wrote: > > > > I'm a +1 for removing pretty much all of this stuff. There are security > > implications to keeping old dependencies around, so the more old code we > > can remove the better. I agree that eventually we need to move to > > supporting only Java 11+, and as our next release will probably be about > 4 > > - 6 months from now that doesn't seem too soon. We could potentially > break > > this in two and remove the deprecated processors and leave 1.x on Java 8, > > and finally start on 2.x which would support only Java 11. I'm unsure of > > what implications changing the date and time handling would have - for > > running systems that use long term historical logs, unexpected impacts to > > time logging could be a problem. > > > > As Joe says I think feature work will have to be dedicated to 2.x and we > > could support 1.x for security fixes for some period of time. 2.x seems > > like a gargantuan task but it's probably time to get started. Not sure > how > > we handle all open PRs and the transition between 1.x and 2.x. > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <[email protected]> wrote: > > > > > Jon > > > > > > You're right we have to be careful and you're right there are still > > > significant Java 8 users out there. But we also have to be careful > > > about security and sustainability of the codebase. If we had talked > > > about this last year when that article came out I'd have agreed it is > > > too early. Interestingly that link seems to get updated and I tried > > > [1] and found more recent data (not sure how recent). Anyway it > > > suggests Java 8 is still the top dog but we see good growth on 11. In > > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem > > > to care about Java 11 until later half last year and now suddenly it > > > is all over the place. > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in > > > work on the 1.x line just being blunt. We did this many years ago > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or > > > so) but it was purely bug fix/security related bits. We would need to > > > do something similar. But feature work would almost certainly go to > > > the 2.x line. Maybe there are other workable models but my instinct > > > suggests this is likely to follow a similar path. > > > > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We > > > need to make the call in both the interests of the user base and the > > > contributor base of the community. > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > > > > > > Thanks > > > Joe > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <[email protected]> wrote: > > > > > > > > Russ > > > > > > > > Yeah the flow registry is a key part of it. But also now you can > > > > download the flow definition in JSON (upload i think is there now > > > > too). Templates offered a series of challenges such as we store them > > > > in the flow definition which has made flows massive in an unintended > > > > way which isn't fun for cluster behavior. > > > > > > > > We have a couple cases where we headed down a particular concept and > > > > came up with better approaches later. We need to reconcile these > with > > > > the benefit of hindsight, and while being careful to be not overly > > > > disruptive to existing users, to reduce the codebase/maintenance > > > > burden and allow continued evolution of the project. > > > > > > > > Thanks > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < > [email protected]> > > > wrote: > > > > > > > > > > Joe, > > > > > > > > > > I apologize for the off-topic intrusion, but what replaces > templates? > > > > > The Registry? Templates rocked and we have used them since 0.5.x. > > > > > > > > > > Russ > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote: > > > > > > David, > > > > > > > > > > > > I think this is a highly reasonable approach and such a focus > will > > > > > > greatly help make a 2.0 release far more approachable to knock > out. > > > > > > Not only that but tech debt reduction would help make work > towards > > > > > > major features we'd think about in a 'major release' sense more > > > > > > approachable. > > > > > > > > > > > > We should remove all deprecated things (as well as verify we > have the > > > > > > right list). We should remove/consider removal of deprecated > > > concepts > > > > > > like templates. We should consider whether we can resolve the > > > various > > > > > > ways we've handled what are now parameters down to one clean > > > approach. > > > > > > We should remove options in the nifi.properties which turn out to > > > > > > never be used quite right (if there are). There is quite a bit > we > > > can > > > > > > do purely in the name of tech debt reduction. > > > > > > > > > > > > Lots to consider here but I think this is the right discussion. > > > > > > > > > > > > Than ks > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <[email protected]> > > > wrote: > > > > > >> I'm a +1 for this... Not sure if this falls under "Removing > > > Deprecated > > > > > >> Components", but I think we should also look at anything that > has > > > been > > > > > >> marked as deprecated throughout the code base as a candidate for > > > > > >> removal. There are quite a few classes, methods, properties, etc > > > that > > > > > >> have been waiting for a chance to be removed. > > > > > >> > > > > > >> On Fri, Jul 23, 2021 at 10:13 AM David Handermann > > > > > >> <[email protected]> wrote: > > > > > >>> Team, > > > > > >>> > > > > > >>> With all of the excellent work that many have contributed to > NiFi > > > over the > > > > > >>> years, the code base has also accumulated some amount of > technical > > > debt. A > > > > > >>> handful of components have been marked as deprecated, and some > > > components > > > > > >>> remain in the code base to support integration with old > versions > > > of various > > > > > >>> products. Following the principles of semantic versioning, > > > introducing a > > > > > >>> major release would provide the opportunity to remove these > > > deprecated and > > > > > >>> unsupported components. > > > > > >>> > > > > > >>> Rather than focusing the next major release on new features, > what > > > do you > > > > > >>> think about focusing on technical debt removal? This approach > > > would not > > > > > >>> make for the most interesting release, but it provides the > > > opportunity to > > > > > >>> clean up elements that involve breaking changes. > > > > > >>> > > > > > >>> Focusing on technical debt, at least three primary goals come > to > > > mind for > > > > > >>> the next major release: > > > > > >>> > > > > > >>> 1. Removal of deprecated and unmaintained components > > > > > >>> 2. Require Java 11 as the minimum supported version > > > > > >>> 3. Transition internal date and time handling to JSR 310 > java.time > > > > > >>> components > > > > > >>> > > > > > >>> *Removing Deprecated Components* > > > > > >>> > > > > > >>> Removing support for older and deprecated components provides a > > > great > > > > > >>> opportunity to improve the overall security posture when it > comes > > > to > > > > > >>> maintaining dependencies. The OWASP dependency plugin report > > > currently > > > > > >>> generates 50 MB of HTML for questionable dependencies, many of > > > which are > > > > > >>> related to old versions of various libraries. > > > > > >>> > > > > > >>> As a starting point, here are a handful of components and > > > extension modules > > > > > >>> that could be targeted for removal in a major version: > > > > > >>> > > > > > >>> - PostHTTP and GetHTTP > > > > > >>> - ListenLumberjack and the entire nifi-lumberjack-bundle > > > > > >>> - ListenBeats and the entire nifi-beats-bundle > > > > > >>> - Elasticsearch 5 components > > > > > >>> - Hive 1 and 2 components > > > > > >>> > > > > > >>> *Requiring Java 11* > > > > > >>> > > > > > >>> Java 8 is now over seven years old, and NiFi has supported > general > > > > > >>> compatibility with Java 11 for several years. NiFi 1.14.0 > > > incorporated > > > > > >>> internal improvements specifically related to TLS 1.3, which > > > allowed > > > > > >>> closing out the long-running Java 11 compatibility epic > NIFI-5174. > > > Making > > > > > >>> Java 11 the minimum required version provides the opportunity > to > > > address > > > > > >>> any lingering edge cases and put NiFi in a better position to > > > support > > > > > >>> current Java versions. > > > > > >>> > > > > > >>> *JSR 310 for Date and Time Handling* > > > > > >>> > > > > > >>> Without making the scope too broad, transitioning internal date > > > and time > > > > > >>> handling to use DateTimeFormatter instead of SimpleDateFormat > > > would provide > > > > > >>> a number of advantages. The Java Time components provide much > > > better > > > > > >>> clarity when it comes to handling localized date and time > > > representations, > > > > > >>> and also avoid the inherent confusion of java.sql.Date > extending > > > > > >>> java.util.Date. Many internal components, specifically > > > Record-oriented > > > > > >>> processors and services, rely on date parsing, leading to > > > confusion and > > > > > >>> various workarounds. The pattern formats of SimpleDateFormat > and > > > > > >>> DateTimeFormatter are very similar, but there are a few subtle > > > differences. > > > > > >>> Making this transition would provide a much better foundation > > > going forward. > > > > > >>> > > > > > >>> *Conclusion* > > > > > >>> > > > > > >>> Thanks for giving this proposal some consideration. Many of you > > > have been > > > > > >>> developing NiFi for years and I look forward to your feedback. > I > > > would be > > > > > >>> glad to put together a more formalized recommendation on > > > Confluence and > > > > > >>> write up Jira epics if this general approach sounds agreeable > to > > > the > > > > > >>> community. > > > > > >>> > > > > > >>> Regards, > > > > > >>> David Handermann > > > > > > > > >
