+1 to all subjects addressed Joe. I have been planning to improve developer advocacy around scripting components. The groovy scripts have been one of my best tools to build flows in NiFi (Rest API development , complex data structures transformation, controlled flows scheduling…..). I’ll take this opportunity to ask Mat Burges whether I can contribute some of my work to http://funnifi.blogspot.com/?m=1.
Mathew On Sat, 15 Jun 2024 at 21:03, Joe Witt <joew...@apache.org> wrote: > Team, > > We are seeing an increase in the unique contributors on a monthly basis to > NiFi and a clear and steady increase in the overall contribution rate over > the past 1-2 years in particular. This is awesome but we do have important > problems to cover and we need to make it easier to help newer contributors > know where to jump in and add the most value. > > For sure people will contribute various things just because they have a > particular interest such as some new extension, etc.. But from a community > direction perspective there are things we need to continue to improve upon > and we want to help make the path to committership and PMC membership more > clear as a function of adding value to the community. > > So not necessarily ordered, here are some key things I've observed that > need more attention and are among the most valuable things for the health > of the nifi system and community. > > Dependency Management > > This is a hugely important area as we have a massive amount of dependencies > and this work is just plain not that fun. But if we want to present a > beautiful landscape we need to pull weeds. We need to reduce dependencies > and keep things recent. This is largely accomplished today by a very small > group of people. We have to increase the contribution level here or > perhaps the automation level. We have tooling that is helping now but > still needs more attention. > > Vulnerability Management particular as it relates to dependencies > > Related to dependency management but broader. This one at times requires > us to do more than simply maintain dependencies. It sometimes forces us to > change libraries and sometimes even makes maintaining > compatibility/configuration complicated. These are among the most critical > and most difficult tasks to take on. This is largely done today by an > extremely small group of people. > > Maintenance of extensions > > We have a ton of components in NiFi. Obviously not all of them really do > require active maintenance but there is often a contribute and forget model > involved. For us to keep these broad range of components under community > care they need active maintenance. I already mentioned the dependency > factor above but the broader point here is that APIs/versions of systems > evolve. We need to be staying up to date. There are systems like Kafka, > Elastic, Mongo, Relational Databases, etc.. that we should always be > supporting the latest on. These are among the most wildly popular sources > and destinations. We do tend to fall behind here and this is powerfully > useful for people to contribute to. These are the things our users > actually use NiFi for! > > Maintenance of things other than core nifi > > If you look at components ancillary to NiFi they receive far less attention > and effort trending toward them going away. The NiFi registry is a great > example of this as NiFi MiNiFi Java for instance. These are things which > are useful and indeed there is a decent user base in both cases. But they > require more active maintenance than they get. We need to evolve when it > comes to security methods offered for authn/authz and that takes focus. > > Performance analysis and improvements targeted as a result > > This is done very rarely as far as I can tell outside a couple of people > who I'm sure we could all name in a few seconds. The codebase is large and > Java and other parts evolve. Continually emphasizing faster and more > efficient operations is vital to NiFi's continued growth in the > community at large. People who spend time running NiFi in a measured > manner with realistic flows and doing things like profiling for CPU, > memory, etc.. and finding bottlenecks and offering solutions - absolutely > can be high impact and valuable. Even if not at a point where a confident > contribution can be made, the very act of doing this level of evaluation > and flagging concerns is valuable. > > Improvements related to tests both unit and integration - but also for > users of NiFi > > As part of our push to NiFi 2.0 we have been running these tests far more > often and more complete than ever before. Github Actions now pretty much > does run all the things. We've deleted a wild amount of junk tests that > would always fail or were spurious or required some complex local > access/setup and therefore nobody ever ran them. We need to always improve > coverage but we want to make tests faster/more efficient and more > leveraging of things like test containers would be highly valuable. > Importantly though here I also mean to highlight we need to do a lot more > for users of NiFi that want to create automated tests for their flows. We > should make it easy for them to provide a flow definition or perhaps a DSL > for their flow and write tests which would leverage actual nifi to validate > a given input results in an expected output. This is something that would > add a ton of value for our user base but also lead into some really > powerful testing we can do against our own framework. > > Development of blogs/videos/conference presentations/social media posts > > Building great software by itself doesn't grow the community nearly as much > as if you build a strong developer advocacy game. People that create these > various forms of blogs/videos and conference talks about NiFi, how it > works, and how to make it better are extremely valuable. That is often > just as valuable or even more so than a PR review, code contribution, etc.. > > Engagement in the Slack/Mailing lists > > Our community needs help. They ask a lot of questions about how to use > NiFi, bugs they are hitting, etc.. Responding to these questions in a > timely and meaningful manner is hugely valuable to them and is also pretty > fun for the person who helps out. > > Meaningful engagement acting as a pull request reviewer > > This one is last but certainly not least! We have a review-then-commit > model which means we need at least one committer or PMC member to review a > contribution before it gets merged. Even if you're not a committer or PMC > member, a review is still super helpful and can save time and also puts you > on a great path to earning merit. Reviews are a critical part of the code > and overall quality of NiFi. 'LGTM' is generally not that helpful and > while warranted for some trivial things we even see it on larger more > meaningful commits. Not every review needs to be a back and forth affair > of code style inputs either. There is a middle ground and you'll find it > as you engage in more reviews! Code reviews are in far more short supply > than code contributions. I encourage everyone to please do at least one > true pull request review for every contribution. In general the goal > should be to contribute to more pull request reviews than new pull requests > created. If you do so this by definition means the community is growing > and you're directly helping make that happen! > > Would love to see additional inputs/missing categories/other thoughts. > Perhaps this turns into a wiki page that helps give more context to people > that want to find ways to add value and be on a path to committer and PMC > membership as well. > > Thanks > Joe >
