matt i'll add a wiki page or set of instructions linked from the release notes.
all, walked through the 250 or so JIRAs in the 1.1.0 release and pulled out highlights. The items noted are as follows. Will likely reduce this down further for the release notes but wanted to put this out in case folks have things they think are really important to highlight. - Core Improvements: - Performance: Session Migration - Stability: Cluster Management - Developer: Framework supports easy user driven classloader extension - Expression Language: Now supports base64 and hex encoded values and Math functions - Repositories now support rollback - Faster startup due to more efficient state restoration algorithm - UX Improvements: - Visual Backpressure Indicator - Introduced more colors to better highlight actions and components - Performance: Validate non-running components - Provenance graph image can be exported - Cron Scheduling for Primary node tasks now supported - Updated versions - Azure Event Hub 0.9.0 - Spark 2.0.1 - Hadoop 2.7.x - New/Improved Processors - new Fetch/Put Elastic Search 5.0 - new ParseCEF to parse CEF formatted logs - improve ExtractEmail now supports TNEF files - new Validate CSV - improved Solr processors now support SSL and Kerberos - new Websocket client and server processors - New Utility - Zookeeper Migrator (move from one zookeeper to another) - Security - Restricted Processors - Site-to-site now supports port forwarding - Improved Policy Management UX - Migration Notes: - Restricted Processors - Twitter Processor Removed On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess <mattyb...@apache.org> wrote: > Is there a good spot for us to put instructions on how to build the > Twitter processor and/or the Social Media NAR in the meantime? Maybe a > Wiki page or something simple to say "go to this directory, run this > Maven command, drop the NAR into your deployment..." ? > > On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt <joe.w...@gmail.com> wrote: >> Team, >> >> We appear to be very close. Andy is working NIFI-3024 but otherwise >> it is focus on testing. >> >> I'm going to prep the RC and release notes now. Unfortunately the >> twitter changes for json.org will need to remain. Consensus forming >> on the legal-discuss thread regarding a grace period has been elusive >> and we're already prepared to make the right steps so we'll just need >> to take that on by being empathetic to the user base. >> >> Thanks >> Joe >> >> On Thu, Nov 17, 2016 at 7:37 AM, Andre <andre-li...@fucs.org> wrote: >>> Andy, >>> >>> Great to see NIFI-3050 implemented and certainly good news that NiFi 1.1.0 >>> is set to include a number of security related improvements. >>> >>> >>> >>> On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto <alopre...@apache.org> wrote: >>> >>>> Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have been >>>> added to my plate for this release. Coordinated with Joe Witt and they >>>> should both be included. >>>> >>>> [1] https://issues.apache.org/jira/browse/NIFI-3050 >>>> [2] https://issues.apache.org/jira/browse/NIFI-3051 >>>> >>>> Andy LoPresto >>>> alopre...@apache.org >>>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* >>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>> >>>> On Nov 16, 2016, at 12:08 PM, Joe Witt <joe.w...@gmail.com> wrote: >>>> >>>> Team >>>> >>>> There is a thread on apache legal-discuss that might allow for a >>>> graceperiod of continued usage of the json library. Am going to keep >>>> a close eye on this and if VP Legal approves we'll be able to keep the >>>> twitter processors in which is definitely a good thing. Will advise >>>> >>>> Thanks >>>> Joe >>>> >>>> On Wed, Nov 16, 2016 at 10:37 AM, Bryan Bende <bbe...@gmail.com> wrote: >>>> >>>> I've noticed an issue with the per-instance class loading capability >>>> introduced in NIFI-2909 where the additional classpath resources can get >>>> incorrectly removed from the class loader. >>>> >>>> I was able to reproduced this with a unit test and have a fix ready. I >>>> believe this is important and needs to go in for the 1.1 release, going to >>>> re-open NIFI-2909 and submit a PR shortly. >>>> >>>> -Bryan >>>> >>>> On Wed, Nov 16, 2016 at 8:11 AM, Matt Gilman <matt.c.gil...@gmail.com> >>>> wrote: >>>> >>>> I have two items that I would like to wrap up prior to creating an RC for >>>> 1.1.0. NIFI-2949 addresses some UX issues around Remote Process Group port >>>> configuration. The work is already completed and I will be reviewing it >>>> this today. Additionally, following recent interest on the mailing list, >>>> I'd like to knock out NIFI-3020. This will allow an admin to configure a >>>> strategy for user identity when logging in via LDAP. Specifically, it will >>>> support usage of the DN (the default and current implementation) as well as >>>> the username the user logged in as. I should be able to have a PR up for >>>> this work later today. >>>> >>>> Thanks! >>>> >>>> Matt >>>> >>>> [1] https://issues.apache.org/jira/browse/NIFI-2949 >>>> [2] https://issues.apache.org/jira/browse/NIFI-3020 >>>> >>>> >>>> On Tue, Nov 15, 2016 at 8:00 PM, Joe Witt <joe.w...@gmail.com> wrote: >>>> >>>> The code is within the twitter4j library itself. I filed a request to >>>> twitter4jg. The most likely case is we will need to submit a PR to them. >>>> However, I don't see this as something that should delay the release. We >>>> can provide instructions for folks wanting to use the processor during >>>> >>>> the >>>> >>>> time we cannot make it available in a convenient manner. I will provide >>>> >>>> a >>>> >>>> meaningful comment about this in release notes and pointers on what folks >>>> can do in the meantime. >>>> >>>> On Nov 15, 2016 7:41 PM, "Andy LoPresto" <alopre...@apache.org> wrote: >>>> >>>> I understand there was a discussion thread within the NiFi community >>>> >>>> for >>>> >>>> this as well and I missed responding to that at that time. It just >>>> >>>> seems >>>> >>>> to >>>> >>>> me like JSON processing is necessary for GetTwitter, which is >>>> >>>> incredibly >>>> >>>> useful for demonstrating NiFi’s ability to read from a high volume >>>> >>>> stream >>>> >>>> out of the box. With NIFI-3019 (Remove GetTwitter from default build), >>>> >>>> is >>>> >>>> there any related effort to substitute an acceptable replacement JSON >>>> library to restore this functionality? >>>> >>>> [1] https://issues.apache.org/jira/browse/NIFI-3019 >>>> >>>> Andy LoPresto >>>> alopre...@apache.org >>>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* >>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>> >>>> On Nov 15, 2016, at 4:36 PM, Andy LoPresto <alopre...@apache.org> >>>> >>>> wrote: >>>> >>>> >>>> I’m working with Bryan Rosander to close out NIFI-3024, NIFI-2655, and >>>> NIFI-2653. I believe Matt Burgess is working on NIFI-3011 and we >>>> investigated some alternate TLS config options for the new version of >>>> >>>> the >>>> >>>> client library. >>>> >>>> Is there any alternative to excluding the GetTwitter processor? Using >>>> Johnzon [1] or the Android re-implementation [2] discussed in the >>>> >>>> mailing >>>> >>>> list thread? >>>> >>>> [1] https://johnzon.apache.org/ >>>> [2] https://developer.android.com/reference/org/json/package- >>>> >>>> summary.html >>>> >>>> >>>> >>>> Andy LoPresto >>>> alopre...@apache.org >>>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* >>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>> >>>> On Nov 15, 2016, at 3:58 PM, Joe Witt <joe.w...@gmail.com> wrote: >>>> >>>> Team >>>> >>>> Very happy to see that we are down to three items remaining tagged to >>>> 1.1.0. Solid effort over the recent weeks to close the gap including >>>> >>>> work >>>> >>>> to get past the now category x Jason dependency we had. The most >>>> >>>> notable >>>> >>>> impact from that is the wildly popular GetTwitter processor, the fav >>>> >>>> new >>>> >>>> nifi user and demo processor, can no longer be included in the default >>>> build. It is optionally available if users choose to build and use it >>>> >>>> but >>>> >>>> we won't distribute binaries that have it. >>>> >>>> I see some review movement on some patch available but untagged items. >>>> >>>> I plan to kick off the 1.1.0 rc work soon. Perhaps Thurs or Fri. Anyone >>>> have any outstanding items? >>>> >>>> Thanks >>>> Joe >>>> >>>> On Nov 8, 2016 2:12 PM, "Joe Witt" <joe.w...@gmail.com> wrote: >>>> >>>> Ryan >>>> >>>> Not officially but I think we should try to close this thing out and >>>> start a vote in the next week or two at most. >>>> >>>> I'm going through the tickets again now. There is also a new issue of >>>> the json-p license falling out of favor in Apache legal terms and >>>> becoming Category-X. Am looking into that now. >>>> >>>> Thanks >>>> Joe >>>> >>>> On Tue, Nov 8, 2016 at 2:05 PM, Ryan Ward <ryan.wa...@gmail.com> >>>> >>>> wrote: >>>> >>>> >>>> Joe - Is there a target date for 1.1? >>>> >>>> On Thu, Oct 27, 2016 at 10:50 AM, Joe Witt <joe.w...@gmail.com> wrote: >>>> >>>> Team, >>>> >>>> Just an update on things with working toward an Apache NiFi 1.1.0 >>>> release. There are still about 33 JIRAs there now and some are >>>> awaiting review and are some are under active progress. Yet there is >>>> good traction and progress. I think we should just stay vigilant with >>>> what makes it in and keep working it down. So let's please shoot for >>>> a couple weeks from now. If it is ready sooner I'll jump on it. >>>> >>>> Thanks >>>> Joe >>>> >>>> On Wed, Oct 19, 2016 at 9:06 AM, Joe Witt <joe.w...@gmail.com> wrote: >>>> >>>> Team, >>>> >>>> There are 31 open JIRAs at present tagged to Apache NiFi 1.1.0. Let's >>>> avoiding putting more in there for now at least without a discussion. >>>> Of the 31 JIRAs there the vast majority need review so we should be >>>> able to close these down fairly quickly as long as we don't let the >>>> list grow. >>>> >>>> Thanks >>>> joe >>>> >>>> On Fri, Oct 14, 2016 at 4:39 PM, Edgardo Vega <edgardo.v...@gmail.com> >>>> >>>> wrote: >>>> >>>> Joe, >>>> >>>> Appreciate the offer it isn't my PR. I was just using it as an >>>> >>>> example. >>>> >>>> All >>>> >>>> mine are currently closed, which I greatly appreciate. >>>> >>>> Cheers, >>>> >>>> Edgardo >>>> >>>> On Friday, October 14, 2016, Joe Witt <joe.w...@gmail.com> wrote: >>>> >>>> Edgardo, >>>> >>>> You mentioned a PR from August. I'd be happy to help you work that >>>> through review. >>>> >>>> Thanks >>>> Joe >>>> >>>> On Fri, Oct 14, 2016 at 10:45 AM, Edgardo Vega < >>>> >>>> edgardo.v...@gmail.com >>>> >>>> <javascript:;>> wrote: >>>> >>>> I have agreed that at this point a release is important. My goal >>>> >>>> was >>>> >>>> try >>>> >>>> to >>>> >>>> squeeze in a much goodness as possible into the release, but the >>>> >>>> important >>>> >>>> bug fixes should come first. Getting 1.x into a state where the >>>> >>>> release >>>> >>>> notes don't say that it is geared toward developers and testers is >>>> >>>> really >>>> >>>> huge. >>>> >>>> I think Nifi is a great community otherwise I would participate in >>>> >>>> the >>>> >>>> mailing list, create Jira tickets and pull requests. I am only >>>> >>>> trying to >>>> >>>> strengthen the great thing that is going on here. We can always do >>>> >>>> better. >>>> >>>> I was not trying to put down this community only to participate and >>>> >>>> make >>>> >>>> it >>>> >>>> better. I think this conversation is an indication of how great >>>> >>>> this >>>> >>>> community is. >>>> >>>> Maybe I am being sensitive about this issue and trying to >>>> >>>> strengthen >>>> >>>> the >>>> >>>> nifi community even more, after coming from a conference where it >>>> >>>> was >>>> >>>> reported there was lots of excitement at first and now the >>>> >>>> participation >>>> >>>> in >>>> >>>> the community has really died down and they are struggling. I don't >>>> >>>> want >>>> >>>> to >>>> >>>> see that happen here. >>>> >>>> Cheers, >>>> >>>> Edgardo >>>> >>>> >>>> >>>> >>>> On Fri, Oct 14, 2016 at 9:37 AM, Andre <andre-li...@fucs.org >>>> >>>> <javascript:;>> wrote: >>>> >>>> >>>> Edgardo, >>>> >>>> Thank you for your feedback. We hear your comments and as a >>>> >>>> committer I >>>> >>>> can >>>> >>>> share we are constantly looking to improve the PR process, having >>>> >>>> already >>>> >>>> taken many of the steps you suggest. >>>> >>>> However, it is important to notice that the number of PRs should >>>> >>>> not be >>>> >>>> seen as a metric of engagement by the development community: Most >>>> >>>> of us >>>> >>>> will submit PRs so that our work can be carefully reviewed by our >>>> >>>> peers >>>> >>>> and >>>> >>>> some of us will use JIRA patches to provide contributions. >>>> >>>> Having said that, it is true that some PRs may sit idle for a long >>>> >>>> time >>>> >>>> and >>>> >>>> we are working to improve this pipeline. >>>> >>>> It was therefore no coincidence that I browsed most of the PRs >>>> >>>> performing >>>> >>>> a triage of items that have been superseded or diverged from the >>>> >>>> current >>>> >>>> code base. >>>> >>>> In fact, less than a month ago the dev team closed a number of >>>> >>>> stalled >>>> >>>> and >>>> >>>> superseded PRs (commit cc5e827aa1dfe2f376e9836380ba63c15269eea8). >>>> >>>> Despite all the above, I think Joe has a point. The master >>>> >>>> contain a >>>> >>>> series >>>> >>>> of important bug fixes and suspect the community would benefit >>>> >>>> from >>>> >>>> a >>>> >>>> release sooner rather than later. >>>> >>>> Once again, thank you for your feedback and contribution. It is >>>> >>>> good to >>>> >>>> have you here. >>>> >>>> Andre >>>> >>>> On Fri, Oct 14, 2016 at 11:30 PM, Edgardo Vega < >>>> >>>> edgardo.v...@gmail.com >>>> >>>> <javascript:;>> >>>> >>>> wrote: >>>> >>>> Joe - You are correct I was mentioning the PRs that are >>>> >>>> currently >>>> >>>> open. >>>> >>>> >>>> Regardless of how it happens reducing the count of open PRs I >>>> >>>> believe >>>> >>>> to >>>> >>>> be >>>> >>>> extremely important. Maybe I was hoping that the release could >>>> >>>> be >>>> >>>> a >>>> >>>> forcing >>>> >>>> function to make that happen. I believe that developers are more >>>> >>>> willing >>>> >>>> to >>>> >>>> contribute when they see that their PRs will actually be able >>>> >>>> accepted >>>> >>>> and >>>> >>>> merged into the code base. Having a low number of open PRs in >>>> >>>> progress >>>> >>>> is a >>>> >>>> great indication that the main nifi developers are fully engaged >>>> >>>> with >>>> >>>> the >>>> >>>> community. >>>> >>>> There are a few PRs that don't have any comments from committers >>>> >>>> at >>>> >>>> all. >>>> >>>> I >>>> >>>> found one from August in that state. If that was my PR I don't >>>> >>>> think I >>>> >>>> would be so willing to put another one in anytime soon. I do get >>>> >>>> that >>>> >>>> sometime PRs get stalled by the originator, if so maybe a rule >>>> >>>> about >>>> >>>> closing them after a certain amount of time or being taken over >>>> >>>> by a >>>> >>>> core >>>> >>>> contributor if they think it worthwhile. >>>> >>>> I would like to shoutout to James Wing on my last PR he was >>>> >>>> quick >>>> >>>> to >>>> >>>> review, provided great comments, testing, and even some >>>> >>>> additional >>>> >>>> code. >>>> >>>> It >>>> >>>> was a great PR experience. >>>> >>>> Cheers, >>>> >>>> Edgardo >>>> >>>> >>>> >>>> On Thu, Oct 13, 2016 at 4:14 PM, Joe Percivall < >>>> >>>> joeperciv...@yahoo.com <javascript:;>. >>>> >>>> invalid> wrote: >>>> >>>> Joe, I think you misread. Edgardo is referring to the Pull >>>> >>>> Requests >>>> >>>> that >>>> >>>> are currently open, not the tickets assigned to the 1.1.0 >>>> >>>> version. >>>> >>>> >>>> I think these goals (releasing 1.1.0 and cutting down the PR >>>> >>>> count) >>>> >>>> should >>>> >>>> be two different efforts. Doing a thorough job reviewing >>>> >>>> takes a >>>> >>>> significant amount of time from both the reviewer and >>>> >>>> contributor. >>>> >>>> In >>>> >>>> order >>>> >>>> to cut it down significantly would take much longer than a >>>> >>>> couple >>>> >>>> days. >>>> >>>> >>>> Also there has already been a lot of great new features and >>>> >>>> bug >>>> >>>> fixes >>>> >>>> contributed to the 1.X line and I don't think it's worth >>>> >>>> holding up >>>> >>>> a >>>> >>>> 1.1.0 >>>> >>>> release for tickets not assigned to this fix version. As an >>>> >>>> added >>>> >>>> bonus >>>> >>>> though, I think many of the tickets tagged as 1.1.0 have PRs >>>> >>>> already >>>> >>>> open >>>> >>>> so closing those will make a large dent in the PR count. >>>> >>>> >>>> Joe >>>> >>>> - - - - - - >>>> Joseph Percivall >>>> linkedin.com/in/Percivall >>>> e: joeperciv...@yahoo.com <javascript:;> >>>> >>>> >>>> >>>> On Thursday, October 13, 2016 3:58 PM, Joe Witt < >>>> >>>> joe.w...@gmail.com >>>> >>>> <javascript:;>> >>>> >>>> wrote: >>>> >>>> >>>> >>>> There are less than 30 right now. Many of the roughly 90+ >>>> >>>> JIRAs >>>> >>>> opened on 1.1.0 were easily dispositioned to 1.2.0 or closed >>>> >>>> or >>>> >>>> just >>>> >>>> had fix versions removed. >>>> >>>> We will need to have a push over the next bunch of days to >>>> >>>> deal >>>> >>>> with >>>> >>>> reviewing/merging/moving the remaining items. >>>> >>>> Thanks >>>> Joe >>>> >>>> >>>> On Thu, Oct 13, 2016 at 3:49 PM, Edgardo Vega < >>>> >>>> edgardo.v...@gmail.com <javascript:;>> >>>> >>>> wrote: >>>> >>>> Joe, >>>> >>>> There are 75 PRs currently open. Why not make a push over >>>> >>>> the >>>> >>>> next >>>> >>>> bunch >>>> >>>> of >>>> >>>> days to get them closed and then cut the release after that. >>>> >>>> Cheers, >>>> >>>> Edgardo >>>> >>>> On Thu, Oct 13, 2016 at 12:44 PM, Joe Witt < >>>> >>>> joe.w...@gmail.com >>>> >>>> <javascript:;>> >>>> >>>> wrote: >>>> >>>> >>>> Team, >>>> >>>> There have been a ton of bugs fixed a few nice features. I >>>> >>>> would >>>> >>>> like >>>> >>>> to move to get Apache NiFi 1.1.0 release going pretty much >>>> >>>> based >>>> >>>> on >>>> >>>> where we are now and plan to move most tickets to a new >>>> >>>> Apache >>>> >>>> NiFi >>>> >>>> 1.2.0 version. We can try to get back on our roughly 6-8 >>>> >>>> week >>>> >>>> release >>>> >>>> schedule and shoot for a mid to late Nov release for NiFi >>>> >>>> 1.2.0 >>>> >>>> this >>>> >>>> way as well. Please advise if anyone has any other views on >>>> >>>> this. In >>>> >>>> the mean time I'll get the wheels in motion so you'll be >>>> >>>> seeing a >>>> >>>> lot >>>> >>>> of JIRA/issue updates to move version around. >>>> >>>> Thanks >>>> Joe >>>> >>>> On Thu, Oct 13, 2016 at 12:02 PM, Tony Kurc < >>>> >>>> trk...@gmail.com >>>> >>>> <javascript:;>> >>>> >>>> wrote: >>>> >>>> Sounds good Joe. I have no issue to you doing the rm'ing >>>> >>>> for >>>> >>>> it. >>>> >>>> >>>> On Oct 13, 2016 8:19 AM, "Joe Witt" <joe.w...@gmail.com >>>> >>>> <javascript:;>> wrote: >>>> >>>> >>>> Team, >>>> >>>> There are a lot of great fixes and improvements on the >>>> >>>> master >>>> >>>> line >>>> >>>> now >>>> >>>> and we're at a good time window to start pushing for a >>>> >>>> release. >>>> >>>> There >>>> >>>> are, however, about 90+ JIRAs assigned to 1.1.0 which >>>> >>>> are >>>> >>>> open. >>>> >>>> I'm >>>> >>>> going to go through them and remove fix versions where >>>> >>>> appropriate. >>>> >>>> >>>> I'm happy to take on RM task for this release though if >>>> >>>> someone >>>> >>>> else >>>> >>>> would like to take that on please advise. >>>> >>>> Thanks >>>> Joe >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> >>>> Edgardo >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> >>>> Edgardo >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> >>>> Edgardo >>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> >>>> Edgardo >>>> >>>> Sent from Gmail Mobile >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>
