I'm biased, but I think the new Query / Scroll Elasticsearch processors are pretty big ;)
On Tue, Nov 22, 2016 at 9:02 AM, Andy LoPresto <alopre...@apache.org> wrote: > Important to call out in the security section that > login-identity-providers.xml is now covered by encrypted config, and > flow.xml.gz processor property encryption/nifi.sensitive.props.key value > can now be migrated by the command-line tool (second part pending > completion of NIFI-3024). > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 21, 2016, at 9:30 PM, Joe Witt <joe.w...@gmail.com> wrote: > > matt > > i'll add a wiki page or set of instructions linked from the release notes. > > all, > > walked through the 250 or so JIRAs in the 1.1.0 release and pulled out > highlights. The items noted are as follows. Will likely reduce this > down further for the release notes but wanted to put this out in case > folks have things they think are really important to highlight. > > - Core Improvements: > - Performance: Session Migration > - Stability: Cluster Management > - Developer: Framework supports easy user driven classloader extension > - Expression Language: Now supports base64 and hex encoded values > and Math functions > - Repositories now support rollback > - Faster startup due to more efficient state restoration algorithm > - UX Improvements: > - Visual Backpressure Indicator > - Introduced more colors to better highlight actions and components > - Performance: Validate non-running components > - Provenance graph image can be exported > - Cron Scheduling for Primary node tasks now supported > - Updated versions > - Azure Event Hub 0.9.0 > - Spark 2.0.1 > - Hadoop 2.7.x > - New/Improved Processors > - new Fetch/Put Elastic Search 5.0 > - new ParseCEF to parse CEF formatted logs > - improve ExtractEmail now supports TNEF files > - new Validate CSV > - improved Solr processors now support SSL and Kerberos > - new Websocket client and server processors > - New Utility > - Zookeeper Migrator (move from one zookeeper to another) > - Security > - Restricted Processors > - Site-to-site now supports port forwarding > - Improved Policy Management UX > - Migration Notes: > - Restricted Processors > - Twitter Processor Removed > > On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess <mattyb...@apache.org> > wrote: > > Is there a good spot for us to put instructions on how to build the > Twitter processor and/or the Social Media NAR in the meantime? Maybe a > Wiki page or something simple to say "go to this directory, run this > Maven command, drop the NAR into your deployment..." ? > > On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt <joe.w...@gmail.com> wrote: > > Team, > > We appear to be very close. Andy is working NIFI-3024 but otherwise > it is focus on testing. > > I'm going to prep the RC and release notes now. Unfortunately the > twitter changes for json.org will need to remain. Consensus forming > on the legal-discuss thread regarding a grace period has been elusive > and we're already prepared to make the right steps so we'll just need > to take that on by being empathetic to the user base. > > Thanks > Joe > > On Thu, Nov 17, 2016 at 7:37 AM, Andre <andre-li...@fucs.org> wrote: > > Andy, > > Great to see NIFI-3050 implemented and certainly good news that NiFi 1.1.0 > is set to include a number of security related improvements. > > > > On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto <alopre...@apache.org> > wrote: > > Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have been > added to my plate for this release. Coordinated with Joe Witt and they > should both be included. > > [1] https://issues.apache.org/jira/browse/NIFI-3050 > [2] https://issues.apache.org/jira/browse/NIFI-3051 > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 16, 2016, at 12:08 PM, Joe Witt <joe.w...@gmail.com> wrote: > > Team > > There is a thread on apache legal-discuss that might allow for a > graceperiod of continued usage of the json library. Am going to keep > a close eye on this and if VP Legal approves we'll be able to keep the > twitter processors in which is definitely a good thing. Will advise > > Thanks > Joe > > On Wed, Nov 16, 2016 at 10:37 AM, Bryan Bende <bbe...@gmail.com> wrote: > > I've noticed an issue with the per-instance class loading capability > introduced in NIFI-2909 where the additional classpath resources can get > incorrectly removed from the class loader. > > I was able to reproduced this with a unit test and have a fix ready. I > believe this is important and needs to go in for the 1.1 release, going to > re-open NIFI-2909 and submit a PR shortly. > > -Bryan > > On Wed, Nov 16, 2016 at 8:11 AM, Matt Gilman <matt.c.gil...@gmail.com> > wrote: > > I have two items that I would like to wrap up prior to creating an RC for > 1.1.0. NIFI-2949 addresses some UX issues around Remote Process Group port > configuration. The work is already completed and I will be reviewing it > this today. Additionally, following recent interest on the mailing list, > I'd like to knock out NIFI-3020. This will allow an admin to configure a > strategy for user identity when logging in via LDAP. Specifically, it will > support usage of the DN (the default and current implementation) as well as > the username the user logged in as. I should be able to have a PR up for > this work later today. > > Thanks! > > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-2949 > [2] https://issues.apache.org/jira/browse/NIFI-3020 > > > On Tue, Nov 15, 2016 at 8:00 PM, Joe Witt <joe.w...@gmail.com> wrote: > > The code is within the twitter4j library itself. I filed a request to > twitter4jg. The most likely case is we will need to submit a PR to them. > However, I don't see this as something that should delay the release. We > can provide instructions for folks wanting to use the processor during > > the > > time we cannot make it available in a convenient manner. I will provide > > a > > meaningful comment about this in release notes and pointers on what folks > can do in the meantime. > > On Nov 15, 2016 7:41 PM, "Andy LoPresto" <alopre...@apache.org> wrote: > > I understand there was a discussion thread within the NiFi community > > for > > this as well and I missed responding to that at that time. It just > > seems > > to > > me like JSON processing is necessary for GetTwitter, which is > > incredibly > > useful for demonstrating NiFi’s ability to read from a high volume > > stream > > out of the box. With NIFI-3019 (Remove GetTwitter from default build), > > is > > there any related effort to substitute an acceptable replacement JSON > library to restore this functionality? > > [1] https://issues.apache.org/jira/browse/NIFI-3019 > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 15, 2016, at 4:36 PM, Andy LoPresto <alopre...@apache.org> > > wrote: > > > I’m working with Bryan Rosander to close out NIFI-3024, NIFI-2655, and > NIFI-2653. I believe Matt Burgess is working on NIFI-3011 and we > investigated some alternate TLS config options for the new version of > > the > > client library. > > Is there any alternative to excluding the GetTwitter processor? Using > Johnzon [1] or the Android re-implementation [2] discussed in the > > mailing > > list thread? > > [1] https://johnzon.apache.org/ > [2] https://developer.android.com/reference/org/json/package- > > summary.html > > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 15, 2016, at 3:58 PM, Joe Witt <joe.w...@gmail.com> wrote: > > Team > > Very happy to see that we are down to three items remaining tagged to > 1.1.0. Solid effort over the recent weeks to close the gap including > > work > > to get past the now category x Jason dependency we had. The most > > notable > > impact from that is the wildly popular GetTwitter processor, the fav > > new > > nifi user and demo processor, can no longer be included in the default > build. It is optionally available if users choose to build and use it > > but > > we won't distribute binaries that have it. > > I see some review movement on some patch available but untagged items. > > I plan to kick off the 1.1.0 rc work soon. Perhaps Thurs or Fri. Anyone > have any outstanding items? > > Thanks > Joe > > On Nov 8, 2016 2:12 PM, "Joe Witt" <joe.w...@gmail.com> wrote: > > Ryan > > Not officially but I think we should try to close this thing out and > start a vote in the next week or two at most. > > I'm going through the tickets again now. There is also a new issue of > the json-p license falling out of favor in Apache legal terms and > becoming Category-X. Am looking into that now. > > Thanks > Joe > > On Tue, Nov 8, 2016 at 2:05 PM, Ryan Ward <ryan.wa...@gmail.com> > > wrote: > > > Joe - Is there a target date for 1.1? > > On Thu, Oct 27, 2016 at 10:50 AM, Joe Witt <joe.w...@gmail.com> wrote: > > Team, > > Just an update on things with working toward an Apache NiFi 1.1.0 > release. There are still about 33 JIRAs there now and some are > awaiting review and are some are under active progress. Yet there is > good traction and progress. I think we should just stay vigilant with > what makes it in and keep working it down. So let's please shoot for > a couple weeks from now. If it is ready sooner I'll jump on it. > > Thanks > Joe > > On Wed, Oct 19, 2016 at 9:06 AM, Joe Witt <joe.w...@gmail.com> wrote: > > Team, > > There are 31 open JIRAs at present tagged to Apache NiFi 1.1.0. Let's > avoiding putting more in there for now at least without a discussion. > Of the 31 JIRAs there the vast majority need review so we should be > able to close these down fairly quickly as long as we don't let the > list grow. > > Thanks > joe > > On Fri, Oct 14, 2016 at 4:39 PM, Edgardo Vega <edgardo.v...@gmail.com> > > wrote: > > Joe, > > Appreciate the offer it isn't my PR. I was just using it as an > > example. > > All > > mine are currently closed, which I greatly appreciate. > > Cheers, > > Edgardo > > On Friday, October 14, 2016, Joe Witt <joe.w...@gmail.com> wrote: > > Edgardo, > > You mentioned a PR from August. I'd be happy to help you work that > through review. > > Thanks > Joe > > On Fri, Oct 14, 2016 at 10:45 AM, Edgardo Vega < > > edgardo.v...@gmail.com > > <javascript:;>> wrote: > > I have agreed that at this point a release is important. My goal > > was > > try > > to > > squeeze in a much goodness as possible into the release, but the > > important > > bug fixes should come first. Getting 1.x into a state where the > > release > > notes don't say that it is geared toward developers and testers is > > really > > huge. > > I think Nifi is a great community otherwise I would participate in > > the > > mailing list, create Jira tickets and pull requests. I am only > > trying to > > strengthen the great thing that is going on here. We can always do > > better. > > I was not trying to put down this community only to participate and > > make > > it > > better. I think this conversation is an indication of how great > > this > > community is. > > Maybe I am being sensitive about this issue and trying to > > strengthen > > the > > nifi community even more, after coming from a conference where it > > was > > reported there was lots of excitement at first and now the > > participation > > in > > the community has really died down and they are struggling. I don't > > want > > to > > see that happen here. > > Cheers, > > Edgardo > > > > > On Fri, Oct 14, 2016 at 9:37 AM, Andre <andre-li...@fucs.org > > <javascript:;>> wrote: > > > Edgardo, > > Thank you for your feedback. We hear your comments and as a > > committer I > > can > > share we are constantly looking to improve the PR process, having > > already > > taken many of the steps you suggest. > > However, it is important to notice that the number of PRs should > > not be > > seen as a metric of engagement by the development community: Most > > of us > > will submit PRs so that our work can be carefully reviewed by our > > peers > > and > > some of us will use JIRA patches to provide contributions. > > Having said that, it is true that some PRs may sit idle for a long > > time > > and > > we are working to improve this pipeline. > > It was therefore no coincidence that I browsed most of the PRs > > performing > > a triage of items that have been superseded or diverged from the > > current > > code base. > > In fact, less than a month ago the dev team closed a number of > > stalled > > and > > superseded PRs (commit cc5e827aa1dfe2f376e9836380ba63c15269eea8). > > Despite all the above, I think Joe has a point. The master > > contain a > > series > > of important bug fixes and suspect the community would benefit > > from > > a > > release sooner rather than later. > > Once again, thank you for your feedback and contribution. It is > > good to > > have you here. > > Andre > > On Fri, Oct 14, 2016 at 11:30 PM, Edgardo Vega < > > edgardo.v...@gmail.com > > <javascript:;>> > > wrote: > > Joe - You are correct I was mentioning the PRs that are > > currently > > open. > > > Regardless of how it happens reducing the count of open PRs I > > believe > > to > > be > > extremely important. Maybe I was hoping that the release could > > be > > a > > forcing > > function to make that happen. I believe that developers are more > > willing > > to > > contribute when they see that their PRs will actually be able > > accepted > > and > > merged into the code base. Having a low number of open PRs in > > progress > > is a > > great indication that the main nifi developers are fully engaged > > with > > the > > community. > > There are a few PRs that don't have any comments from committers > > at > > all. > > I > > found one from August in that state. If that was my PR I don't > > think I > > would be so willing to put another one in anytime soon. I do get > > that > > sometime PRs get stalled by the originator, if so maybe a rule > > about > > closing them after a certain amount of time or being taken over > > by a > > core > > contributor if they think it worthwhile. > > I would like to shoutout to James Wing on my last PR he was > > quick > > to > > review, provided great comments, testing, and even some > > additional > > code. > > It > > was a great PR experience. > > Cheers, > > Edgardo > > > > On Thu, Oct 13, 2016 at 4:14 PM, Joe Percivall < > > joeperciv...@yahoo.com <javascript:;>. > > invalid> wrote: > > Joe, I think you misread. Edgardo is referring to the Pull > > Requests > > that > > are currently open, not the tickets assigned to the 1.1.0 > > version. > > > I think these goals (releasing 1.1.0 and cutting down the PR > > count) > > should > > be two different efforts. Doing a thorough job reviewing > > takes a > > significant amount of time from both the reviewer and > > contributor. > > In > > order > > to cut it down significantly would take much longer than a > > couple > > days. > > > Also there has already been a lot of great new features and > > bug > > fixes > > contributed to the 1.X line and I don't think it's worth > > holding up > > a > > 1.1.0 > > release for tickets not assigned to this fix version. As an > > added > > bonus > > though, I think many of the tickets tagged as 1.1.0 have PRs > > already > > open > > so closing those will make a large dent in the PR count. > > > Joe > > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: joeperciv...@yahoo.com <javascript:;> > > > > On Thursday, October 13, 2016 3:58 PM, Joe Witt < > > joe.w...@gmail.com > > <javascript:;>> > > wrote: > > > > There are less than 30 right now. Many of the roughly 90+ > > JIRAs > > opened on 1.1.0 were easily dispositioned to 1.2.0 or closed > > or > > just > > had fix versions removed. > > We will need to have a push over the next bunch of days to > > deal > > with > > reviewing/merging/moving the remaining items. > > Thanks > Joe > > > On Thu, Oct 13, 2016 at 3:49 PM, Edgardo Vega < > > edgardo.v...@gmail.com <javascript:;>> > > wrote: > > Joe, > > There are 75 PRs currently open. Why not make a push over > > the > > next > > bunch > > of > > days to get them closed and then cut the release after that. > > Cheers, > > Edgardo > > On Thu, Oct 13, 2016 at 12:44 PM, Joe Witt < > > joe.w...@gmail.com > > <javascript:;>> > > wrote: > > > Team, > > There have been a ton of bugs fixed a few nice features. I > > would > > like > > to move to get Apache NiFi 1.1.0 release going pretty much > > based > > on > > where we are now and plan to move most tickets to a new > > Apache > > NiFi > > 1.2.0 version. We can try to get back on our roughly 6-8 > > week > > release > > schedule and shoot for a mid to late Nov release for NiFi > > 1.2.0 > > this > > way as well. Please advise if anyone has any other views on > > this. In > > the mean time I'll get the wheels in motion so you'll be > > seeing a > > lot > > of JIRA/issue updates to move version around. > > Thanks > Joe > > On Thu, Oct 13, 2016 at 12:02 PM, Tony Kurc < > > trk...@gmail.com > > <javascript:;>> > > wrote: > > Sounds good Joe. I have no issue to you doing the rm'ing > > for > > it. > > > On Oct 13, 2016 8:19 AM, "Joe Witt" <joe.w...@gmail.com > > <javascript:;>> wrote: > > > Team, > > There are a lot of great fixes and improvements on the > > master > > line > > now > > and we're at a good time window to start pushing for a > > release. > > There > > are, however, about 90+ JIRAs assigned to 1.1.0 which > > are > > open. > > I'm > > going to go through them and remove fix versions where > > appropriate. > > > I'm happy to take on RM task for this release though if > > someone > > else > > would like to take that on please advise. > > Thanks > Joe > > > > > > -- > Cheers, > > Edgardo > > > > > > -- > Cheers, > > Edgardo > > > > > > -- > Cheers, > > Edgardo > > > > > -- > Cheers, > > Edgardo > > Sent from Gmail Mobile > > > > > > > > > > > -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength. *-Philippians 4:12-13*
