Re: Disconnected node log, can you explain what is happening ?

[Joe Witt]

Hello Alessio

The concerning bit from the log is the ClassCastException.

Have you altered the lib folder in some way?

Thanks
Joe

On Tue, Nov 22, 2016 at 8:22 AM, Alessio Palma
 wrote:
> Hello all,
> I'm working on 3 nodes cluster, sometimes nodes get disconnected and there
> is no way to connect them again using the Cluster menu.
> I checked the logs and this is what is reported, can you explain what is
> going on ?
>
>
> AP
>
>

Re: PostHttp processor: read timeout error (socketTimeoutException)

[balacode63]

Thanks Andrew,
Will check with influxdb mail list.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/PostHttp-processor-read-timeout-error-socketTimeoutException-tp13879p13946.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: nifi configuration file for process group

[balacode63]

Thanks for your detailed reply  Joseph Percivalllinkedin.
Yes Im using multiple http processors. As you suggested , I will try to
explore  Getfile and extracttext processors.

Sorry for the late reply.

Thanks Again.




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-configuration-file-for-process-group-tp13908p13945.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: [DISCUSS] NiFi 1.1.0 release

[Andy LoPresto]

Important to call out in the security section that login-identity-providers.xml 
is now covered by encrypted config, and flow.xml.gz processor property 
encryption/nifi.sensitive.props.key value can now be migrated by the 
command-line tool (second part pending completion of NIFI-3024).

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Nov 21, 2016, at 9:30 PM, Joe Witt  wrote:
> 
> matt
> 
> i'll add a wiki page or set of instructions linked from the release notes.
> 
> all,
> 
> walked through the 250 or so JIRAs in the 1.1.0 release and pulled out
> highlights.  The items noted are as follows.  Will likely reduce this
> down further for the release notes but wanted to put this out in case
> folks have things they think are really important to highlight.
> 
> - Core Improvements:
>   - Performance: Session Migration
>   - Stability: Cluster Management
>   - Developer: Framework supports easy user driven classloader extension
>   - Expression Language: Now supports base64 and hex encoded values
> and Math functions
>   - Repositories now support rollback
>   - Faster startup due to more efficient state restoration algorithm
> - UX Improvements:
>   - Visual Backpressure Indicator
>   - Introduced more colors to better highlight actions and components
>   - Performance: Validate non-running components
>   - Provenance graph image can be exported
>   - Cron Scheduling for Primary node tasks now supported
> - Updated versions
>   - Azure Event Hub 0.9.0
>   - Spark 2.0.1
>   - Hadoop 2.7.x
> - New/Improved Processors
>   - new Fetch/Put Elastic Search 5.0
>   - new ParseCEF to parse CEF formatted logs
>   - improve ExtractEmail now supports TNEF files
>   - new Validate CSV
>   - improved Solr processors now support SSL and Kerberos
>   - new Websocket client and server processors
> - New Utility
>   - Zookeeper Migrator (move from one zookeeper to another)
> - Security
>   - Restricted Processors
>   - Site-to-site now supports port forwarding
>   - Improved Policy Management UX
> - Migration Notes:
>   - Restricted Processors
>   - Twitter Processor Removed
> 
> On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess  wrote:
>> Is there a good spot for us to put instructions on how to build the
>> Twitter processor and/or the Social Media NAR in the meantime? Maybe a
>> Wiki page or something simple to say "go to this directory, run this
>> Maven command, drop the NAR into your deployment..." ?
>> 
>> On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt  wrote:
>>> Team,
>>> 
>>> We appear to be very close.  Andy is working NIFI-3024 but otherwise
>>> it is focus on testing.
>>> 
>>> I'm going to prep the RC and release notes now.  Unfortunately the
>>> twitter changes for json.org will need to remain.  Consensus forming
>>> on the legal-discuss thread regarding a grace period has been elusive
>>> and we're already prepared to make the right steps so we'll just need
>>> to take that on by being empathetic to the user base.
>>> 
>>> Thanks
>>> Joe
>>> 
>>> On Thu, Nov 17, 2016 at 7:37 AM, Andre  wrote:
 Andy,
 
 Great to see NIFI-3050 implemented and certainly good news that NiFi 1.1.0
 is set to include a number of security related improvements.
 
 
 
 On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto  
 wrote:
 
> Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have been
> added to my plate for this release. Coordinated with Joe Witt and they
> should both be included.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-3050
> [2] https://issues.apache.org/jira/browse/NIFI-3051
> 
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
> On Nov 16, 2016, at 12:08 PM, Joe Witt  wrote:
> 
> Team
> 
> There is a thread on apache legal-discuss that might allow for a
> graceperiod of continued usage of the json library.  Am going to keep
> a close eye on this and if VP Legal approves we'll be able to keep the
> twitter processors in which is definitely a good thing.  Will advise
> 
> Thanks
> Joe
> 
> On Wed, Nov 16, 2016 at 10:37 AM, Bryan Bende  wrote:
> 
> I've noticed an issue with the per-instance class loading capability
> introduced in NIFI-2909 where the additional classpath resources can get
> incorrectly removed from the class loader.
> 
> I was able to reproduced this with a unit test and have a fix ready. I
> believe this is important and needs to go in for the 1.1 release, going to
> re-open NIFI-2909 and submit a PR shortly.
> 
> -Bryan
> 
> On Wed, Nov 16, 2016 at 8:11 AM, Matt 

Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Aldrin Piri]

Hi Daniel,

Thanks for mailing and getting involved with MiNiFi!

There is no inherent coupling of versions rather, as has been the custom
within the NiFi community, we use, roughly, semantic versioning.  As a
result, our initial release of MiNiFi C++ started as 0.0.1 and since there
have been some new features, our next release is considered a minor one,
putting us at 0.1.0.  There is no correlation of the C++ version with that
of the Java codebase, rather both being in their relative infancy are
starting from a common place.  Instead, this release is predicated on
keeping a regular cadence of making the community's efforts available for
general consumption.  Additionally, the release is also being suggested as
we've reached a point with the filed issues where those items not scheduled
for the next release require considerable effort and another period of
development to incorporate.

In terms of roadmap, there is nothing rigidly defined and is very much open
to the direction of the community and its will.  There is a rough outline
that was captured on the wiki which I'd imagine you have seen, but is
additionally provided as context for anyone else reading along [1].
Please feel free to share thoughts on specifics in terms of milestones for
challenges you are looking to address on the Wiki and we can look to grow
and evolve that information.

>From my perspective, I feel the most important area is that of chasing down
framework parity with the core features and building blocks of all that our
Java equivalents have to offer.  In addition to this, extending the
flexibility of the user experience in dataflow to our MiNiFi instances as
outlined with some of the feature proposals such as Command & Control
(C2)[2] are areas that can be worked on in a parallel fashion.  In general,
my instincts lead me to believe that the C++ version will be a "generation"
behind Java for the immediate future.  MiNiFi Java allows us to make use of
much of the effort done in NiFi to try new things out, iterate, and expand
functionality quickly.   Once established, we can then perform the same in
C++.

In terms of your view of the hub and spoke model, I believe this is the
shared vision of the community long term; the efforts of (C2) would allow
users to more effectively execute such architectures more efficiently,
extending the core principles of NiFi beyond the traditional data center.

We have ensured initial connectivity to core NiFi via the raw socket Site
to Site implementation in C++.  Later, this will be further extended with
the HTTP equivalent, which will make its debut in the MiNiFi realm with the
next release of MiNiFi Java.

Please let us know if there are additional questions you would like to
discuss and appreciate your feedback on the process and look forward to
your continued engagement.

Thanks!

[1] https://cwiki.apache.org/confluence/display/MINIFI/Proposed+Roadmap
[2]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control

On Tue, Nov 22, 2016 at 1:29 PM, Daniel Cave  wrote:

> Having been out of touch since MiNiFi C++ got added and just getting into
> it,
> is there a reason the C++ version is trying to follow closely the MiNiFi
> Java version rather than just insuring connectivity with NiFi?  I have not
> been able to find alot of details regarding the roadmap for MiNiFi C++.
>
> It seems to me that this tight coupling is coming at the cost of the
> efficiency that should be gained through a C++ version.  MiNiFi C++ should
> lend itself to a hub and spoke model with MiNiFi C++ acting as the spoke
> clients and NiFi as the hub.  This only works, however, if maximum
> efficiency is maintained as spoke needs may range from servers to embedded.
> Additional to embedded advantages, MiNiFi C++ also has the ability to run
> natively as a Windows service with direct interaction with the Windows API
> which is also difficult at best with the Java version.
>
> Can you please provide some clarity on where things are headed?  For
> reference, I have been through the wiki, JIRA, Confluence, Git, etc.
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13957.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Daniel Cave]

For me personally, I don't see a value add of MiNiFi Java.  The needs that
NiFi can't address MiNiFi Java can't either, so my focus is MiNiFi C++ as
that is the hole that needs fixing, again in my opinion, so that is where my
MiNiFi focus is going to be.

As I go through things I am sure I will have more questions about choices
that have been made so far regarding MiNiFi C++ (as with all things, we all
have different views on how do things and there isn't necessarily a
right/wrong answer).  If there is a better forum to address these more
specific to MiNiFi C++, please let me know.  My most pressing question is
the choice to use LevelDB for the provenance repository rather than LMDB.  A
core tenant of NiFi is fault tolerance in near all cases (as well as full
data provenance).  As LevelDB is vulnerable to corruption during write
operations due to unexpected application interruptions, would not something
more fault tolerant such as LMDB (covered under OpenLDAP Public License) be
preferable?  The question of fault tolerance applies to the flowfile
repository as well.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13959.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Joe Witt]

Regarding OpenLDAP Public License

This does not appear to be listed under the approved apache licenses
to use for source or binary dependencies.  However, it looks like it
is probably considered a variant of the BSD 3-Clause license which
would be ok as category a.  It would be helpful to validate this with
a question to legal though.

Regarding LevelDB vs Anything

Right now we have something we can discuss and validate the pros and
cons of.  If there are other implementations that allow us to do that
let's bring them forth and do that.  I don't think we're at the phase
now where any decision is necessarily reflective of a long term intent
so plenty of opportunity to choose the best tradeoffs.

This thread:

We should probably root this thread on the release discussion and fork
it out to its own thread to discuss some of these points raised.

Thanks
Joe

On Tue, Nov 22, 2016 at 4:03 PM, Andy LoPresto  wrote:
> Daniel,
>
> I think one reason that MiNiFi Java has been successful, both as a prototype
> and tool deployed in real environments today, is that it was able to
> leverage much of the existing NiFi Java codebase. This greatly reduced the
> "time-to-market” and allowed the development team to get working code out
> into the ecosystem and start to build a community and refine the
> application. Part of that community is gathering feedback from interested
> parties, which you are helpfully providing here. However, I do believe there
> is legitimate value to MiNiFi Java. Developers are hard at work on both the
> Java and C++ versions and are pushing for feature parity so that choice of
> language is not a barrier to feature availability. We welcome community
> focus on various aspects of the project and would not try to direct you to
> get involved with MiNiFi Java if that’s not what you are interested in.
>
> For now, this list is the best place to have conversations about MiNiFi C++
> topics. The wiki does have room for feature requests, roadmap evolution,
> etc. As the MiNiFi community grows, we may split the mailing lists, but that
> would likely not be for some time.
>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 22, 2016, at 11:23 AM, Daniel Cave  wrote:
>
> For me personally, I don't see a value add of MiNiFi Java.  The needs that
> NiFi can't address MiNiFi Java can't either, so my focus is MiNiFi C++ as
> that is the hole that needs fixing, again in my opinion, so that is where my
> MiNiFi focus is going to be.
>
> As I go through things I am sure I will have more questions about choices
> that have been made so far regarding MiNiFi C++ (as with all things, we all
> have different views on how do things and there isn't necessarily a
> right/wrong answer).  If there is a better forum to address these more
> specific to MiNiFi C++, please let me know.  My most pressing question is
> the choice to use LevelDB for the provenance repository rather than LMDB.  A
> core tenant of NiFi is fault tolerance in near all cases (as well as full
> data provenance).  As LevelDB is vulnerable to corruption during write
> operations due to unexpected application interruptions, would not something
> more fault tolerant such as LMDB (covered under OpenLDAP Public License) be
> preferable?  The question of fault tolerance applies to the flowfile
> repository as well.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13959.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
>
>

Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Andy LoPresto]

Daniel,

I think one reason that MiNiFi Java has been successful, both as a prototype 
and tool deployed in real environments today, is that it was able to leverage 
much of the existing NiFi Java codebase. This greatly reduced the 
"time-to-market” and allowed the development team to get working code out into 
the ecosystem and start to build a community and refine the application. Part 
of that community is gathering feedback from interested parties, which you are 
helpfully providing here. However, I do believe there is legitimate value to 
MiNiFi Java. Developers are hard at work on both the Java and C++ versions and 
are pushing for feature parity so that choice of language is not a barrier to 
feature availability. We welcome community focus on various aspects of the 
project and would not try to direct you to get involved with MiNiFi Java if 
that’s not what you are interested in.

For now, this list is the best place to have conversations about MiNiFi C++ 
topics. The wiki does have room for feature requests, roadmap evolution, etc. 
As the MiNiFi community grows, we may split the mailing lists, but that would 
likely not be for some time.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Nov 22, 2016, at 11:23 AM, Daniel Cave  wrote:
> 
> For me personally, I don't see a value add of MiNiFi Java.  The needs that
> NiFi can't address MiNiFi Java can't either, so my focus is MiNiFi C++ as
> that is the hole that needs fixing, again in my opinion, so that is where my
> MiNiFi focus is going to be.
> 
> As I go through things I am sure I will have more questions about choices
> that have been made so far regarding MiNiFi C++ (as with all things, we all
> have different views on how do things and there isn't necessarily a
> right/wrong answer).  If there is a better forum to address these more
> specific to MiNiFi C++, please let me know.  My most pressing question is
> the choice to use LevelDB for the provenance repository rather than LMDB.  A
> core tenant of NiFi is fault tolerance in near all cases (as well as full
> data provenance).  As LevelDB is vulnerable to corruption during write
> operations due to unexpected application interruptions, would not something
> more fault tolerant such as LMDB (covered under OpenLDAP Public License) be
> preferable?  The question of fault tolerance applies to the flowfile
> repository as well.
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13959.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.



signature.asc
Description: Message signed with OpenPGP using GPGMail

[DISCUSS] MiNiFi C++ 0.1.0 Release

[Aldrin Piri]

Hey folks,

There have been some nice improvements increasing stability and another
 step toward framework parity with its Java counterparts.  There was also
work done to update build tooling to provide a better foundation to tackle
specific environments and flexibility in generating differing builds.

To capture this progress, I would like to generate another release of
MiNiFi C++ as we start establishing the basis for higher level capabilities
such as command and control.

I think there may be a couple of other issues that may be candidates for
simple fixes to make this release.  I will scope out the issues registered
in JIRA and collect those.

I will volunteer to act as RM for this release if no one else has any
interest.

Thanks!

Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Daniel Cave]

Having been out of touch since MiNiFi C++ got added and just getting into it,
is there a reason the C++ version is trying to follow closely the MiNiFi
Java version rather than just insuring connectivity with NiFi?  I have not
been able to find alot of details regarding the roadmap for MiNiFi C++.

It seems to me that this tight coupling is coming at the cost of the
efficiency that should be gained through a C++ version.  MiNiFi C++ should
lend itself to a hub and spoke model with MiNiFi C++ acting as the spoke
clients and NiFi as the hub.  This only works, however, if maximum
efficiency is maintained as spoke needs may range from servers to embedded. 
Additional to embedded advantages, MiNiFi C++ also has the ability to run
natively as a Windows service with direct interaction with the Windows API
which is also difficult at best with the Java version.

Can you please provide some clarity on where things are headed?  For
reference, I have been through the wiki, JIRA, Confluence, Git, etc.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13957.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Disconnected node log, can you explain what is happening ?

[Alessio Palma]

Hi Joe,
we are running a modified code, basically we removed the ORC table support from 
NIFI. Do you think this problem is due to our patches?
Actually we don't use anything which is related to ORC.


From: Joe Witt 
Sent: Tuesday, November 22, 2016 2:25:47 PM
To: dev@nifi.apache.org
Subject: Re: Disconnected node log, can you explain what is happening ?

Hello Alessio

The concerning bit from the log is the ClassCastException.

Have you altered the lib folder in some way?

Thanks
Joe

On Tue, Nov 22, 2016 at 8:22 AM, Alessio Palma
 wrote:
> Hello all,
> I'm working on 3 nodes cluster, sometimes nodes get disconnected and there
> is no way to connect them again using the Cluster menu.
> I checked the logs and this is what is reported, can you explain what is
> going on ?
>
>
> AP
>
>

Re: [DISCUSS] NiFi 1.1.0 release

[Matt Burgess]

I've reviewed NIFI-3064 and it's good to go, would like to add that one as well.

Thanks,
Matt

On Tue, Nov 22, 2016 at 10:51 AM, Joe Witt  wrote:
> thanks Joe and Andy.  Have those on the list now.
>
> On Tue, Nov 22, 2016 at 5:09 AM, Joe Gresock  wrote:
>> I'm biased, but I think the new Query / Scroll Elasticsearch processors are
>> pretty big ;)
>>
>> On Tue, Nov 22, 2016 at 9:02 AM, Andy LoPresto  wrote:
>>
>>> Important to call out in the security section that
>>> login-identity-providers.xml is now covered by encrypted config, and
>>> flow.xml.gz processor property encryption/nifi.sensitive.props.key value
>>> can now be migrated by the command-line tool (second part pending
>>> completion of NIFI-3024).
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Nov 21, 2016, at 9:30 PM, Joe Witt  wrote:
>>>
>>> matt
>>>
>>> i'll add a wiki page or set of instructions linked from the release notes.
>>>
>>> all,
>>>
>>> walked through the 250 or so JIRAs in the 1.1.0 release and pulled out
>>> highlights.  The items noted are as follows.  Will likely reduce this
>>> down further for the release notes but wanted to put this out in case
>>> folks have things they think are really important to highlight.
>>>
>>> - Core Improvements:
>>>   - Performance: Session Migration
>>>   - Stability: Cluster Management
>>>   - Developer: Framework supports easy user driven classloader extension
>>>   - Expression Language: Now supports base64 and hex encoded values
>>> and Math functions
>>>   - Repositories now support rollback
>>>   - Faster startup due to more efficient state restoration algorithm
>>> - UX Improvements:
>>>   - Visual Backpressure Indicator
>>>   - Introduced more colors to better highlight actions and components
>>>   - Performance: Validate non-running components
>>>   - Provenance graph image can be exported
>>>   - Cron Scheduling for Primary node tasks now supported
>>> - Updated versions
>>>   - Azure Event Hub 0.9.0
>>>   - Spark 2.0.1
>>>   - Hadoop 2.7.x
>>> - New/Improved Processors
>>>   - new Fetch/Put Elastic Search 5.0
>>>   - new ParseCEF to parse CEF formatted logs
>>>   - improve ExtractEmail now supports TNEF files
>>>   - new Validate CSV
>>>   - improved Solr processors now support SSL and Kerberos
>>>   - new Websocket client and server processors
>>> - New Utility
>>>   - Zookeeper Migrator (move from one zookeeper to another)
>>> - Security
>>>   - Restricted Processors
>>>   - Site-to-site now supports port forwarding
>>>   - Improved Policy Management UX
>>> - Migration Notes:
>>>   - Restricted Processors
>>>   - Twitter Processor Removed
>>>
>>> On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess 
>>> wrote:
>>>
>>> Is there a good spot for us to put instructions on how to build the
>>> Twitter processor and/or the Social Media NAR in the meantime? Maybe a
>>> Wiki page or something simple to say "go to this directory, run this
>>> Maven command, drop the NAR into your deployment..." ?
>>>
>>> On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt  wrote:
>>>
>>> Team,
>>>
>>> We appear to be very close.  Andy is working NIFI-3024 but otherwise
>>> it is focus on testing.
>>>
>>> I'm going to prep the RC and release notes now.  Unfortunately the
>>> twitter changes for json.org will need to remain.  Consensus forming
>>> on the legal-discuss thread regarding a grace period has been elusive
>>> and we're already prepared to make the right steps so we'll just need
>>> to take that on by being empathetic to the user base.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Nov 17, 2016 at 7:37 AM, Andre  wrote:
>>>
>>> Andy,
>>>
>>> Great to see NIFI-3050 implemented and certainly good news that NiFi 1.1.0
>>> is set to include a number of security related improvements.
>>>
>>>
>>>
>>> On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto 
>>> wrote:
>>>
>>> Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have been
>>> added to my plate for this release. Coordinated with Joe Witt and they
>>> should both be included.
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-3050
>>> [2] https://issues.apache.org/jira/browse/NIFI-3051
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Nov 16, 2016, at 12:08 PM, Joe Witt  wrote:
>>>
>>> Team
>>>
>>> There is a thread on apache legal-discuss that might allow for a
>>> graceperiod of continued usage of the json library.  Am going to keep
>>> a close eye on this and if VP Legal approves we'll be able to keep the
>>> twitter processors in which is definitely a good thing.  Will advise
>>>
>>> 

Re: [DISCUSS] NiFi 1.1.0 release

[Joe Witt]

thanks Joe and Andy.  Have those on the list now.

On Tue, Nov 22, 2016 at 5:09 AM, Joe Gresock  wrote:
> I'm biased, but I think the new Query / Scroll Elasticsearch processors are
> pretty big ;)
>
> On Tue, Nov 22, 2016 at 9:02 AM, Andy LoPresto  wrote:
>
>> Important to call out in the security section that
>> login-identity-providers.xml is now covered by encrypted config, and
>> flow.xml.gz processor property encryption/nifi.sensitive.props.key value
>> can now be migrated by the command-line tool (second part pending
>> completion of NIFI-3024).
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 21, 2016, at 9:30 PM, Joe Witt  wrote:
>>
>> matt
>>
>> i'll add a wiki page or set of instructions linked from the release notes.
>>
>> all,
>>
>> walked through the 250 or so JIRAs in the 1.1.0 release and pulled out
>> highlights.  The items noted are as follows.  Will likely reduce this
>> down further for the release notes but wanted to put this out in case
>> folks have things they think are really important to highlight.
>>
>> - Core Improvements:
>>   - Performance: Session Migration
>>   - Stability: Cluster Management
>>   - Developer: Framework supports easy user driven classloader extension
>>   - Expression Language: Now supports base64 and hex encoded values
>> and Math functions
>>   - Repositories now support rollback
>>   - Faster startup due to more efficient state restoration algorithm
>> - UX Improvements:
>>   - Visual Backpressure Indicator
>>   - Introduced more colors to better highlight actions and components
>>   - Performance: Validate non-running components
>>   - Provenance graph image can be exported
>>   - Cron Scheduling for Primary node tasks now supported
>> - Updated versions
>>   - Azure Event Hub 0.9.0
>>   - Spark 2.0.1
>>   - Hadoop 2.7.x
>> - New/Improved Processors
>>   - new Fetch/Put Elastic Search 5.0
>>   - new ParseCEF to parse CEF formatted logs
>>   - improve ExtractEmail now supports TNEF files
>>   - new Validate CSV
>>   - improved Solr processors now support SSL and Kerberos
>>   - new Websocket client and server processors
>> - New Utility
>>   - Zookeeper Migrator (move from one zookeeper to another)
>> - Security
>>   - Restricted Processors
>>   - Site-to-site now supports port forwarding
>>   - Improved Policy Management UX
>> - Migration Notes:
>>   - Restricted Processors
>>   - Twitter Processor Removed
>>
>> On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess 
>> wrote:
>>
>> Is there a good spot for us to put instructions on how to build the
>> Twitter processor and/or the Social Media NAR in the meantime? Maybe a
>> Wiki page or something simple to say "go to this directory, run this
>> Maven command, drop the NAR into your deployment..." ?
>>
>> On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt  wrote:
>>
>> Team,
>>
>> We appear to be very close.  Andy is working NIFI-3024 but otherwise
>> it is focus on testing.
>>
>> I'm going to prep the RC and release notes now.  Unfortunately the
>> twitter changes for json.org will need to remain.  Consensus forming
>> on the legal-discuss thread regarding a grace period has been elusive
>> and we're already prepared to make the right steps so we'll just need
>> to take that on by being empathetic to the user base.
>>
>> Thanks
>> Joe
>>
>> On Thu, Nov 17, 2016 at 7:37 AM, Andre  wrote:
>>
>> Andy,
>>
>> Great to see NIFI-3050 implemented and certainly good news that NiFi 1.1.0
>> is set to include a number of security related improvements.
>>
>>
>>
>> On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto 
>> wrote:
>>
>> Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have been
>> added to my plate for this release. Coordinated with Joe Witt and they
>> should both be included.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3050
>> [2] https://issues.apache.org/jira/browse/NIFI-3051
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 16, 2016, at 12:08 PM, Joe Witt  wrote:
>>
>> Team
>>
>> There is a thread on apache legal-discuss that might allow for a
>> graceperiod of continued usage of the json library.  Am going to keep
>> a close eye on this and if VP Legal approves we'll be able to keep the
>> twitter processors in which is definitely a good thing.  Will advise
>>
>> Thanks
>> Joe
>>
>> On Wed, Nov 16, 2016 at 10:37 AM, Bryan Bende  wrote:
>>
>> I've noticed an issue with the per-instance class loading capability
>> introduced in NIFI-2909 where the additional classpath resources can get
>> incorrectly removed from the class loader.
>>
>> I 

Re: [DISCUSS] NiFi 1.1.0 release

[Joe Witt]

mike - also to clarify based on what you brought up I updated the JIRA
description as follows

The following also would be true:
1) Apache NiFi 1.0.0 repositories should work just fine when applied
to an Apache NiFi 1.1.0 installation.

2) Repositories made/updated in Apache NiFi 1.1.0 onward would not
work in older Apache NiFi releases (such as 1.0.0)

On Tue, Nov 22, 2016 at 12:33 PM, Joe Witt  wrote:
> Mike - yeah good catch and good question.  It does support the old
> format.  We've always been pretty good about being diligent to allow
> folks to upgrade and it honor existing state and in many cases
> configurations and it would automatically port them over.  What has
> always been far more problematic is 'rollback' where people ran on
> newer configurations but could not then go back to old framework code.
> That is what NIFI-2854 tackles at least as far as the
> content/prov/flowfile repositories go.  Now, the code and
> serialization is done in such a way that older version can simply
> ignore what never versions encoded if they don't understand it but
> they should be able to continue on.
>
> I just tried out a 1.0.0 flow with data queued up.  Upgraded to a
> latest NiFi 1.1.0-SNAPSHOT.  Moved the repos over.  And it came up
> perfectly with all the queue data ready to roll.
>
> Thanks
> Joe
>
> On Tue, Nov 22, 2016 at 12:16 PM, Michael Moser  wrote:
>> After seeing "Repositories now support rollback" in the release notes and
>> reading NIFI-2854 [1], I have a question.
>>
>> Are repositories created using NiFi 1.0.0 compatible with NiFi 1.1.0
>> software?  This is the goal that the ticket seems to indicate with 1.1.0
>> onward, but it's not clear whether 1.0.0 -> 1.1.0 is included.
>>
>> Thanks,
>> -- Mike
>>
>> [1] - https://issues.apache.org/jira/browse/NIFI-2854
>>
>> On Tue, Nov 22, 2016 at 12:08 PM, James Wing  wrote:
>>
>>> Going through the tickets, it seems like quite a release.  A few more
>>> things for your list:
>>>
>>> GenerateFlowFile updated to support literal/expression content and
>>> attributes
>>>
>>> AWS-related:
>>> * New processors PutCloudWatchMetric, PutKinesisStream
>>> * Updated processors PutS3Object (content type, signer options), ListS3
>>> (performance, versions)
>>> * Added support for AWS assume role credentials with proxy
>>>
>>>
>>> Thanks,
>>>
>>> James
>>>
>>> On Mon, Nov 21, 2016 at 9:30 PM, Joe Witt  wrote:
>>>
>>> > matt
>>> >
>>> > i'll add a wiki page or set of instructions linked from the release
>>> notes.
>>> >
>>> > all,
>>> >
>>> > walked through the 250 or so JIRAs in the 1.1.0 release and pulled out
>>> > highlights.  The items noted are as follows.  Will likely reduce this
>>> > down further for the release notes but wanted to put this out in case
>>> > folks have things they think are really important to highlight.
>>> >
>>> > - Core Improvements:
>>> >- Performance: Session Migration
>>> >- Stability: Cluster Management
>>> >- Developer: Framework supports easy user driven classloader extension
>>> >- Expression Language: Now supports base64 and hex encoded values
>>> > and Math functions
>>> >- Repositories now support rollback
>>> >- Faster startup due to more efficient state restoration algorithm
>>> > - UX Improvements:
>>> >- Visual Backpressure Indicator
>>> >- Introduced more colors to better highlight actions and components
>>> >- Performance: Validate non-running components
>>> >- Provenance graph image can be exported
>>> >- Cron Scheduling for Primary node tasks now supported
>>> > - Updated versions
>>> >- Azure Event Hub 0.9.0
>>> >- Spark 2.0.1
>>> >- Hadoop 2.7.x
>>> > - New/Improved Processors
>>> >- new Fetch/Put Elastic Search 5.0
>>> >- new ParseCEF to parse CEF formatted logs
>>> >- improve ExtractEmail now supports TNEF files
>>> >- new Validate CSV
>>> >- improved Solr processors now support SSL and Kerberos
>>> >- new Websocket client and server processors
>>> > - New Utility
>>> >- Zookeeper Migrator (move from one zookeeper to another)
>>> > - Security
>>> >- Restricted Processors
>>> >- Site-to-site now supports port forwarding
>>> >- Improved Policy Management UX
>>> > - Migration Notes:
>>> >- Restricted Processors
>>> >- Twitter Processor Removed
>>> >
>>> > On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess 
>>> > wrote:
>>> > > Is there a good spot for us to put instructions on how to build the
>>> > > Twitter processor and/or the Social Media NAR in the meantime? Maybe a
>>> > > Wiki page or something simple to say "go to this directory, run this
>>> > > Maven command, drop the NAR into your deployment..." ?
>>> > >
>>> > > On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt  wrote:
>>> > >> Team,
>>> > >>
>>> > >> We appear to be very close.  Andy is working NIFI-3024 but otherwise
>>> > >> it is focus on 

Re: [DISCUSS] NiFi 1.1.0 release

[Joe Witt]

Thanks James.  Will add the AWS stuff now.  The generate flow file
I'll leave off since it is mostly a developer/debug tool unless you
disagree.

Thanks
Joe

On Tue, Nov 22, 2016 at 12:08 PM, James Wing  wrote:
> Going through the tickets, it seems like quite a release.  A few more
> things for your list:
>
> GenerateFlowFile updated to support literal/expression content and
> attributes
>
> AWS-related:
> * New processors PutCloudWatchMetric, PutKinesisStream
> * Updated processors PutS3Object (content type, signer options), ListS3
> (performance, versions)
> * Added support for AWS assume role credentials with proxy
>
>
> Thanks,
>
> James
>
> On Mon, Nov 21, 2016 at 9:30 PM, Joe Witt  wrote:
>
>> matt
>>
>> i'll add a wiki page or set of instructions linked from the release notes.
>>
>> all,
>>
>> walked through the 250 or so JIRAs in the 1.1.0 release and pulled out
>> highlights.  The items noted are as follows.  Will likely reduce this
>> down further for the release notes but wanted to put this out in case
>> folks have things they think are really important to highlight.
>>
>> - Core Improvements:
>>- Performance: Session Migration
>>- Stability: Cluster Management
>>- Developer: Framework supports easy user driven classloader extension
>>- Expression Language: Now supports base64 and hex encoded values
>> and Math functions
>>- Repositories now support rollback
>>- Faster startup due to more efficient state restoration algorithm
>> - UX Improvements:
>>- Visual Backpressure Indicator
>>- Introduced more colors to better highlight actions and components
>>- Performance: Validate non-running components
>>- Provenance graph image can be exported
>>- Cron Scheduling for Primary node tasks now supported
>> - Updated versions
>>- Azure Event Hub 0.9.0
>>- Spark 2.0.1
>>- Hadoop 2.7.x
>> - New/Improved Processors
>>- new Fetch/Put Elastic Search 5.0
>>- new ParseCEF to parse CEF formatted logs
>>- improve ExtractEmail now supports TNEF files
>>- new Validate CSV
>>- improved Solr processors now support SSL and Kerberos
>>- new Websocket client and server processors
>> - New Utility
>>- Zookeeper Migrator (move from one zookeeper to another)
>> - Security
>>- Restricted Processors
>>- Site-to-site now supports port forwarding
>>- Improved Policy Management UX
>> - Migration Notes:
>>- Restricted Processors
>>- Twitter Processor Removed
>>
>> On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess 
>> wrote:
>> > Is there a good spot for us to put instructions on how to build the
>> > Twitter processor and/or the Social Media NAR in the meantime? Maybe a
>> > Wiki page or something simple to say "go to this directory, run this
>> > Maven command, drop the NAR into your deployment..." ?
>> >
>> > On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt  wrote:
>> >> Team,
>> >>
>> >> We appear to be very close.  Andy is working NIFI-3024 but otherwise
>> >> it is focus on testing.
>> >>
>> >> I'm going to prep the RC and release notes now.  Unfortunately the
>> >> twitter changes for json.org will need to remain.  Consensus forming
>> >> on the legal-discuss thread regarding a grace period has been elusive
>> >> and we're already prepared to make the right steps so we'll just need
>> >> to take that on by being empathetic to the user base.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Thu, Nov 17, 2016 at 7:37 AM, Andre  wrote:
>> >>> Andy,
>> >>>
>> >>> Great to see NIFI-3050 implemented and certainly good news that NiFi
>> 1.1.0
>> >>> is set to include a number of security related improvements.
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto 
>> wrote:
>> >>>
>>  Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have
>> been
>>  added to my plate for this release. Coordinated with Joe Witt and they
>>  should both be included.
>> 
>>  [1] https://issues.apache.org/jira/browse/NIFI-3050
>>  [2] https://issues.apache.org/jira/browse/NIFI-3051
>> 
>>  Andy LoPresto
>>  alopre...@apache.org
>>  *alopresto.apa...@gmail.com *
>>  PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>>  On Nov 16, 2016, at 12:08 PM, Joe Witt  wrote:
>> 
>>  Team
>> 
>>  There is a thread on apache legal-discuss that might allow for a
>>  graceperiod of continued usage of the json library.  Am going to keep
>>  a close eye on this and if VP Legal approves we'll be able to keep the
>>  twitter processors in which is definitely a good thing.  Will advise
>> 
>>  Thanks
>>  Joe
>> 
>>  On Wed, Nov 16, 2016 at 10:37 AM, Bryan Bende 
>> wrote:
>> 
>>  I've noticed an issue with the per-instance class 

Re: [DISCUSS] NiFi 1.1.0 release

[Michael Moser]

After seeing "Repositories now support rollback" in the release notes and
reading NIFI-2854 [1], I have a question.

Are repositories created using NiFi 1.0.0 compatible with NiFi 1.1.0
software?  This is the goal that the ticket seems to indicate with 1.1.0
onward, but it's not clear whether 1.0.0 -> 1.1.0 is included.

Thanks,
-- Mike

[1] - https://issues.apache.org/jira/browse/NIFI-2854

On Tue, Nov 22, 2016 at 12:08 PM, James Wing  wrote:

> Going through the tickets, it seems like quite a release.  A few more
> things for your list:
>
> GenerateFlowFile updated to support literal/expression content and
> attributes
>
> AWS-related:
> * New processors PutCloudWatchMetric, PutKinesisStream
> * Updated processors PutS3Object (content type, signer options), ListS3
> (performance, versions)
> * Added support for AWS assume role credentials with proxy
>
>
> Thanks,
>
> James
>
> On Mon, Nov 21, 2016 at 9:30 PM, Joe Witt  wrote:
>
> > matt
> >
> > i'll add a wiki page or set of instructions linked from the release
> notes.
> >
> > all,
> >
> > walked through the 250 or so JIRAs in the 1.1.0 release and pulled out
> > highlights.  The items noted are as follows.  Will likely reduce this
> > down further for the release notes but wanted to put this out in case
> > folks have things they think are really important to highlight.
> >
> > - Core Improvements:
> >- Performance: Session Migration
> >- Stability: Cluster Management
> >- Developer: Framework supports easy user driven classloader extension
> >- Expression Language: Now supports base64 and hex encoded values
> > and Math functions
> >- Repositories now support rollback
> >- Faster startup due to more efficient state restoration algorithm
> > - UX Improvements:
> >- Visual Backpressure Indicator
> >- Introduced more colors to better highlight actions and components
> >- Performance: Validate non-running components
> >- Provenance graph image can be exported
> >- Cron Scheduling for Primary node tasks now supported
> > - Updated versions
> >- Azure Event Hub 0.9.0
> >- Spark 2.0.1
> >- Hadoop 2.7.x
> > - New/Improved Processors
> >- new Fetch/Put Elastic Search 5.0
> >- new ParseCEF to parse CEF formatted logs
> >- improve ExtractEmail now supports TNEF files
> >- new Validate CSV
> >- improved Solr processors now support SSL and Kerberos
> >- new Websocket client and server processors
> > - New Utility
> >- Zookeeper Migrator (move from one zookeeper to another)
> > - Security
> >- Restricted Processors
> >- Site-to-site now supports port forwarding
> >- Improved Policy Management UX
> > - Migration Notes:
> >- Restricted Processors
> >- Twitter Processor Removed
> >
> > On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess 
> > wrote:
> > > Is there a good spot for us to put instructions on how to build the
> > > Twitter processor and/or the Social Media NAR in the meantime? Maybe a
> > > Wiki page or something simple to say "go to this directory, run this
> > > Maven command, drop the NAR into your deployment..." ?
> > >
> > > On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt  wrote:
> > >> Team,
> > >>
> > >> We appear to be very close.  Andy is working NIFI-3024 but otherwise
> > >> it is focus on testing.
> > >>
> > >> I'm going to prep the RC and release notes now.  Unfortunately the
> > >> twitter changes for json.org will need to remain.  Consensus forming
> > >> on the legal-discuss thread regarding a grace period has been elusive
> > >> and we're already prepared to make the right steps so we'll just need
> > >> to take that on by being empathetic to the user base.
> > >>
> > >> Thanks
> > >> Joe
> > >>
> > >> On Thu, Nov 17, 2016 at 7:37 AM, Andre  wrote:
> > >>> Andy,
> > >>>
> > >>> Great to see NIFI-3050 implemented and certainly good news that NiFi
> > 1.1.0
> > >>> is set to include a number of security related improvements.
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto  >
> > wrote:
> > >>>
> >  Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have
> > been
> >  added to my plate for this release. Coordinated with Joe Witt and
> they
> >  should both be included.
> > 
> >  [1] https://issues.apache.org/jira/browse/NIFI-3050
> >  [2] https://issues.apache.org/jira/browse/NIFI-3051
> > 
> >  Andy LoPresto
> >  alopre...@apache.org
> >  *alopresto.apa...@gmail.com *
> >  PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > 
> >  On Nov 16, 2016, at 12:08 PM, Joe Witt  wrote:
> > 
> >  Team
> > 
> >  There is a thread on apache legal-discuss that might allow for a
> >  graceperiod of continued usage of the json library.  Am going to
> keep
> > 

Re: [DISCUSS] NiFi 1.1.0 release

[James Wing]

Going through the tickets, it seems like quite a release.  A few more
things for your list:

GenerateFlowFile updated to support literal/expression content and
attributes

AWS-related:
* New processors PutCloudWatchMetric, PutKinesisStream
* Updated processors PutS3Object (content type, signer options), ListS3
(performance, versions)
* Added support for AWS assume role credentials with proxy


Thanks,

James

On Mon, Nov 21, 2016 at 9:30 PM, Joe Witt  wrote:

> matt
>
> i'll add a wiki page or set of instructions linked from the release notes.
>
> all,
>
> walked through the 250 or so JIRAs in the 1.1.0 release and pulled out
> highlights.  The items noted are as follows.  Will likely reduce this
> down further for the release notes but wanted to put this out in case
> folks have things they think are really important to highlight.
>
> - Core Improvements:
>- Performance: Session Migration
>- Stability: Cluster Management
>- Developer: Framework supports easy user driven classloader extension
>- Expression Language: Now supports base64 and hex encoded values
> and Math functions
>- Repositories now support rollback
>- Faster startup due to more efficient state restoration algorithm
> - UX Improvements:
>- Visual Backpressure Indicator
>- Introduced more colors to better highlight actions and components
>- Performance: Validate non-running components
>- Provenance graph image can be exported
>- Cron Scheduling for Primary node tasks now supported
> - Updated versions
>- Azure Event Hub 0.9.0
>- Spark 2.0.1
>- Hadoop 2.7.x
> - New/Improved Processors
>- new Fetch/Put Elastic Search 5.0
>- new ParseCEF to parse CEF formatted logs
>- improve ExtractEmail now supports TNEF files
>- new Validate CSV
>- improved Solr processors now support SSL and Kerberos
>- new Websocket client and server processors
> - New Utility
>- Zookeeper Migrator (move from one zookeeper to another)
> - Security
>- Restricted Processors
>- Site-to-site now supports port forwarding
>- Improved Policy Management UX
> - Migration Notes:
>- Restricted Processors
>- Twitter Processor Removed
>
> On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess 
> wrote:
> > Is there a good spot for us to put instructions on how to build the
> > Twitter processor and/or the Social Media NAR in the meantime? Maybe a
> > Wiki page or something simple to say "go to this directory, run this
> > Maven command, drop the NAR into your deployment..." ?
> >
> > On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt  wrote:
> >> Team,
> >>
> >> We appear to be very close.  Andy is working NIFI-3024 but otherwise
> >> it is focus on testing.
> >>
> >> I'm going to prep the RC and release notes now.  Unfortunately the
> >> twitter changes for json.org will need to remain.  Consensus forming
> >> on the legal-discuss thread regarding a grace period has been elusive
> >> and we're already prepared to make the right steps so we'll just need
> >> to take that on by being empathetic to the user base.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Thu, Nov 17, 2016 at 7:37 AM, Andre  wrote:
> >>> Andy,
> >>>
> >>> Great to see NIFI-3050 implemented and certainly good news that NiFi
> 1.1.0
> >>> is set to include a number of security related improvements.
> >>>
> >>>
> >>>
> >>> On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto 
> wrote:
> >>>
>  Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have
> been
>  added to my plate for this release. Coordinated with Joe Witt and they
>  should both be included.
> 
>  [1] https://issues.apache.org/jira/browse/NIFI-3050
>  [2] https://issues.apache.org/jira/browse/NIFI-3051
> 
>  Andy LoPresto
>  alopre...@apache.org
>  *alopresto.apa...@gmail.com *
>  PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>  On Nov 16, 2016, at 12:08 PM, Joe Witt  wrote:
> 
>  Team
> 
>  There is a thread on apache legal-discuss that might allow for a
>  graceperiod of continued usage of the json library.  Am going to keep
>  a close eye on this and if VP Legal approves we'll be able to keep the
>  twitter processors in which is definitely a good thing.  Will advise
> 
>  Thanks
>  Joe
> 
>  On Wed, Nov 16, 2016 at 10:37 AM, Bryan Bende 
> wrote:
> 
>  I've noticed an issue with the per-instance class loading capability
>  introduced in NIFI-2909 where the additional classpath resources can
> get
>  incorrectly removed from the class loader.
> 
>  I was able to reproduced this with a unit test and have a fix ready. I
>  believe this is important and needs to go in for the 1.1 release,
> going to
>  re-open NIFI-2909 and submit a PR shortly.
> 

Re: [DISCUSS] NiFi 1.1.0 release

[Michael Moser]

Awesome.  Your prompt and detailed feedback is very much appreciated!


On Tue, Nov 22, 2016 at 12:47 PM, Joe Witt  wrote:

> mike - also to clarify based on what you brought up I updated the JIRA
> description as follows
>
> The following also would be true:
> 1) Apache NiFi 1.0.0 repositories should work just fine when applied
> to an Apache NiFi 1.1.0 installation.
>
> 2) Repositories made/updated in Apache NiFi 1.1.0 onward would not
> work in older Apache NiFi releases (such as 1.0.0)
>
> On Tue, Nov 22, 2016 at 12:33 PM, Joe Witt  wrote:
> > Mike - yeah good catch and good question.  It does support the old
> > format.  We've always been pretty good about being diligent to allow
> > folks to upgrade and it honor existing state and in many cases
> > configurations and it would automatically port them over.  What has
> > always been far more problematic is 'rollback' where people ran on
> > newer configurations but could not then go back to old framework code.
> > That is what NIFI-2854 tackles at least as far as the
> > content/prov/flowfile repositories go.  Now, the code and
> > serialization is done in such a way that older version can simply
> > ignore what never versions encoded if they don't understand it but
> > they should be able to continue on.
> >
> > I just tried out a 1.0.0 flow with data queued up.  Upgraded to a
> > latest NiFi 1.1.0-SNAPSHOT.  Moved the repos over.  And it came up
> > perfectly with all the queue data ready to roll.
> >
> > Thanks
> > Joe
> >
> > On Tue, Nov 22, 2016 at 12:16 PM, Michael Moser 
> wrote:
> >> After seeing "Repositories now support rollback" in the release notes
> and
> >> reading NIFI-2854 [1], I have a question.
> >>
> >> Are repositories created using NiFi 1.0.0 compatible with NiFi 1.1.0
> >> software?  This is the goal that the ticket seems to indicate with 1.1.0
> >> onward, but it's not clear whether 1.0.0 -> 1.1.0 is included.
> >>
> >> Thanks,
> >> -- Mike
> >>
> >> [1] - https://issues.apache.org/jira/browse/NIFI-2854
> >>
> >> On Tue, Nov 22, 2016 at 12:08 PM, James Wing  wrote:
> >>
> >>> Going through the tickets, it seems like quite a release.  A few more
> >>> things for your list:
> >>>
> >>> GenerateFlowFile updated to support literal/expression content and
> >>> attributes
> >>>
> >>> AWS-related:
> >>> * New processors PutCloudWatchMetric, PutKinesisStream
> >>> * Updated processors PutS3Object (content type, signer options), ListS3
> >>> (performance, versions)
> >>> * Added support for AWS assume role credentials with proxy
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> James
> >>>
> >>> On Mon, Nov 21, 2016 at 9:30 PM, Joe Witt  wrote:
> >>>
> >>> > matt
> >>> >
> >>> > i'll add a wiki page or set of instructions linked from the release
> >>> notes.
> >>> >
> >>> > all,
> >>> >
> >>> > walked through the 250 or so JIRAs in the 1.1.0 release and pulled
> out
> >>> > highlights.  The items noted are as follows.  Will likely reduce this
> >>> > down further for the release notes but wanted to put this out in case
> >>> > folks have things they think are really important to highlight.
> >>> >
> >>> > - Core Improvements:
> >>> >- Performance: Session Migration
> >>> >- Stability: Cluster Management
> >>> >- Developer: Framework supports easy user driven classloader
> extension
> >>> >- Expression Language: Now supports base64 and hex encoded values
> >>> > and Math functions
> >>> >- Repositories now support rollback
> >>> >- Faster startup due to more efficient state restoration algorithm
> >>> > - UX Improvements:
> >>> >- Visual Backpressure Indicator
> >>> >- Introduced more colors to better highlight actions and
> components
> >>> >- Performance: Validate non-running components
> >>> >- Provenance graph image can be exported
> >>> >- Cron Scheduling for Primary node tasks now supported
> >>> > - Updated versions
> >>> >- Azure Event Hub 0.9.0
> >>> >- Spark 2.0.1
> >>> >- Hadoop 2.7.x
> >>> > - New/Improved Processors
> >>> >- new Fetch/Put Elastic Search 5.0
> >>> >- new ParseCEF to parse CEF formatted logs
> >>> >- improve ExtractEmail now supports TNEF files
> >>> >- new Validate CSV
> >>> >- improved Solr processors now support SSL and Kerberos
> >>> >- new Websocket client and server processors
> >>> > - New Utility
> >>> >- Zookeeper Migrator (move from one zookeeper to another)
> >>> > - Security
> >>> >- Restricted Processors
> >>> >- Site-to-site now supports port forwarding
> >>> >- Improved Policy Management UX
> >>> > - Migration Notes:
> >>> >- Restricted Processors
> >>> >- Twitter Processor Removed
> >>> >
> >>> > On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess  >
> >>> > wrote:
> >>> > > Is there a good spot for us to put instructions on how to build the
> >>> > > Twitter processor and/or the Social 

Re: [DISCUSS] NiFi 1.1.0 release

[Joe Gresock]

I'm biased, but I think the new Query / Scroll Elasticsearch processors are
pretty big ;)

On Tue, Nov 22, 2016 at 9:02 AM, Andy LoPresto  wrote:

> Important to call out in the security section that
> login-identity-providers.xml is now covered by encrypted config, and
> flow.xml.gz processor property encryption/nifi.sensitive.props.key value
> can now be migrated by the command-line tool (second part pending
> completion of NIFI-3024).
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 21, 2016, at 9:30 PM, Joe Witt  wrote:
>
> matt
>
> i'll add a wiki page or set of instructions linked from the release notes.
>
> all,
>
> walked through the 250 or so JIRAs in the 1.1.0 release and pulled out
> highlights.  The items noted are as follows.  Will likely reduce this
> down further for the release notes but wanted to put this out in case
> folks have things they think are really important to highlight.
>
> - Core Improvements:
>   - Performance: Session Migration
>   - Stability: Cluster Management
>   - Developer: Framework supports easy user driven classloader extension
>   - Expression Language: Now supports base64 and hex encoded values
> and Math functions
>   - Repositories now support rollback
>   - Faster startup due to more efficient state restoration algorithm
> - UX Improvements:
>   - Visual Backpressure Indicator
>   - Introduced more colors to better highlight actions and components
>   - Performance: Validate non-running components
>   - Provenance graph image can be exported
>   - Cron Scheduling for Primary node tasks now supported
> - Updated versions
>   - Azure Event Hub 0.9.0
>   - Spark 2.0.1
>   - Hadoop 2.7.x
> - New/Improved Processors
>   - new Fetch/Put Elastic Search 5.0
>   - new ParseCEF to parse CEF formatted logs
>   - improve ExtractEmail now supports TNEF files
>   - new Validate CSV
>   - improved Solr processors now support SSL and Kerberos
>   - new Websocket client and server processors
> - New Utility
>   - Zookeeper Migrator (move from one zookeeper to another)
> - Security
>   - Restricted Processors
>   - Site-to-site now supports port forwarding
>   - Improved Policy Management UX
> - Migration Notes:
>   - Restricted Processors
>   - Twitter Processor Removed
>
> On Mon, Nov 21, 2016 at 10:17 PM, Matt Burgess 
> wrote:
>
> Is there a good spot for us to put instructions on how to build the
> Twitter processor and/or the Social Media NAR in the meantime? Maybe a
> Wiki page or something simple to say "go to this directory, run this
> Maven command, drop the NAR into your deployment..." ?
>
> On Mon, Nov 21, 2016 at 9:34 PM, Joe Witt  wrote:
>
> Team,
>
> We appear to be very close.  Andy is working NIFI-3024 but otherwise
> it is focus on testing.
>
> I'm going to prep the RC and release notes now.  Unfortunately the
> twitter changes for json.org will need to remain.  Consensus forming
> on the legal-discuss thread regarding a grace period has been elusive
> and we're already prepared to make the right steps so we'll just need
> to take that on by being empathetic to the user base.
>
> Thanks
> Joe
>
> On Thu, Nov 17, 2016 at 7:37 AM, Andre  wrote:
>
> Andy,
>
> Great to see NIFI-3050 implemented and certainly good news that NiFi 1.1.0
> is set to include a number of security related improvements.
>
>
>
> On Thu, Nov 17, 2016 at 2:38 PM, Andy LoPresto 
> wrote:
>
> Just updating this thread that NIFI-3050 [1] and NIFI-3051 [2] have been
> added to my plate for this release. Coordinated with Joe Witt and they
> should both be included.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3050
> [2] https://issues.apache.org/jira/browse/NIFI-3051
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 16, 2016, at 12:08 PM, Joe Witt  wrote:
>
> Team
>
> There is a thread on apache legal-discuss that might allow for a
> graceperiod of continued usage of the json library.  Am going to keep
> a close eye on this and if VP Legal approves we'll be able to keep the
> twitter processors in which is definitely a good thing.  Will advise
>
> Thanks
> Joe
>
> On Wed, Nov 16, 2016 at 10:37 AM, Bryan Bende  wrote:
>
> I've noticed an issue with the per-instance class loading capability
> introduced in NIFI-2909 where the additional classpath resources can get
> incorrectly removed from the class loader.
>
> I was able to reproduced this with a unit test and have a fix ready. I
> believe this is important and needs to go in for the 1.1 release, going to
> re-open NIFI-2909 and submit a PR shortly.
>
> -Bryan
>
> On Wed, Nov 16, 2016 at 8:11 AM, Matt Gilman