Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andrew Grande]

I am observing one assumption in this thread. For some reason we are
implying all these will be hadoop compatible file systems. They don't
always have an HDFS plugin, nor should they as a mandatory requirement.
Untie completely from the Hadoop nar. This allows for effective minifi
interaction without the weight of hadoop libs for example. Massive size
savings where it matters.

For the deployment, it's easy enough for an admin to either rely on a
standard tar or rpm if the NAR modules are already available in the distro
(well, I won't talk registry till it arrives). Mounting a common directory
on every node or distributing additional jars everywhere, plus configs, and
then keeping it consistent across is something which can be avoided by
simpler packaging.

Andrew

On Tue, Feb 21, 2017, 6:47 PM Andre  wrote:

> Andrew,
>
> Thank you for contributing.
>
> On 22 Feb 2017 10:21 AM, "Andrew Grande"  wrote:
>
> Andre,
>
> I came across multiple NiFi use cases where going through the HDFS layer
> and the fs plugin may not be possible. I.e. when no HDFS layer present at
> all, so no NN to connect to.
>
>
> Not sure I understand what you mean.
>
>
> Another important aspect is operations. Current PutHDFS model with
> additional jar location, well, it kinda works, but I very much dislike it.
> Too many possibilities for a human error in addition to deployment pain,
> especially in a cluster.
>
>
> Fair enough. Would you mind expanding a bit on what sort of  challenges
> currently apply in terms of cluster deployment?
>
>
> Finally, native object storage processors have features which may not even
> apply to the HDFS layer. E.g. the Azure storage has Table storage, etc.
>
>
> This is a very valid point but I am sure exceptions (in this case a NoSQL
> DB operating under the umbrella term of "storage").
>
> I perhaps should have made it more explicit but the requirements are:
>
> - existence of a hadoop compatible interface
> - ability to handle files
>
> Again, thank you for the input, truly appreciated.
>
> Andre
>
> I agree consolidating various efforts is worthwhile, but only within a
> context of a specific storage solution. Not 'unifying' them into a single
> layer.
>
> Andrew
>
> On Tue, Feb 21, 2017, 6:10 PM Andre  wrote:
>
> > dev,
> >
> > I was having a chat with Pierre around PR#379 and we thought it would be
> > worth sharing this with the wider group:
> >
> >
> > I recently noticed that we merged a number of PRs and merges around
> > scale-out/cloud based object store into the master.
> >
> > Would it make sense to start considering adopting a pattern where
> > Put/Get/ListHDFS are used in tandem with implementations of the
> > hadoop.filesystem interfaces instead of creating new processors, except
> > where a particular deficiency/incompatibility in the hadoop.filesystem
> > implementation exists?
> >
> > Candidates for removal / non merge would be:
> >
> > - Alluxio (PR#379)
> > - WASB (PR#626)
> >  - Azure* (PR#399)
> > - *GCP (recently merged as PR#1482)
> > - *S3 (although this has been in code so it would have to be deprecated)
> >
> > The pattern would be pretty much the same as the one documented and
> > successfully deployed here:
> >
> > https://community.hortonworks.com/articles/71916/connecting-
> > to-azure-data-lake-from-a-nifi-dataflow.html
> >
> > Which means that in the case of Alluxio, one would use the properties
> > documented here:
> >
> > https://www.alluxio.com/docs/community/1.3/en/Running-
> > Hadoop-MapReduce-on-Alluxio.html
> >
> > While with Google Cloud Storage we would use the properties documented
> > here:
> >
> > https://cloud.google.com/hadoop/google-cloud-storage-connector
> >
> > I noticed that specific processors could have the ability to handle
> > particular properties to a filesystem, however I would like to believe
> the
> > same issue would plague hadoop users, and therefore is reasonable to
> > believe the Hadoop compatible implementations would have ways of exposing
> > those properties as well?
> >
> > In the case the properties are exposed, we perhaps simply adjust the
> *HDFS
> > processors to use dynamic properties to pass those to the underlying
> > module, therefore providing a way to explore particular settings of an
> > underlying storage platforms.
> >
> > Any opinion would be welcome
> >
> > PS-sent it again with proper subject label
> >
>

Re: nar deployment

[Joe Witt]

I'd be interested to see the full nifi-app.log of startup.

There should be at least something in there about this processor.

Thanks
Joe

On Tue, Feb 21, 2017 at 7:19 PM, Andy LoPresto  wrote:
> Arthur,
>
> Sorry to hear this isn’t working for you. You are following the best
> practices for custom NAR deployment; it should be as easy as copying the nar
> file into the lib/ directory and restarting NiFi.
>
> Can you post the code (I understand if you can’t share the Java code, but at
> least the pom.xml and META-INF/services/org.apache.nifi.processor.Processor
> file (which should contain the fully-qualified class name of each
> processor))?
>
> When NiFi starts, the logs/nifi-app.log file will contain a list of each NAR
> that was loaded. Can you confirm that your NAR is not listed there?
>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 21, 2017, at 3:39 PM, Bleeker, Arthur H 
> wrote:
>
> Hi,
>
> About six months ago I created custom processors for NiFi. Recently I needed
> to dust off the code and make some changes. I pushed the updated nar to my
> local development nifi instance and... nothing. The changes didn't get
> picked up. So I deleted the nar, restarted nifi, copied the new nar in, and
> then restarted nifi again. The processors disappeared.
>
> I have been unable to get any nar to deploy since then. I have even tried a
> 100% clean nifi instance version 1.1.1 with a maven archetype nar processor
> project. Nothing. I can see the nar in the lib directory. I can see that it
> is intact. But it does not deploy. There's nothing in the logs indicating an
> exception or warning of any kind. Hopefully someone can help me figure this
> out.
>
> Thanks,
> 
> Arthur Bleeker
> Software Architect and Developer
> Pacific Northwest National Laboratory
>
>
>

Re: nar deployment

[Andy LoPresto]

Arthur,

Sorry to hear this isn’t working for you. You are following the best practices 
for custom NAR deployment; it should be as easy as copying the nar file into 
the lib/ directory and restarting NiFi.

Can you post the code (I understand if you can’t share the Java code, but at 
least the pom.xml and META-INF/services/org.apache.nifi.processor.Processor 
file (which should contain the fully-qualified class name of each processor))?

When NiFi starts, the logs/nifi-app.log file will contain a list of each NAR 
that was loaded. Can you confirm that your NAR is not listed there?


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 21, 2017, at 3:39 PM, Bleeker, Arthur H  
> wrote:
> 
> Hi,
> 
> About six months ago I created custom processors for NiFi. Recently I needed 
> to dust off the code and make some changes. I pushed the updated nar to my 
> local development nifi instance and... nothing. The changes didn't get picked 
> up. So I deleted the nar, restarted nifi, copied the new nar in, and then 
> restarted nifi again. The processors disappeared.
> 
> I have been unable to get any nar to deploy since then. I have even tried a 
> 100% clean nifi instance version 1.1.1 with a maven archetype nar processor 
> project. Nothing. I can see the nar in the lib directory. I can see that it 
> is intact. But it does not deploy. There's nothing in the logs indicating an 
> exception or warning of any kind. Hopefully someone can help me figure this 
> out.
> 
> Thanks,
> 
> Arthur Bleeker
> Software Architect and Developer
> Pacific Northwest National Laboratory
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Jeremy Dyer]

Congratulations Jeff! Good to have you aboard!

On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
> Storck has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> generous contributions and look forward to continued involvement in the
> project.
>
> Jeff's contributions include significant efforts toward upgrade and
> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
> and the ZooKeeper migration toolkit.
>
> Congrats, Jeff!
>

[ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Aldrin Piri]

On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff 
Storck has accepted the PMC's invitation to become a committer on the Apache 
NiFi project. We greatly appreciate all of Jeff's hard work and generous 
contributions and look forward to continued involvement in the project.

Jeff's contributions include significant efforts toward upgrade and migration 
processes inclusive of flow layout when upgrading from 0.x to 1.x and the 
ZooKeeper migration toolkit.

Congrats, Jeff!


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Rob Moran]

Thanks and congrats, Jeff!

Rob

On Tue, Feb 21, 2017 at 3:01 PM, Pierre Villard  wrote:

> Congrats !
>
> 2017-02-21 20:58 GMT+01:00 Joe Witt :
>
> > Congrats Jeff!
> >
> > On Tue, Feb 21, 2017 at 2:43 PM, Jeremy Dyer  wrote:
> > > Congratulations Jeff! Good to have you aboard!
> > >
> > > On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri 
> wrote:
> > >
> > >> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Jeff
> > >> Storck has accepted the PMC's invitation to become a committer on the
> > >> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> > >> generous contributions and look forward to continued involvement in
> the
> > >> project.
> > >>
> > >> Jeff's contributions include significant efforts toward upgrade and
> > >> migration processes inclusive of flow layout when upgrading from 0.x
> to
> > 1.x
> > >> and the ZooKeeper migration toolkit.
> > >>
> > >> Congrats, Jeff!
> > >>
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Matt Burgess]

Congrats Jeff, well deserved.  Welcome aboard!

On Tue, Feb 21, 2017 at 3:05 PM, Rob Moran  wrote:
> Thanks and congrats, Jeff!
>
> Rob
>
> On Tue, Feb 21, 2017 at 3:01 PM, Pierre Villard > wrote:
>
>> Congrats !
>>
>> 2017-02-21 20:58 GMT+01:00 Joe Witt :
>>
>> > Congrats Jeff!
>> >
>> > On Tue, Feb 21, 2017 at 2:43 PM, Jeremy Dyer  wrote:
>> > > Congratulations Jeff! Good to have you aboard!
>> > >
>> > > On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri 
>> wrote:
>> > >
>> > >> On behalf of the Apache NiFI PMC, I am very pleased to announce that
>> > Jeff
>> > >> Storck has accepted the PMC's invitation to become a committer on the
>> > >> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
>> > >> generous contributions and look forward to continued involvement in
>> the
>> > >> project.
>> > >>
>> > >> Jeff's contributions include significant efforts toward upgrade and
>> > >> migration processes inclusive of flow layout when upgrading from 0.x
>> to
>> > 1.x
>> > >> and the ZooKeeper migration toolkit.
>> > >>
>> > >> Congrats, Jeff!
>> > >>
>> >
>>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[James Wing]

Congratulations, Jeff!

On Tue, Feb 21, 2017 at 11:41 AM, Aldrin Piri  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
> Storck has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> generous contributions and look forward to continued involvement in the
> project.
>
> Jeff's contributions include significant efforts toward upgrade and
> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
> and the ZooKeeper migration toolkit.
>
> Congrats, Jeff!
>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Bryan Bende]

Congrats Jeff!

On Tue, Feb 21, 2017 at 3:18 PM, James Wing  wrote:
> Congratulations, Jeff!
>
> On Tue, Feb 21, 2017 at 11:41 AM, Aldrin Piri  wrote:
>
>> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
>> Storck has accepted the PMC's invitation to become a committer on the
>> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
>> generous contributions and look forward to continued involvement in the
>> project.
>>
>> Jeff's contributions include significant efforts toward upgrade and
>> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
>> and the ZooKeeper migration toolkit.
>>
>> Congrats, Jeff!
>>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Yolanda Davis]

Congratulations Jeff!

On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
> Storck has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> generous contributions and look forward to continued involvement in the
> project.
>
> Jeff's contributions include significant efforts toward upgrade and
> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
> and the ZooKeeper migration toolkit.
>
> Congrats, Jeff!
>



-- 
--
yolanda.m.da...@gmail.com
@YolandaMDavis

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andrew Grande]

Andre,

I came across multiple NiFi use cases where going through the HDFS layer
and the fs plugin may not be possible. I.e. when no HDFS layer present at
all, so no NN to connect to.

Another important aspect is operations. Current PutHDFS model with
additional jar location, well, it kinda works, but I very much dislike it.
Too many possibilities for a human error in addition to deployment pain,
especially in a cluster.

Finally, native object storage processors have features which may not even
apply to the HDFS layer. E.g. the Azure storage has Table storage, etc.

I agree consolidating various efforts is worthwhile, but only within a
context of a specific storage solution. Not 'unifying' them into a single
layer.

Andrew

On Tue, Feb 21, 2017, 6:10 PM Andre  wrote:

> dev,
>
> I was having a chat with Pierre around PR#379 and we thought it would be
> worth sharing this with the wider group:
>
>
> I recently noticed that we merged a number of PRs and merges around
> scale-out/cloud based object store into the master.
>
> Would it make sense to start considering adopting a pattern where
> Put/Get/ListHDFS are used in tandem with implementations of the
> hadoop.filesystem interfaces instead of creating new processors, except
> where a particular deficiency/incompatibility in the hadoop.filesystem
> implementation exists?
>
> Candidates for removal / non merge would be:
>
> - Alluxio (PR#379)
> - WASB (PR#626)
>  - Azure* (PR#399)
> - *GCP (recently merged as PR#1482)
> - *S3 (although this has been in code so it would have to be deprecated)
>
> The pattern would be pretty much the same as the one documented and
> successfully deployed here:
>
> https://community.hortonworks.com/articles/71916/connecting-
> to-azure-data-lake-from-a-nifi-dataflow.html
>
> Which means that in the case of Alluxio, one would use the properties
> documented here:
>
> https://www.alluxio.com/docs/community/1.3/en/Running-
> Hadoop-MapReduce-on-Alluxio.html
>
> While with Google Cloud Storage we would use the properties documented
> here:
>
> https://cloud.google.com/hadoop/google-cloud-storage-connector
>
> I noticed that specific processors could have the ability to handle
> particular properties to a filesystem, however I would like to believe the
> same issue would plague hadoop users, and therefore is reasonable to
> believe the Hadoop compatible implementations would have ways of exposing
> those properties as well?
>
> In the case the properties are exposed, we perhaps simply adjust the *HDFS
> processors to use dynamic properties to pass those to the underlying
> module, therefore providing a way to explore particular settings of an
> underlying storage platforms.
>
> Any opinion would be welcome
>
> PS-sent it again with proper subject label
>

nar deployment

[Bleeker, Arthur H]

Hi,

About six months ago I created custom processors for NiFi. Recently I needed to 
dust off the code and make some changes. I pushed the updated nar to my local 
development nifi instance and... nothing. The changes didn't get picked up. So 
I deleted the nar, restarted nifi, copied the new nar in, and then restarted 
nifi again. The processors disappeared.

I have been unable to get any nar to deploy since then. I have even tried a 
100% clean nifi instance version 1.1.1 with a maven archetype nar processor 
project. Nothing. I can see the nar in the lib directory. I can see that it is 
intact. But it does not deploy. There's nothing in the logs indicating an 
exception or warning of any kind. Hopefully someone can help me figure this out.

Thanks,

Arthur Bleeker
Software Architect and Developer
Pacific Northwest National Laboratory

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andre]

Andrew,

Thank you for contributing.

On 22 Feb 2017 10:21 AM, "Andrew Grande"  wrote:

Andre,

I came across multiple NiFi use cases where going through the HDFS layer
and the fs plugin may not be possible. I.e. when no HDFS layer present at
all, so no NN to connect to.


Not sure I understand what you mean.


Another important aspect is operations. Current PutHDFS model with
additional jar location, well, it kinda works, but I very much dislike it.
Too many possibilities for a human error in addition to deployment pain,
especially in a cluster.


Fair enough. Would you mind expanding a bit on what sort of  challenges
currently apply in terms of cluster deployment?


Finally, native object storage processors have features which may not even
apply to the HDFS layer. E.g. the Azure storage has Table storage, etc.


This is a very valid point but I am sure exceptions (in this case a NoSQL
DB operating under the umbrella term of "storage").

I perhaps should have made it more explicit but the requirements are:

- existence of a hadoop compatible interface
- ability to handle files

Again, thank you for the input, truly appreciated.

Andre

I agree consolidating various efforts is worthwhile, but only within a
context of a specific storage solution. Not 'unifying' them into a single
layer.

Andrew

On Tue, Feb 21, 2017, 6:10 PM Andre  wrote:

> dev,
>
> I was having a chat with Pierre around PR#379 and we thought it would be
> worth sharing this with the wider group:
>
>
> I recently noticed that we merged a number of PRs and merges around
> scale-out/cloud based object store into the master.
>
> Would it make sense to start considering adopting a pattern where
> Put/Get/ListHDFS are used in tandem with implementations of the
> hadoop.filesystem interfaces instead of creating new processors, except
> where a particular deficiency/incompatibility in the hadoop.filesystem
> implementation exists?
>
> Candidates for removal / non merge would be:
>
> - Alluxio (PR#379)
> - WASB (PR#626)
>  - Azure* (PR#399)
> - *GCP (recently merged as PR#1482)
> - *S3 (although this has been in code so it would have to be deprecated)
>
> The pattern would be pretty much the same as the one documented and
> successfully deployed here:
>
> https://community.hortonworks.com/articles/71916/connecting-
> to-azure-data-lake-from-a-nifi-dataflow.html
>
> Which means that in the case of Alluxio, one would use the properties
> documented here:
>
> https://www.alluxio.com/docs/community/1.3/en/Running-
> Hadoop-MapReduce-on-Alluxio.html
>
> While with Google Cloud Storage we would use the properties documented
> here:
>
> https://cloud.google.com/hadoop/google-cloud-storage-connector
>
> I noticed that specific processors could have the ability to handle
> particular properties to a filesystem, however I would like to believe the
> same issue would plague hadoop users, and therefore is reasonable to
> believe the Hadoop compatible implementations would have ways of exposing
> those properties as well?
>
> In the case the properties are exposed, we perhaps simply adjust the *HDFS
> processors to use dynamic properties to pass those to the underlying
> module, therefore providing a way to explore particular settings of an
> underlying storage platforms.
>
> Any opinion would be welcome
>
> PS-sent it again with proper subject label
>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Scott Aslan]

Congrats Jeff! Well deserved!

On Tue, Feb 21, 2017 at 6:36 PM, Koji Kawamura 
wrote:

> Congratulations Jeff!
>
> On Wed, Feb 22, 2017 at 7:52 AM, Andre  wrote:
> > Welcome aboard Jeff! Well deserved
> >
> > On Wed, Feb 22, 2017 at 6:41 AM, Aldrin Piri  wrote:
> >
> >> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Jeff
> >> Storck has accepted the PMC's invitation to become a committer on the
> >> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> >> generous contributions and look forward to continued involvement in the
> >> project.
> >>
> >> Jeff's contributions include significant efforts toward upgrade and
> >> migration processes inclusive of flow layout when upgrading from 0.x to
> 1.x
> >> and the ZooKeeper migration toolkit.
> >>
> >> Congrats, Jeff!
> >>
>



-- 
*Scott Aslan = new WebDeveloper(*
*{"location": {"city": "Saint Cloud","state": "FL",
"zip": "34771"},"contact": {"email":
"scottyas...@gmail.com ","linkedin":
"http://www.linkedin.com/in/scottyaslan
"}});*

Email processor test failure

[Chris Herrera]

Hi All,

Apologies for apparently going braindead…I’m sure I’m doing something silly…. I 
am in the process of starting to work on some custom processors and controller 
services, and I am running into an issue with a few tests in the email 
processor.

Specifically org.apache.nifi.processors.email.TestListenSMTP is timing out on 
validateSuccesfulInteraction… when stepping through it seems as if it is timing 
out. However, I also see that there is a test smtp server that should be stood 
up for the test…has anyone run into this before.

Thanks a lot!
Chris Herrera

Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 48.071 sec <<< 
FAILURE! - in org.apache.nifi.processors.email.TestListenSMTP
validateSuccessfulInteraction(org.apache.nifi.processors.email.TestListenSMTP)  
Time elapsed: 15.04 sec  <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteraction(TestListenSMTP.java:91)

validateSuccessfulInteractionWithTls(org.apache.nifi.processors.email.TestListenSMTP)
  Time elapsed: 16.518 sec  <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteractionWithTls(TestListenSMTP.java:157)

validateTooLargeMessage(org.apache.nifi.processors.email.TestListenSMTP)  Time 
elapsed: 16.513 sec  <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.nifi.processors.email.TestListenSMTP.validateTooLargeMessage(TestListenSMTP.java:203)

Re: Email processor test failure

[Joe Witt]

Chris,

Can you look for additional information in the test logsi've not
seen this behavior before on these processors.  That they're all
failing at around 15/16 seconds is interesting too as this is not a
typical timeout value.  That said each does is setting a 10 second
timeout.  Anyway, lets see what else we can get out of the logs.

Thanks
Joe

On Tue, Feb 21, 2017 at 1:57 PM, Chris Herrera
 wrote:
> Hi All,
>
> Apologies for apparently going braindead…I’m sure I’m doing something silly…. 
> I am in the process of starting to work on some custom processors and 
> controller services, and I am running into an issue with a few tests in the 
> email processor.
>
> Specifically org.apache.nifi.processors.email.TestListenSMTP is timing out on 
> validateSuccesfulInteraction… when stepping through it seems as if it is 
> timing out. However, I also see that there is a test smtp server that should 
> be stood up for the test…has anyone run into this before.
>
> Thanks a lot!
> Chris Herrera
>
> Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 48.071 sec 
> <<< FAILURE! - in org.apache.nifi.processors.email.TestListenSMTP
> validateSuccessfulInteraction(org.apache.nifi.processors.email.TestListenSMTP)
>   Time elapsed: 15.04 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteraction(TestListenSMTP.java:91)
>
> validateSuccessfulInteractionWithTls(org.apache.nifi.processors.email.TestListenSMTP)
>   Time elapsed: 16.518 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteractionWithTls(TestListenSMTP.java:157)
>
> validateTooLargeMessage(org.apache.nifi.processors.email.TestListenSMTP)  
> Time elapsed: 16.513 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.processors.email.TestListenSMTP.validateTooLargeMessage(TestListenSMTP.java:203)
>

Re: Email processor test failure

[Aldrin Piri]

Could you additionally specify your environment?  mvn -version should be
sufficient.

I feel like we have seen this intermittently on Windows with certain JDK
updates.


On Tue, Feb 21, 2017 at 2:06 PM, Joe Witt  wrote:

> Chris,
>
> Can you look for additional information in the test logsi've not
> seen this behavior before on these processors.  That they're all
> failing at around 15/16 seconds is interesting too as this is not a
> typical timeout value.  That said each does is setting a 10 second
> timeout.  Anyway, lets see what else we can get out of the logs.
>
> Thanks
> Joe
>
> On Tue, Feb 21, 2017 at 1:57 PM, Chris Herrera
>  wrote:
> > Hi All,
> >
> > Apologies for apparently going braindead…I’m sure I’m doing something
> silly…. I am in the process of starting to work on some custom processors
> and controller services, and I am running into an issue with a few tests in
> the email processor.
> >
> > Specifically org.apache.nifi.processors.email.TestListenSMTP is timing
> out on validateSuccesfulInteraction… when stepping through it seems as if
> it is timing out. However, I also see that there is a test smtp server that
> should be stood up for the test…has anyone run into this before.
> >
> > Thanks a lot!
> > Chris Herrera
> >
> > Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 48.071
> sec <<< FAILURE! - in org.apache.nifi.processors.email.TestListenSMTP
> > validateSuccessfulInteraction(org.apache.nifi.processors.email.TestListenSMTP)
> Time elapsed: 15.04 sec  <<< FAILURE!
> > java.lang.AssertionError: null
> > at org.junit.Assert.fail(Assert.java:86)
> > at org.junit.Assert.assertTrue(Assert.java:41)
> > at org.junit.Assert.assertTrue(Assert.java:52)
> > at org.apache.nifi.processors.email.TestListenSMTP.
> validateSuccessfulInteraction(TestListenSMTP.java:91)
> >
> > validateSuccessfulInteractionWithTls(org.apache.nifi.processors.email.TestListenSMTP)
> Time elapsed: 16.518 sec  <<< FAILURE!
> > java.lang.AssertionError: null
> > at org.junit.Assert.fail(Assert.java:86)
> > at org.junit.Assert.assertTrue(Assert.java:41)
> > at org.junit.Assert.assertTrue(Assert.java:52)
> > at org.apache.nifi.processors.email.TestListenSMTP.
> validateSuccessfulInteractionWithTls(TestListenSMTP.java:157)
> >
> > validateTooLargeMessage(org.apache.nifi.processors.email.TestListenSMTP)
> Time elapsed: 16.513 sec  <<< FAILURE!
> > java.lang.AssertionError: null
> > at org.junit.Assert.fail(Assert.java:86)
> > at org.junit.Assert.assertTrue(Assert.java:41)
> > at org.junit.Assert.assertTrue(Assert.java:52)
> > at org.apache.nifi.processors.email.TestListenSMTP.
> validateTooLargeMessage(TestListenSMTP.java:203)
> >
>

Re: Email processor test failure

[Oleg Zhurakousky]

Chris

ListenSMTP is essentially an email server. This processor essentially allows 
you to send emails to it. So by running it you are starting a simple email 
server. 
That said something is obviously not going well on your end. Any firewalls or 
other specific networking setup that you have that may cause it?
Anyway, you must have some stack traces to share.

Cheers
Oleg

> On Feb 21, 2017, at 1:57 PM, Chris Herrera  wrote:
> 
> Hi All,
> 
> Apologies for apparently going braindead…I’m sure I’m doing something silly…. 
> I am in the process of starting to work on some custom processors and 
> controller services, and I am running into an issue with a few tests in the 
> email processor.
> 
> Specifically org.apache.nifi.processors.email.TestListenSMTP is timing out on 
> validateSuccesfulInteraction… when stepping through it seems as if it is 
> timing out. However, I also see that there is a test smtp server that should 
> be stood up for the test…has anyone run into this before.
> 
> Thanks a lot!
> Chris Herrera
> 
> Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 48.071 sec 
> <<< FAILURE! - in org.apache.nifi.processors.email.TestListenSMTP
> validateSuccessfulInteraction(org.apache.nifi.processors.email.TestListenSMTP)
>   Time elapsed: 15.04 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteraction(TestListenSMTP.java:91)
> 
> validateSuccessfulInteractionWithTls(org.apache.nifi.processors.email.TestListenSMTP)
>   Time elapsed: 16.518 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteractionWithTls(TestListenSMTP.java:157)
> 
> validateTooLargeMessage(org.apache.nifi.processors.email.TestListenSMTP)  
> Time elapsed: 16.513 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.processors.email.TestListenSMTP.validateTooLargeMessage(TestListenSMTP.java:203)
> 

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Michael Moser]

Congrats Jeff!


On Tue, Feb 21, 2017 at 4:25 PM, Joe Skora  wrote:

> Congrats and welcome aboard Jeff!
>
> On Tue, Feb 21, 2017 at 4:01 PM, Tony Kurc  wrote:
>
> > Nice work Jeff!
> >
> > On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:
> >
> > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Jeff
> > > Storck has accepted the PMC's invitation to become a committer on the
> > > Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> > > generous contributions and look forward to continued involvement in the
> > > project.
> > >
> > > Jeff's contributions include significant efforts toward upgrade and
> > > migration processes inclusive of flow layout when upgrading from 0.x to
> > 1.x
> > > and the ZooKeeper migration toolkit.
> > >
> > > Congrats, Jeff!
> > >
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Joe Skora]

Congrats and welcome aboard Jeff!

On Tue, Feb 21, 2017 at 4:01 PM, Tony Kurc  wrote:

> Nice work Jeff!
>
> On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:
>
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
> > Storck has accepted the PMC's invitation to become a committer on the
> > Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> > generous contributions and look forward to continued involvement in the
> > project.
> >
> > Jeff's contributions include significant efforts toward upgrade and
> > migration processes inclusive of flow layout when upgrading from 0.x to
> 1.x
> > and the ZooKeeper migration toolkit.
> >
> > Congrats, Jeff!
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Bryan Rosander]

Congrats!

On Tue, Feb 21, 2017 at 3:47 PM, Joe Percivall 
wrote:

>  Congrats Jeff!
>
> On Tue, Feb 21, 2017 at 3:33 PM, Yolanda Davis 
> wrote:
>
> > Congratulations Jeff!
> >
> > On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:
> >
> > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Jeff
> > > Storck has accepted the PMC's invitation to become a committer on the
> > > Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> > > generous contributions and look forward to continued involvement in the
> > > project.
> > >
> > > Jeff's contributions include significant efforts toward upgrade and
> > > migration processes inclusive of flow layout when upgrading from 0.x to
> > 1.x
> > > and the ZooKeeper migration toolkit.
> > >
> > > Congrats, Jeff!
> > >
> >
> >
> >
> > --
> > --
> > yolanda.m.da...@gmail.com
> > @YolandaMDavis
> >
>
>
>
> --
> *Joe Percivall*
> linkedin.com/in/Percivall
> e: jperciv...@apache.com
>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Joe Percivall]

 Congrats Jeff!

On Tue, Feb 21, 2017 at 3:33 PM, Yolanda Davis 
wrote:

> Congratulations Jeff!
>
> On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:
>
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
> > Storck has accepted the PMC's invitation to become a committer on the
> > Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> > generous contributions and look forward to continued involvement in the
> > project.
> >
> > Jeff's contributions include significant efforts toward upgrade and
> > migration processes inclusive of flow layout when upgrading from 0.x to
> 1.x
> > and the ZooKeeper migration toolkit.
> >
> > Congrats, Jeff!
> >
>
>
>
> --
> --
> yolanda.m.da...@gmail.com
> @YolandaMDavis
>



-- 
*Joe Percivall*
linkedin.com/in/Percivall
e: jperciv...@apache.com

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Tony Kurc]

Nice work Jeff!

On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
> Storck has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> generous contributions and look forward to continued involvement in the
> project.
>
> Jeff's contributions include significant efforts toward upgrade and
> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
> and the ZooKeeper migration toolkit.
>
> Congrats, Jeff!
>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Andy LoPresto]

Congratulations Jeff. Well deserved.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 21, 2017, at 11:58 AM, Joe Witt  wrote:
> 
> Congrats Jeff!
> 
> On Tue, Feb 21, 2017 at 2:43 PM, Jeremy Dyer  wrote:
>> Congratulations Jeff! Good to have you aboard!
>> 
>> On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:
>> 
>>> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
>>> Storck has accepted the PMC's invitation to become a committer on the
>>> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
>>> generous contributions and look forward to continued involvement in the
>>> project.
>>> 
>>> Jeff's contributions include significant efforts toward upgrade and
>>> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
>>> and the ZooKeeper migration toolkit.
>>> 
>>> Congrats, Jeff!
>>> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Joe Witt]

Congrats Jeff!

On Tue, Feb 21, 2017 at 2:43 PM, Jeremy Dyer  wrote:
> Congratulations Jeff! Good to have you aboard!
>
> On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:
>
>> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
>> Storck has accepted the PMC's invitation to become a committer on the
>> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
>> generous contributions and look forward to continued involvement in the
>> project.
>>
>> Jeff's contributions include significant efforts toward upgrade and
>> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
>> and the ZooKeeper migration toolkit.
>>
>> Congrats, Jeff!
>>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Pierre Villard]

Congrats !

2017-02-21 20:58 GMT+01:00 Joe Witt :

> Congrats Jeff!
>
> On Tue, Feb 21, 2017 at 2:43 PM, Jeremy Dyer  wrote:
> > Congratulations Jeff! Good to have you aboard!
> >
> > On Tue, Feb 21, 2017 at 2:41 PM, Aldrin Piri  wrote:
> >
> >> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Jeff
> >> Storck has accepted the PMC's invitation to become a committer on the
> >> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> >> generous contributions and look forward to continued involvement in the
> >> project.
> >>
> >> Jeff's contributions include significant efforts toward upgrade and
> >> migration processes inclusive of flow layout when upgrading from 0.x to
> 1.x
> >> and the ZooKeeper migration toolkit.
> >>
> >> Congrats, Jeff!
> >>
>

Re: Email processor test failure

[Chris Herrera]

Thanks All!

Here is some additional information:

Interestingly enough it seems in the surefire report that the SMTP server is 
starting on a different port than the test is trying to connect to, unless I’m 
reading it wrong.

Nothing else strange on the networking side, verified my hosts file is normal, 
no weird firewalls, etc...

Env Info:
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
2015-11-10T10:41:47-06:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.3", arch: "x86_64", family: “mac"

Additional Surefire Report info:
[pool-15-thread-1] INFO org.subethamail.smtp.server.SMTPServer - SMTP server 
*:58738 starting
[org.subethamail.smtp.server.ServerThread *:58738] INFO 
org.subethamail.smtp.server.ServerThread - SMTP server *:58738 started
[pool-18-thread-1] INFO org.subethamail.smtp.server.SMTPServer - SMTP server 
*:58840 starting
[org.subethamail.smtp.server.ServerThread *:58840] INFO 
org.subethamail.smtp.server.ServerThread - SMTP server *:58840 started
org.apache.commons.mail.EmailException: Sending the email to the following 
server failed : localhost:58738
at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1421)
at org.apache.commons.mail.Email.send(Email.java:1448)
at org.apache.nifi.processors.email.TestListenSMTP$1.run(TestListenSMTP.java:78)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.MessagingException: [EOF]
at com.sun.mail.smtp.SMTPTransport.issueCommand(SMTPTransport.java:2074)
at com.sun.mail.smtp.SMTPTransport.helo(SMTPTransport.java:1469)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:660)
at javax.mail.Service.connect(Service.java:295)
at javax.mail.Service.connect(Service.java:176)
at javax.mail.Service.connect(Service.java:125)
at javax.mail.Transport.send0(Transport.java:194)
at javax.mail.Transport.send(Transport.java:124)
at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1411)
... 9 more
[pool-21-thread-1] INFO org.subethamail.smtp.server.SMTPServer - SMTP server 
*:58923 starting
[org.subethamail.smtp.server.ServerThread *:58923] INFO 
org.subethamail.smtp.server.ServerThread - SMTP server *:58923 started
org.apache.commons.mail.EmailException: Sending the email to the following 
server failed : localhost:58840
at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1421)
at org.apache.commons.mail.Email.send(Email.java:1448)
at 
org.apache.nifi.processors.email.TestListenSMTP$2.run(TestListenSMTP.java:144)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.MessagingException: [EOF]
at com.sun.mail.smtp.SMTPTransport.issueCommand(SMTPTransport.java:2074)
at com.sun.mail.smtp.SMTPTransport.helo(SMTPTransport.java:1469)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:660)
at javax.mail.Service.connect(Service.java:295)
at javax.mail.Service.connect(Service.java:176)
at javax.mail.Service.connect(Service.java:125)
at javax.mail.Transport.send0(Transport.java:194)
at javax.mail.Transport.send(Transport.java:124)
at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1411)
... 9 more
Regards,
Chris

On Feb 21, 2017, 1:16 PM -0600, Aldrin Piri , wrote:
> Could you additionally specify your environment? mvn -version should be
> sufficient.
>
> I feel like we have seen this intermittently on Windows with certain JDK
> updates.
>
>
> On Tue, Feb 21, 2017 at 2:06 PM, Joe Witt  wrote:
>
> > Chris,
> >
> > Can you look for additional information in the test logsi've not
> > seen this behavior before on these processors. That they're all
> > failing at around 15/16 seconds is interesting too as this is not a
> > typical timeout value. That said 

Scale-out/Object Storage - taming the deviersity of processors

[Andre]

dev,

I was having a chat with Pierre around PR#379 and we thought it would be
worth sharing this with the wider group:


I recently noticed that we merged a number of PRs and merges around
scale-out/cloud based object store into the master.

Would it make sense to start considering adopting a pattern where
Put/Get/ListHDFS are used in tandem with implementations of the
hadoop.filesystem interfaces instead of creating new processors, except
where a particular deficiency/incompatibility in the hadoop.filesystem
implementation exists?

Candidates for removal / non merge would be:

- Alluxio (PR#379)
- WASB (PR#626)
 - Azure* (PR#399)
- *GCP (recently merged as PR#1482)
- *S3 (although this has been in code so it would have to be deprecated)

The pattern would be pretty much the same as the one documented and
successfully deployed here:

https://community.hortonworks.com/articles/71916/connecting-to-azure-data-lake-from-a-nifi-dataflow.html

Which means that in the case of Alluxio, one would use the properties
documented here:

https://www.alluxio.com/docs/community/1.3/en/Running-Hadoop-MapReduce-on-Alluxio.html

While with Google Cloud Storage we would use the properties documented here:

https://cloud.google.com/hadoop/google-cloud-storage-connector

I noticed that specific processors could have the ability to handle
particular properties to a filesystem, however I would like to believe the
same issue would plague hadoop users, and therefore is reasonable to
believe the Hadoop compatible implementations would have ways of exposing
those properties as well?

In the case the properties are exposed, we perhaps simply adjust the *HDFS
processors to use dynamic properties to pass those to the underlying
module, therefore providing a way to explore particular settings of an
underlying storage platforms.

Any opinion would be welcome

[DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andre]

dev,

I was having a chat with Pierre around PR#379 and we thought it would be
worth sharing this with the wider group:


I recently noticed that we merged a number of PRs and merges around
scale-out/cloud based object store into the master.

Would it make sense to start considering adopting a pattern where
Put/Get/ListHDFS are used in tandem with implementations of the
hadoop.filesystem interfaces instead of creating new processors, except
where a particular deficiency/incompatibility in the hadoop.filesystem
implementation exists?

Candidates for removal / non merge would be:

- Alluxio (PR#379)
- WASB (PR#626)
 - Azure* (PR#399)
- *GCP (recently merged as PR#1482)
- *S3 (although this has been in code so it would have to be deprecated)

The pattern would be pretty much the same as the one documented and
successfully deployed here:

https://community.hortonworks.com/articles/71916/connecting-
to-azure-data-lake-from-a-nifi-dataflow.html

Which means that in the case of Alluxio, one would use the properties
documented here:

https://www.alluxio.com/docs/community/1.3/en/Running-
Hadoop-MapReduce-on-Alluxio.html

While with Google Cloud Storage we would use the properties documented here:

https://cloud.google.com/hadoop/google-cloud-storage-connector

I noticed that specific processors could have the ability to handle
particular properties to a filesystem, however I would like to believe the
same issue would plague hadoop users, and therefore is reasonable to
believe the Hadoop compatible implementations would have ways of exposing
those properties as well?

In the case the properties are exposed, we perhaps simply adjust the *HDFS
processors to use dynamic properties to pass those to the underlying
module, therefore providing a way to explore particular settings of an
underlying storage platforms.

Any opinion would be welcome

PS-sent it again with proper subject label

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Koji Kawamura]

Congratulations Jeff!

On Wed, Feb 22, 2017 at 7:52 AM, Andre  wrote:
> Welcome aboard Jeff! Well deserved
>
> On Wed, Feb 22, 2017 at 6:41 AM, Aldrin Piri  wrote:
>
>> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
>> Storck has accepted the PMC's invitation to become a committer on the
>> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
>> generous contributions and look forward to continued involvement in the
>> project.
>>
>> Jeff's contributions include significant efforts toward upgrade and
>> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
>> and the ZooKeeper migration toolkit.
>>
>> Congrats, Jeff!
>>

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Matt Burgess]

I agree with Andrew in the operations sense, and would like to add
that the user experience around dynamic properties (and even
"conditional" properties that are not dynamic but can be exposed when
other properties are "Applied") can be less-than-ideal and IMHO should
be used sparingly. Full disclosure: My latest processor uses
"conditional" properties at the moment, choosing them over dynamic
properties in the hopes that the user experience is better, but
without in-place updates (possibly implemented under [1]) and/or the
UI making it obvious that dynamic properties are supported (under
[2]), I'm not sure which is better (or if I should create different
processors for my case as well).

Under the hood, if it makes sense to group these processors and
abstract away common code, then I'm all for it.  Especially if we can
use something like the nifi-hadoop-libraries-nar as an ancestor NAR to
provide a common set of libraries to all the Hadoop-Compatible File
System (HCFS) implementations.  However I fear based on versions of
the specific HCFS implementations, they may also need different
versions of the HFS client dependencies, in which case we'd be looking
for the Extension Registry and some smart classloading to alleviate
those pain points without ballooning the NiFi footprint.

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-1121
[2] https://issues.apache.org/jira/browse/NIFI-2629


On Tue, Feb 21, 2017 at 6:21 PM, Andrew Grande  wrote:
> Andre,
>
> I came across multiple NiFi use cases where going through the HDFS layer
> and the fs plugin may not be possible. I.e. when no HDFS layer present at
> all, so no NN to connect to.
>
> Another important aspect is operations. Current PutHDFS model with
> additional jar location, well, it kinda works, but I very much dislike it.
> Too many possibilities for a human error in addition to deployment pain,
> especially in a cluster.
>
> Finally, native object storage processors have features which may not even
> apply to the HDFS layer. E.g. the Azure storage has Table storage, etc.
>
> I agree consolidating various efforts is worthwhile, but only within a
> context of a specific storage solution. Not 'unifying' them into a single
> layer.
>
> Andrew
>
> On Tue, Feb 21, 2017, 6:10 PM Andre  wrote:
>
>> dev,
>>
>> I was having a chat with Pierre around PR#379 and we thought it would be
>> worth sharing this with the wider group:
>>
>>
>> I recently noticed that we merged a number of PRs and merges around
>> scale-out/cloud based object store into the master.
>>
>> Would it make sense to start considering adopting a pattern where
>> Put/Get/ListHDFS are used in tandem with implementations of the
>> hadoop.filesystem interfaces instead of creating new processors, except
>> where a particular deficiency/incompatibility in the hadoop.filesystem
>> implementation exists?
>>
>> Candidates for removal / non merge would be:
>>
>> - Alluxio (PR#379)
>> - WASB (PR#626)
>>  - Azure* (PR#399)
>> - *GCP (recently merged as PR#1482)
>> - *S3 (although this has been in code so it would have to be deprecated)
>>
>> The pattern would be pretty much the same as the one documented and
>> successfully deployed here:
>>
>> https://community.hortonworks.com/articles/71916/connecting-
>> to-azure-data-lake-from-a-nifi-dataflow.html
>>
>> Which means that in the case of Alluxio, one would use the properties
>> documented here:
>>
>> https://www.alluxio.com/docs/community/1.3/en/Running-
>> Hadoop-MapReduce-on-Alluxio.html
>>
>> While with Google Cloud Storage we would use the properties documented
>> here:
>>
>> https://cloud.google.com/hadoop/google-cloud-storage-connector
>>
>> I noticed that specific processors could have the ability to handle
>> particular properties to a filesystem, however I would like to believe the
>> same issue would plague hadoop users, and therefore is reasonable to
>> believe the Hadoop compatible implementations would have ways of exposing
>> those properties as well?
>>
>> In the case the properties are exposed, we perhaps simply adjust the *HDFS
>> processors to use dynamic properties to pass those to the underlying
>> module, therefore providing a way to explore particular settings of an
>> underlying storage platforms.
>>
>> Any opinion would be welcome
>>
>> PS-sent it again with proper subject label
>>