Re: Odd inter-operation behavior when using clustered NiFi against secure MapR cluster

[Sumanth]

I had same issue with Secure MapR Cluster + NiFi cluster with embedded zk + 
PutHDFS setup.
Switched back to non-cluster NiFi to avoid conflict between NiFi enabled zk and 
MapR. Waiting for  stable solution to get back NiFi cluster. 
Thanks
Sumo  


 

Sent from my iPad

> On Oct 26, 2016, at 7:21 AM, Andre  wrote:
> 
> Bryan,
> 
> My apologies as the original email wasn't explicit about this:
> 
> Your assumption is correct: My flow contains a processor (PutHDFS) with a
> core-site.xml configured. The file contains the property you refer to (as
> this is a cleaner way to force NiFi to connect to the secure MapR cluster).
> 
> Funny enough, when used without zk, the processor works fine. Same way zk
> works if correctly configured,
> 
> However, in order for both to work at the same time, I had to use to JAAS
> workaround.
> 
> 
> As a side note, In case you wonder: MapR's JAAS contains both the Server
> and Client stanzas required to run a secure zk, however, they are designed
> to use MapR's security mechanism and their packaged version of zookeeper.
> As consequence, their Stanzas require jars to be added to class path and
> all sort of weird stuff that I preferred not to introduce (since I am using
> the zk embedded within NiFi).
> 
> Had not been the case, I could point arg.15 to MapR's default JAAS as
> described here: http://doc.mapr.com/display/MapR/mapr.login.conf and here:
> http://doc.mapr.com/pages/viewpage.action?pageId=32506648
> 
> 
> Cheers
> 
> 
> 
>> On Thu, Oct 27, 2016 at 12:51 AM, Bryan Bende  wrote:
>> 
>> Meant to say the config instance somehow got
>> "hadoop.security.authentication"
>> set to "kerberos"
>> 
>>> On Wed, Oct 26, 2016 at 9:50 AM, Bryan Bende  wrote:
>>> 
>>> Andre,
>>> 
>>> This definitely seems weird that somehow using embedded ZooKeeper is
>>> causing this.
>>> 
>>> One thing I can say though, is that in order to get into the code in your
>>> stacktrace, it had to pass through SecurityUtil.
>> isSecurityEnabled(config)
>>> which does the following:
>>> 
>>> public static boolean isSecurityEnabled(final Configuration config) {
>>>Validate.notNull(config);
>>>return "kerberos".equalsIgnoreCase(config.get("hadoop.security.
>>> authentication"));
>>> }
>>> 
>>> The Configuration instance passed in is created using the default
>>> constructor Configuration config = new Configuration(); and then any
>>> files/paths entered into the processor's resource property is added to
>> the
>>> config.
>>> 
>>> So in order for isSecurityEnabled to return true, it means the config
>>> instance somehow got "hadoop.security.authentication" set to true, which
>>> usually only happens if you put a core-site.xml on the classpath with
>> that
>>> value set.
>>> 
>>> Is it possible some JAR from the MapR dependencies has a core-site.xml
>>> embedded in it?
>>> 
>>> -Bryan
>>> 
 On Wed, Oct 26, 2016 at 6:09 AM, Andre  wrote:
 
 Hi there,
 
 I've notice an odd behavior when using embedded Zookeeper on a NiFi
 cluster
 with MapR compatible processors:
 
 I noticed that every time I enable embedded zookeeper, NiFi's HDFS
 processors (e.g. PutHDFS) start complaining about Kerberos identities:
 
 2016-10-26 20:07:22,376 ERROR [StandardProcessScheduler Thread-2]
 o.apache.nifi.processors.hadoop.PutHDFS
 java.io.IOException: Login failure for princical@REALM-NAME-GOES-HERE
 from
 keytab /path/to/keytab_file/nifi.keytab
at
 org.apache.hadoop.security.UserGroupInformation.loginUserFro
 mKeytabAndReturnUGI(UserGroupInformation.java:1084)
 ~[hadoop-common-2.7.0-mapr-1602.jar:na]
at
 org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:52)
 ~[nifi-hadoop-utils-1.0.0.jar:1.0.0]
at
 org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.re
 setHDFSResources(AbstractHadoopProcessor.java:285)
 ~[nifi-hdfs-processors-1.0.0.jar:1.0.0]
at
 org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.ab
 stractOnScheduled(AbstractHadoopProcessor.java:213)
 ~[nifi-hdfs-processors-1.0.0.jar:1.0.0]
at
 org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:181)
 [nifi-hdfs-processors-1.0.0.jar:1.0.0]
 
 So far so good, these errors are quite familiar to people using NiFi
 against secure MapR clusters and caused by issues around the custom JAAS
 settings required by Java applications relying on the MapR client to
>> work.
 
 The normal workaround this would be instructing NiFi to where the the
>> JAAS
 settings via bootstrap.conf [1]:
 
 $grep jaas
 java.arg.15=-Djava.security.auth.login.config=./conf/nifi-jaas.conf
 
 The contents of nifi-jaas.conf are a copy of the relevant MapR JAAS
 stanza:
 
 While the workaround seems to work (still doing tests) I ask:
 

Re: 0.7.1 download items

[Joe Skora]

JoeW,

I made the link changes.  Following the 1.0.0 link pattern I changed the
source and binary file links but not the hash file links.

Is there a commit job that pushes from the repository to the website?

Regards,
JoeS

On Wed, Oct 26, 2016 at 3:45 PM, Joe Skora  wrote:

> No problem.  Will do!
>
> On Wed, Oct 26, 2016 at 8:35 PM, Joe Witt  wrote:
>
>> JoeS,
>>
>> Do you mind updating the download page for the 0.7.1 sources and
>> binaries to use the mirror based URLs rather than the specific dist
>> download site.  See the 1.0 download links for reference.  It should
>> end up looking like this
>> https://www.apache.org/dyn/closer.lua?path=/nifi/0.7.1/nifi-
>> 0.7.1-bin.tar.gz
>>
>> Thanks
>> Joe
>>
>
>

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Andre]

Welcome aboard Byan!

Your contributions are much appreciated by many NiFi users (TLS Toolkit I
am looking at you..) and I am looking forward for your future contributions.

Regarding the use of my name by Joe W, Rest assured that was not a copy and
paste mistake...

Joe is just trying to make right the mistake he committed in Melbourne...
:-)

Cheers

On Thu, Oct 27, 2016 at 4:15 AM, Joe Witt  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Bryan Rosander has accepted the PMC's invitation to become a committer
> on the Apache NiFi project. We greatly appreciate all of Andre's hard
> work and generous contributions and look forward to continued
> involvement in the project.
>
> Bryan has contributed to a range of features and discussions such as
> the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> Expression Language, MiNiFi, and event log parsing.
>
> Welcome Bryan!
>

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Tony Kurc]

Great to have you aboard, Bryan!

On Wed, Oct 26, 2016 at 3:05 PM, Bryan Bende  wrote:

> Congrats Bryan!
>
> On Wed, Oct 26, 2016 at 1:20 PM, Andy LoPresto 
> wrote:
>
> > Congratulations Bryan. I can say the TLS toolkit has made my life much
> > easier and I think that goes for a lot of people. Looking forward to more
> > great contributions.
> >
> > Andy LoPresto
> > alopre...@apache.org
> > *alopresto.apa...@gmail.com *
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Oct 26, 2016, at 10:18 AM, Joe Witt  wrote:
> >
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Bryan Rosander has accepted the PMC's invitation to become a committer
> > on the Apache NiFi project. We greatly appreciate all of Bryan's hard
> > work and generous contributions and look forward to continued
> > involvement in the project.
> >
> > Bryan has contributed to a range of features and discussions such as
> > the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> > Expression Language, MiNiFi, and event log parsing.
> >
> > Welcome Bryan!
> >
> > On Wed, Oct 26, 2016 at 1:16 PM, Oleg Zhurakousky
> >  wrote:
> >
> > Congrats Bryan!
> > Glad to have you!
> >
> > On Oct 26, 2016, at 1:15 PM, Joe Witt  wrote:
> >
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Bryan Rosander has accepted the PMC's invitation to become a committer
> > on the Apache NiFi project. We greatly appreciate all of Andre's hard
> > work and generous contributions and look forward to continued
> > involvement in the project.
> >
> > Bryan has contributed to a range of features and discussions such as
> > the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> > Expression Language, MiNiFi, and event log parsing.
> >
> > Welcome Bryan!
> >
> >
> >
> >
>

Re: 0.7.1 download items

[Joe Skora]

No problem.  Will do!

On Wed, Oct 26, 2016 at 8:35 PM, Joe Witt  wrote:

> JoeS,
>
> Do you mind updating the download page for the 0.7.1 sources and
> binaries to use the mirror based URLs rather than the specific dist
> download site.  See the 1.0 download links for reference.  It should
> end up looking like this
> https://www.apache.org/dyn/closer.lua?path=/nifi/0.7.1/
> nifi-0.7.1-bin.tar.gz
>
> Thanks
> Joe
>

0.7.1 download items

[Joe Witt]

JoeS,

Do you mind updating the download page for the 0.7.1 sources and
binaries to use the mirror based URLs rather than the specific dist
download site.  See the 1.0 download links for reference.  It should
end up looking like this
https://www.apache.org/dyn/closer.lua?path=/nifi/0.7.1/nifi-0.7.1-bin.tar.gz

Thanks
Joe

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Bryan Bende]

Congrats Bryan!

On Wed, Oct 26, 2016 at 1:20 PM, Andy LoPresto  wrote:

> Congratulations Bryan. I can say the TLS toolkit has made my life much
> easier and I think that goes for a lot of people. Looking forward to more
> great contributions.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Oct 26, 2016, at 10:18 AM, Joe Witt  wrote:
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Bryan Rosander has accepted the PMC's invitation to become a committer
> on the Apache NiFi project. We greatly appreciate all of Bryan's hard
> work and generous contributions and look forward to continued
> involvement in the project.
>
> Bryan has contributed to a range of features and discussions such as
> the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> Expression Language, MiNiFi, and event log parsing.
>
> Welcome Bryan!
>
> On Wed, Oct 26, 2016 at 1:16 PM, Oleg Zhurakousky
>  wrote:
>
> Congrats Bryan!
> Glad to have you!
>
> On Oct 26, 2016, at 1:15 PM, Joe Witt  wrote:
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Bryan Rosander has accepted the PMC's invitation to become a committer
> on the Apache NiFi project. We greatly appreciate all of Andre's hard
> work and generous contributions and look forward to continued
> involvement in the project.
>
> Bryan has contributed to a range of features and discussions such as
> the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> Expression Language, MiNiFi, and event log parsing.
>
> Welcome Bryan!
>
>
>
>

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Pierre Villard]

Congrats Bryan! (and I definitely agree with Andy)

2016-10-26 19:20 GMT+02:00 Andy LoPresto :

> Congratulations Bryan. I can say the TLS toolkit has made my life much
> easier and I think that goes for a lot of people. Looking forward to more
> great contributions.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Oct 26, 2016, at 10:18 AM, Joe Witt  wrote:
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Bryan Rosander has accepted the PMC's invitation to become a committer
> on the Apache NiFi project. We greatly appreciate all of Bryan's hard
> work and generous contributions and look forward to continued
> involvement in the project.
>
> Bryan has contributed to a range of features and discussions such as
> the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> Expression Language, MiNiFi, and event log parsing.
>
> Welcome Bryan!
>
> On Wed, Oct 26, 2016 at 1:16 PM, Oleg Zhurakousky
>  wrote:
>
> Congrats Bryan!
> Glad to have you!
>
> On Oct 26, 2016, at 1:15 PM, Joe Witt  wrote:
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Bryan Rosander has accepted the PMC's invitation to become a committer
> on the Apache NiFi project. We greatly appreciate all of Andre's hard
> work and generous contributions and look forward to continued
> involvement in the project.
>
> Bryan has contributed to a range of features and discussions such as
> the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> Expression Language, MiNiFi, and event log parsing.
>
> Welcome Bryan!
>
>
>
>

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Yolanda Davis]

Congrats Bryan!

On Wed, Oct 26, 2016 at 1:15 PM, Joe Witt  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Bryan Rosander has accepted the PMC's invitation to become a committer
> on the Apache NiFi project. We greatly appreciate all of Andre's hard
> work and generous contributions and look forward to continued
> involvement in the project.
>
> Bryan has contributed to a range of features and discussions such as
> the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> Expression Language, MiNiFi, and event log parsing.
>
> Welcome Bryan!
>



-- 
--
yolanda.m.da...@gmail.com
@YolandaMDavis

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Joe Witt]

On behalf of the Apache NiFI PMC, I am very pleased to announce that
Bryan Rosander has accepted the PMC's invitation to become a committer
on the Apache NiFi project. We greatly appreciate all of Bryan's hard
work and generous contributions and look forward to continued
involvement in the project.

Bryan has contributed to a range of features and discussions such as
the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
Expression Language, MiNiFi, and event log parsing.

Welcome Bryan!

On Wed, Oct 26, 2016 at 1:16 PM, Oleg Zhurakousky
 wrote:
> Congrats Bryan!
> Glad to have you!
>> On Oct 26, 2016, at 1:15 PM, Joe Witt  wrote:
>>
>> On behalf of the Apache NiFI PMC, I am very pleased to announce that
>> Bryan Rosander has accepted the PMC's invitation to become a committer
>> on the Apache NiFi project. We greatly appreciate all of Andre's hard
>> work and generous contributions and look forward to continued
>> involvement in the project.
>>
>> Bryan has contributed to a range of features and discussions such as
>> the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
>> Expression Language, MiNiFi, and event log parsing.
>>
>> Welcome Bryan!
>>
>

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Scott Aslan]

Congrats Bryan! Glad to see you have Andre doing all your hard work ;p

On Wed, Oct 26, 2016 at 1:16 PM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> Congrats Bryan!
> Glad to have you!
> > On Oct 26, 2016, at 1:15 PM, Joe Witt  wrote:
> >
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Bryan Rosander has accepted the PMC's invitation to become a committer
> > on the Apache NiFi project. We greatly appreciate all of Andre's hard
> > work and generous contributions and look forward to continued
> > involvement in the project.
> >
> > Bryan has contributed to a range of features and discussions such as
> > the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> > Expression Language, MiNiFi, and event log parsing.
> >
> > Welcome Bryan!
> >
>
>


-- 
*Scott Aslan = new WebDeveloper(*
*{"location": {"city": "Saint Cloud","state": "FL",
"zip": "34771"},"contact": {"mobile":
"(321)-591-0870","email": "scottyas...@gmail.com
","linkedin":
"http://www.linkedin.com/in/scottaslan
","skype": "astechdev"
}});*

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Oleg Zhurakousky]

Congrats Bryan!
Glad to have you!
> On Oct 26, 2016, at 1:15 PM, Joe Witt  wrote:
> 
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Bryan Rosander has accepted the PMC's invitation to become a committer
> on the Apache NiFi project. We greatly appreciate all of Andre's hard
> work and generous contributions and look forward to continued
> involvement in the project.
> 
> Bryan has contributed to a range of features and discussions such as
> the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> Expression Language, MiNiFi, and event log parsing.
> 
> Welcome Bryan!
> 

[ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Joe Witt]

On behalf of the Apache NiFI PMC, I am very pleased to announce that
Bryan Rosander has accepted the PMC's invitation to become a committer
on the Apache NiFi project. We greatly appreciate all of Andre's hard
work and generous contributions and look forward to continued
involvement in the project.

Bryan has contributed to a range of features and discussions such as
the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
Expression Language, MiNiFi, and event log parsing.

Welcome Bryan!

Re: PutS3Object submitting file many times

[Joe Skora]

I agree with Joe, it sounds like the NiFi commit was failing after the file
was sent.

If the processor was administratively yielding, possibly due to the disk
issues you mentioned, after completing the upload to S3, the file would be
left in the queue and picked up again for processing later, creating the
situation you observed.


On Wed, Oct 26, 2016 at 3:13 PM, Joe Witt  wrote:

> Joe
>
> I have not looked at this code but can say that processors are or should be
> built such that in error cases we prefer duplication over loss.  So a case
> like you describe could be a good example where session commit fails or
> something like it causing retries but in the mean time we keep sending to
> the destination.  It generally can be avoided that it would be so
> pronounced but just as a heads up I thought id call this out.
>
> Thanks
> Joe
>
> On Oct 26, 2016 9:58 AM, "Joe Gresock"  wrote:
>
> > Has anyone run into problems where PutS3Object submits the same file many
> > (hundreds) of times?  This turned out to be a problem for me since we
> have
> > bucket versioning enabled, so we have many copies of the same file.
> >
> > I can't reproduce this behavior anymore in the normal flow of things, but
> > I'm wondering if it's possible for this to happen if the processor is
> > forced to administratively yield (i.e., due to disk full or too many open
> > files).
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.*-Philippians 4:12-13*
> >
>

Re: Odd inter-operation behavior when using clustered NiFi against secure MapR cluster

[Andre]

Bryan,

My apologies as the original email wasn't explicit about this:

Your assumption is correct: My flow contains a processor (PutHDFS) with a
core-site.xml configured. The file contains the property you refer to (as
this is a cleaner way to force NiFi to connect to the secure MapR cluster).

Funny enough, when used without zk, the processor works fine. Same way zk
works if correctly configured,

However, in order for both to work at the same time, I had to use to JAAS
workaround.


As a side note, In case you wonder: MapR's JAAS contains both the Server
and Client stanzas required to run a secure zk, however, they are designed
to use MapR's security mechanism and their packaged version of zookeeper.
As consequence, their Stanzas require jars to be added to class path and
all sort of weird stuff that I preferred not to introduce (since I am using
the zk embedded within NiFi).

Had not been the case, I could point arg.15 to MapR's default JAAS as
described here: http://doc.mapr.com/display/MapR/mapr.login.conf and here:
http://doc.mapr.com/pages/viewpage.action?pageId=32506648


Cheers



On Thu, Oct 27, 2016 at 12:51 AM, Bryan Bende  wrote:

> Meant to say the config instance somehow got
> "hadoop.security.authentication"
> set to "kerberos"
>
> On Wed, Oct 26, 2016 at 9:50 AM, Bryan Bende  wrote:
>
> > Andre,
> >
> > This definitely seems weird that somehow using embedded ZooKeeper is
> > causing this.
> >
> > One thing I can say though, is that in order to get into the code in your
> > stacktrace, it had to pass through SecurityUtil.
> isSecurityEnabled(config)
> > which does the following:
> >
> > public static boolean isSecurityEnabled(final Configuration config) {
> > Validate.notNull(config);
> > return "kerberos".equalsIgnoreCase(config.get("hadoop.security.
> > authentication"));
> > }
> >
> > The Configuration instance passed in is created using the default
> > constructor Configuration config = new Configuration(); and then any
> > files/paths entered into the processor's resource property is added to
> the
> > config.
> >
> > So in order for isSecurityEnabled to return true, it means the config
> > instance somehow got "hadoop.security.authentication" set to true, which
> > usually only happens if you put a core-site.xml on the classpath with
> that
> > value set.
> >
> > Is it possible some JAR from the MapR dependencies has a core-site.xml
> > embedded in it?
> >
> > -Bryan
> >
> > On Wed, Oct 26, 2016 at 6:09 AM, Andre  wrote:
> >
> >> Hi there,
> >>
> >> I've notice an odd behavior when using embedded Zookeeper on a NiFi
> >> cluster
> >> with MapR compatible processors:
> >>
> >> I noticed that every time I enable embedded zookeeper, NiFi's HDFS
> >> processors (e.g. PutHDFS) start complaining about Kerberos identities:
> >>
> >> 2016-10-26 20:07:22,376 ERROR [StandardProcessScheduler Thread-2]
> >> o.apache.nifi.processors.hadoop.PutHDFS
> >> java.io.IOException: Login failure for princical@REALM-NAME-GOES-HERE
> >> from
> >> keytab /path/to/keytab_file/nifi.keytab
> >> at
> >> org.apache.hadoop.security.UserGroupInformation.loginUserFro
> >> mKeytabAndReturnUGI(UserGroupInformation.java:1084)
> >> ~[hadoop-common-2.7.0-mapr-1602.jar:na]
> >> at
> >> org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:52)
> >> ~[nifi-hadoop-utils-1.0.0.jar:1.0.0]
> >> at
> >> org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.re
> >> setHDFSResources(AbstractHadoopProcessor.java:285)
> >> ~[nifi-hdfs-processors-1.0.0.jar:1.0.0]
> >> at
> >> org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.ab
> >> stractOnScheduled(AbstractHadoopProcessor.java:213)
> >> ~[nifi-hdfs-processors-1.0.0.jar:1.0.0]
> >> at
> >> org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:181)
> >> [nifi-hdfs-processors-1.0.0.jar:1.0.0]
> >>
> >> So far so good, these errors are quite familiar to people using NiFi
> >> against secure MapR clusters and caused by issues around the custom JAAS
> >> settings required by Java applications relying on the MapR client to
> work.
> >>
> >> The normal workaround this would be instructing NiFi to where the the
> JAAS
> >> settings via bootstrap.conf [1]:
> >>
> >> $grep jaas
> >> java.arg.15=-Djava.security.auth.login.config=./conf/nifi-jaas.conf
> >>
> >> The contents of nifi-jaas.conf are a copy of the relevant MapR JAAS
> >> stanza:
> >>
> >> While the workaround seems to work (still doing tests) I ask:
> >>
> >> Should setting
> >>
> >> nifi.state.management.embedded.zookeeper.start=true
> >>
> >> Cause this behavior?
> >>
> >> Cheers
> >>
> >
> >
>

Re: PutS3Object submitting file many times

[Joe Witt]

Joe

I have not looked at this code but can say that processors are or should be
built such that in error cases we prefer duplication over loss.  So a case
like you describe could be a good example where session commit fails or
something like it causing retries but in the mean time we keep sending to
the destination.  It generally can be avoided that it would be so
pronounced but just as a heads up I thought id call this out.

Thanks
Joe

On Oct 26, 2016 9:58 AM, "Joe Gresock"  wrote:

> Has anyone run into problems where PutS3Object submits the same file many
> (hundreds) of times?  This turned out to be a problem for me since we have
> bucket versioning enabled, so we have many copies of the same file.
>
> I can't reproduce this behavior anymore in the normal flow of things, but
> I'm wondering if it's possible for this to happen if the processor is
> forced to administratively yield (i.e., due to disk full or too many open
> files).
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*
>

PutS3Object submitting file many times

[Joe Gresock]

Has anyone run into problems where PutS3Object submits the same file many
(hundreds) of times?  This turned out to be a problem for me since we have
bucket versioning enabled, so we have many copies of the same file.

I can't reproduce this behavior anymore in the normal flow of things, but
I'm wondering if it's possible for this to happen if the processor is
forced to administratively yield (i.e., due to disk full or too many open
files).

-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*

Re: Odd inter-operation behavior when using clustered NiFi against secure MapR cluster

[Bryan Bende]

Meant to say the config instance somehow got
"hadoop.security.authentication"
set to "kerberos"

On Wed, Oct 26, 2016 at 9:50 AM, Bryan Bende  wrote:

> Andre,
>
> This definitely seems weird that somehow using embedded ZooKeeper is
> causing this.
>
> One thing I can say though, is that in order to get into the code in your
> stacktrace, it had to pass through SecurityUtil.isSecurityEnabled(config)
> which does the following:
>
> public static boolean isSecurityEnabled(final Configuration config) {
> Validate.notNull(config);
> return "kerberos".equalsIgnoreCase(config.get("hadoop.security.
> authentication"));
> }
>
> The Configuration instance passed in is created using the default
> constructor Configuration config = new Configuration(); and then any
> files/paths entered into the processor's resource property is added to the
> config.
>
> So in order for isSecurityEnabled to return true, it means the config
> instance somehow got "hadoop.security.authentication" set to true, which
> usually only happens if you put a core-site.xml on the classpath with that
> value set.
>
> Is it possible some JAR from the MapR dependencies has a core-site.xml
> embedded in it?
>
> -Bryan
>
> On Wed, Oct 26, 2016 at 6:09 AM, Andre  wrote:
>
>> Hi there,
>>
>> I've notice an odd behavior when using embedded Zookeeper on a NiFi
>> cluster
>> with MapR compatible processors:
>>
>> I noticed that every time I enable embedded zookeeper, NiFi's HDFS
>> processors (e.g. PutHDFS) start complaining about Kerberos identities:
>>
>> 2016-10-26 20:07:22,376 ERROR [StandardProcessScheduler Thread-2]
>> o.apache.nifi.processors.hadoop.PutHDFS
>> java.io.IOException: Login failure for princical@REALM-NAME-GOES-HERE
>> from
>> keytab /path/to/keytab_file/nifi.keytab
>> at
>> org.apache.hadoop.security.UserGroupInformation.loginUserFro
>> mKeytabAndReturnUGI(UserGroupInformation.java:1084)
>> ~[hadoop-common-2.7.0-mapr-1602.jar:na]
>> at
>> org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:52)
>> ~[nifi-hadoop-utils-1.0.0.jar:1.0.0]
>> at
>> org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.re
>> setHDFSResources(AbstractHadoopProcessor.java:285)
>> ~[nifi-hdfs-processors-1.0.0.jar:1.0.0]
>> at
>> org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.ab
>> stractOnScheduled(AbstractHadoopProcessor.java:213)
>> ~[nifi-hdfs-processors-1.0.0.jar:1.0.0]
>> at
>> org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:181)
>> [nifi-hdfs-processors-1.0.0.jar:1.0.0]
>>
>> So far so good, these errors are quite familiar to people using NiFi
>> against secure MapR clusters and caused by issues around the custom JAAS
>> settings required by Java applications relying on the MapR client to work.
>>
>> The normal workaround this would be instructing NiFi to where the the JAAS
>> settings via bootstrap.conf [1]:
>>
>> $grep jaas
>> java.arg.15=-Djava.security.auth.login.config=./conf/nifi-jaas.conf
>>
>> The contents of nifi-jaas.conf are a copy of the relevant MapR JAAS
>> stanza:
>>
>> While the workaround seems to work (still doing tests) I ask:
>>
>> Should setting
>>
>> nifi.state.management.embedded.zookeeper.start=true
>>
>> Cause this behavior?
>>
>> Cheers
>>
>
>

Re: Replace Text for updating the flow file content

[Matt Burgess]

Arsalan,

If I understand your use case correctly, you could use SplitJson with
a JSONPath of "$.Records", this will emit one flow file for each entry
in the array. Then you could use ConvertJSONtoAvro (it should infer
the schema but you could either use InferAvroSchema or supply one if
you know it). At that point you'd have flow files each containing a
single Avro record corresponding to an entry in the JSON "Records"
array.

If you'd rather keep the records together, you could use
JoltTransformJSON [1] processor to remove "NrRecords", "FileRecords",
etc. or just keep "Records". Then you could use ConvertJSONtoAvro
which should give you a single Avro file with multiple entries inside.

Regards,
Matt

[1] 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.JoltTransformJSON/index.html

On Wed, Oct 26, 2016 at 7:52 AM, Arsalan Siddiqi
 wrote:
> Hi I am not very familiar with Regexs. The following is the json format data
> being delivered to the replace text processor.
>
> {"NrRecords":1,"FileRecords":-1,"Records":[{"ISN":250,"AA":"1122","AF":"S","AG":"F"}]}
>
> I want to replace this with
> [{"ISN":250,"AA":"1122","AF":"S","AG":"F"}]
>
> and obviously the records can be more than one. I need to convert this JSON
> to avro so i would need to define the schema first. The infer schema would
> infer it based on the number of records that are retrieved. If they are more
> than one then it thinks that this entire file is one record and passing it
> through the "split avro" does not split it correctly (it does not split the
> file and the output is the same original file). The ideal case would be to
> get rid of this header in the file and then I think it will be able to
> process the records. I want to then split the file into seperate files equal
> to the number of records retrieved.
>
> I am not very familiar with regex, groups and back-references. I was
> wondering if there was a way to write a regex that extracts the value
> between the first [ and the last ] and assign it back to the flow file
> content.
>
> thanks
> Arsalan
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Replace-Text-for-updating-the-flow-file-content-tp13712.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Replace Text for updating the flow file content

[Arsalan Siddiqi]

Hi I am not very familiar with Regexs. The following is the json format data
being delivered to the replace text processor. 

{"NrRecords":1,"FileRecords":-1,"Records":[{"ISN":250,"AA":"1122","AF":"S","AG":"F"}]}

I want to replace this with 
[{"ISN":250,"AA":"1122","AF":"S","AG":"F"}]

and obviously the records can be more than one. I need to convert this JSON
to avro so i would need to define the schema first. The infer schema would
infer it based on the number of records that are retrieved. If they are more
than one then it thinks that this entire file is one record and passing it
through the "split avro" does not split it correctly (it does not split the
file and the output is the same original file). The ideal case would be to 
get rid of this header in the file and then I think it will be able to
process the records. I want to then split the file into seperate files equal
to the number of records retrieved.  

I am not very familiar with regex, groups and back-references. I was
wondering if there was a way to write a regex that extracts the value
between the first [ and the last ] and assign it back to the flow file
content. 

thanks
Arsalan



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Replace-Text-for-updating-the-flow-file-content-tp13712.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Odd inter-operation behavior when using clustered NiFi against secure MapR cluster

[Andre]

Hi there,

I've notice an odd behavior when using embedded Zookeeper on a NiFi cluster
with MapR compatible processors:

I noticed that every time I enable embedded zookeeper, NiFi's HDFS
processors (e.g. PutHDFS) start complaining about Kerberos identities:

2016-10-26 20:07:22,376 ERROR [StandardProcessScheduler Thread-2]
o.apache.nifi.processors.hadoop.PutHDFS
java.io.IOException: Login failure for princical@REALM-NAME-GOES-HERE from
keytab /path/to/keytab_file/nifi.keytab
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1084)
~[hadoop-common-2.7.0-mapr-1602.jar:na]
at
org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:52)
~[nifi-hadoop-utils-1.0.0.jar:1.0.0]
at
org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:285)
~[nifi-hdfs-processors-1.0.0.jar:1.0.0]
at
org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:213)
~[nifi-hdfs-processors-1.0.0.jar:1.0.0]
at
org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:181)
[nifi-hdfs-processors-1.0.0.jar:1.0.0]

So far so good, these errors are quite familiar to people using NiFi
against secure MapR clusters and caused by issues around the custom JAAS
settings required by Java applications relying on the MapR client to work.

The normal workaround this would be instructing NiFi to where the the JAAS
settings via bootstrap.conf [1]:

$grep jaas
java.arg.15=-Djava.security.auth.login.config=./conf/nifi-jaas.conf

The contents of nifi-jaas.conf are a copy of the relevant MapR JAAS stanza:

While the workaround seems to work (still doing tests) I ask:

Should setting

nifi.state.management.embedded.zookeeper.start=true

Cause this behavior?

Cheers