Re: Only get file when a set exists.

[Koji Kawamura]

The order of arrival does not matter.

Wait processor has 'Expiration Duration' configuration, defaults to 10
min. Please adjust it according to your needs, the longest period to
wait for a delayed file.
If a FlowFile exceeds the duration, it will be sent to 'expired'
relationship, and can be treated differently, e.g. write ERROR log

> If we have a longer wait for a file, we'd like processing for the next 
> groupid to still be able to continue.

In order to achieve that, use Wait Mode = 'Transfer to wait
relationship', and the 'wait' relationship should be configured to use
FirstInFirstOutPrioritizer.
If FirstInFirstOutPrioritizer is not set, the same FlowFile will be
processed again while it blocks other FlowFiles.
With FirstInFirstOutPrioritizer, the processed FlowFile will be
re-queued at the end of wait queue.

I've updated my example to make it more realistic, by adding delay for
certain set and type.
https://gist.github.com/ijokarumawak/06b3b071eeb4d10d8a27507981422edd

Thanks,
Koji

On Thu, May 31, 2018 at 10:56 AM, Martijn Dekkers
 wrote:
> Cool, that will make things a lot simpler. Does it matter that the ext2
> files arrive in random order? Sometimes there can be a very long delay in
> some of them showing up, and we have some concerns about the overall flow
> blocking. If we have a longer wait for a file, we'd like processing for the
> next groupid to still be able to continue.
>
> Thank you for your help (and for writing Wait/Notify!!)
>
> Martijn
>
> On 31 May 2018 at 03:49, Koji Kawamura  wrote:
>>
>> Glad to hear that was helpful.
>>
>> "4 same type for each extension", can be treated as "8 distinct types"
>> if an extension is included in a type.
>> ab.ex1, cd.ex1, ef.ex1, gh.ex1, ab.ex2, cd.ex2, ef.ex2, gh.ex2
>>
>> Then route only 'ab.ex1' (or whichever but just 1 of them) to the Wait
>> branch, and the rest to Notify branch.
>> That will simplify the flow, if I'm not missing any other requirement.
>>
>> Thanks!
>> Koji
>>
>> On Thu, May 31, 2018 at 10:30 AM, Martijn Dekkers
>>  wrote:
>> > Hi Koji, Many thanks for your continued assistance!
>> >
>> >>
>> >> - 1 file per second is relatively low in terms of traffic, it should
>> >> be processed fine with 1 thread
>> >> - A flow like this, which is stateful across different parts of the
>> >> flow works at best with single thread, because using multiple threads
>> >> would cause race condition or concurrency issue if there's any
>> >> implementation error
>> >
>> >
>> > Yes, we had similar thoughts.
>> >
>> >>
>> >> - Based on above, I strongly recommend to NOT increase "concurrent
>> >> tasks". If you see FlowFiles staying in a wait queue, then there must
>> >> be different issue
>> >
>> >
>> > We don't see many flowfiles stuck in a wait queue, I ran a test over a
>> > few
>> > hours yesterday that simulates the way in which these files would appear
>> > (we
>> > would have 4 of "ext1" show up every second, and the "ext2" can show up
>> > a
>> > few seconds later, and not always in the same order) and we found
>> > perhaps 6
>> > flowfiles stuck in a wait queue.
>> >
>> >>
>> >> - Also, using concurrent tasks number like 400 is too much in general
>> >> for all processors. I recommend to increment it as 2, 3, 4 .. up to 8
>> >> or so, only if you see the clear benefit by doing so
>> >
>> >
>> > Indeed, thanks for the suggestion. Once we have the logic finished and
>> > tested we will have to optimise this Flow. The next step is to try to
>> > load
>> > the required processors into MiNiFy, as this will be running on many
>> > systems
>> > with limited capacity. If we don't manage with MiNiFy, we will still be
>> > good, but we prefer to have the smaller footprint and ease of management
>> > we
>> > can obtain with MiNiFy.
>> >
>> >>
>> >> - The important part of this flow is extracting 'groupId' and 'type'
>> >> from file names. Regular Expression needs to be configured properly.
>> >> - I recommend using https://regex101.com/ to test your Regular
>> >> Expression to see whether it can extract correct groupId and type
>> >
>> >
>> > Yes, we have tested our RegExes for this extensively
>> >
>> >>
>> >>
>> >> Lastly, regardless of how many files should be there for 'ext1' and
>> >> 'ext2', the flow structure is simple as below.
>> >> Let's say there should be 8 files to start processing those.
>> >> 4 x ex1, and 4 ex2 in your case, but let's think it as 8 file types.
>> >> And I assume the types are known, meaning, static, not dynamically
>> >> change.
>> >
>> >
>> > Correct, the format is . where:
>> >
>> > - groupId is unique for each set of 8
>> > - type has 4 variants (ab, cd, ef, gh), the same type-set for each ext
>> >
>> >> So, the rule is, "a set of files consists of 8 files, and a set should
>> >> wait to be processed until all 8 files are ready", that's all.
>> >
>> >
>> > For our use case it is important that we have positive identification
>> > that
>> > we have exact "positive identification" of each file.
>> >
>> >>
>> >> 

Re: Only get file when a set exists.

[Martijn Dekkers]

Cool, that will make things a lot simpler. Does it matter that the ext2
files arrive in random order? Sometimes there can be a very long delay in
some of them showing up, and we have some concerns about the overall flow
blocking. If we have a longer wait for a file, we'd like processing for the
next groupid to still be able to continue.

Thank you for your help (and for writing Wait/Notify!!)

Martijn

On 31 May 2018 at 03:49, Koji Kawamura  wrote:

> Glad to hear that was helpful.
>
> "4 same type for each extension", can be treated as "8 distinct types"
> if an extension is included in a type.
> ab.ex1, cd.ex1, ef.ex1, gh.ex1, ab.ex2, cd.ex2, ef.ex2, gh.ex2
>
> Then route only 'ab.ex1' (or whichever but just 1 of them) to the Wait
> branch, and the rest to Notify branch.
> That will simplify the flow, if I'm not missing any other requirement.
>
> Thanks!
> Koji
>
> On Thu, May 31, 2018 at 10:30 AM, Martijn Dekkers
>  wrote:
> > Hi Koji, Many thanks for your continued assistance!
> >
> >>
> >> - 1 file per second is relatively low in terms of traffic, it should
> >> be processed fine with 1 thread
> >> - A flow like this, which is stateful across different parts of the
> >> flow works at best with single thread, because using multiple threads
> >> would cause race condition or concurrency issue if there's any
> >> implementation error
> >
> >
> > Yes, we had similar thoughts.
> >
> >>
> >> - Based on above, I strongly recommend to NOT increase "concurrent
> >> tasks". If you see FlowFiles staying in a wait queue, then there must
> >> be different issue
> >
> >
> > We don't see many flowfiles stuck in a wait queue, I ran a test over a
> few
> > hours yesterday that simulates the way in which these files would appear
> (we
> > would have 4 of "ext1" show up every second, and the "ext2" can show up a
> > few seconds later, and not always in the same order) and we found
> perhaps 6
> > flowfiles stuck in a wait queue.
> >
> >>
> >> - Also, using concurrent tasks number like 400 is too much in general
> >> for all processors. I recommend to increment it as 2, 3, 4 .. up to 8
> >> or so, only if you see the clear benefit by doing so
> >
> >
> > Indeed, thanks for the suggestion. Once we have the logic finished and
> > tested we will have to optimise this Flow. The next step is to try to
> load
> > the required processors into MiNiFy, as this will be running on many
> systems
> > with limited capacity. If we don't manage with MiNiFy, we will still be
> > good, but we prefer to have the smaller footprint and ease of management
> we
> > can obtain with MiNiFy.
> >
> >>
> >> - The important part of this flow is extracting 'groupId' and 'type'
> >> from file names. Regular Expression needs to be configured properly.
> >> - I recommend using https://regex101.com/ to test your Regular
> >> Expression to see whether it can extract correct groupId and type
> >
> >
> > Yes, we have tested our RegExes for this extensively
> >
> >>
> >>
> >> Lastly, regardless of how many files should be there for 'ext1' and
> >> 'ext2', the flow structure is simple as below.
> >> Let's say there should be 8 files to start processing those.
> >> 4 x ex1, and 4 ex2 in your case, but let's think it as 8 file types.
> >> And I assume the types are known, meaning, static, not dynamically
> change.
> >
> >
> > Correct, the format is . where:
> >
> > - groupId is unique for each set of 8
> > - type has 4 variants (ab, cd, ef, gh), the same type-set for each ext
> >
> >> So, the rule is, "a set of files consists of 8 files, and a set should
> >> wait to be processed until all 8 files are ready", that's all.
> >
> >
> > For our use case it is important that we have positive identification
> that
> > we have exact "positive identification" of each file.
> >
> >>
> >> Then, the flow should be designed like below:
> >> 1. List files, each file will be sent as a FlowFile
> >
> >
> > Correct - we have several different listfiles for other sections of the
> > flow, we are actually collecting many different sets, all variants of the
> > above. However, those are far simpler (sets of 2 - ext1 and ext2 only)
> >
> >>
> >> 2. Extract groupId and type from filename
> >
> >
> > Correct
> >
> >>
> >> 3. Route FlowFiles into two branches, let's call these 'Notify' branch
> >> and 'Wait' branch, and pass only 1 type for a set to Wait-branch, and
> >> the rest 7 types to Notify-branch
> >
> >
> > OK, we currently split Notify branch to "all ext1" and wait branch to
> "all
> > ext2"
> >
> >>
> >> At Notify branch (for the rest 7 types FlowFile, e.g. type 2, 3, 4 ...
> 8)
> >
> >
> > As mentioned, we only have 4 distinct types.
> >
> >>
> >> 1. Notify that the type for a group has arrived.
> >> 2. Discard the FlowFile, because there's nothing to do with it in this
> >> branch
> >
> >
> >
> >>
> >> At Wait branch (for the type 1 FlowFile)
> >> 1. Wait for type 2 for the groupId.
> >> 2. Wait for type 3 for the groupId, type 4, 5 and so on
> >> 3. After passing Wait 

Re: Only get file when a set exists.

[Koji Kawamura]

Glad to hear that was helpful.

"4 same type for each extension", can be treated as "8 distinct types"
if an extension is included in a type.
ab.ex1, cd.ex1, ef.ex1, gh.ex1, ab.ex2, cd.ex2, ef.ex2, gh.ex2

Then route only 'ab.ex1' (or whichever but just 1 of them) to the Wait
branch, and the rest to Notify branch.
That will simplify the flow, if I'm not missing any other requirement.

Thanks!
Koji

On Thu, May 31, 2018 at 10:30 AM, Martijn Dekkers
 wrote:
> Hi Koji, Many thanks for your continued assistance!
>
>>
>> - 1 file per second is relatively low in terms of traffic, it should
>> be processed fine with 1 thread
>> - A flow like this, which is stateful across different parts of the
>> flow works at best with single thread, because using multiple threads
>> would cause race condition or concurrency issue if there's any
>> implementation error
>
>
> Yes, we had similar thoughts.
>
>>
>> - Based on above, I strongly recommend to NOT increase "concurrent
>> tasks". If you see FlowFiles staying in a wait queue, then there must
>> be different issue
>
>
> We don't see many flowfiles stuck in a wait queue, I ran a test over a few
> hours yesterday that simulates the way in which these files would appear (we
> would have 4 of "ext1" show up every second, and the "ext2" can show up a
> few seconds later, and not always in the same order) and we found perhaps 6
> flowfiles stuck in a wait queue.
>
>>
>> - Also, using concurrent tasks number like 400 is too much in general
>> for all processors. I recommend to increment it as 2, 3, 4 .. up to 8
>> or so, only if you see the clear benefit by doing so
>
>
> Indeed, thanks for the suggestion. Once we have the logic finished and
> tested we will have to optimise this Flow. The next step is to try to load
> the required processors into MiNiFy, as this will be running on many systems
> with limited capacity. If we don't manage with MiNiFy, we will still be
> good, but we prefer to have the smaller footprint and ease of management we
> can obtain with MiNiFy.
>
>>
>> - The important part of this flow is extracting 'groupId' and 'type'
>> from file names. Regular Expression needs to be configured properly.
>> - I recommend using https://regex101.com/ to test your Regular
>> Expression to see whether it can extract correct groupId and type
>
>
> Yes, we have tested our RegExes for this extensively
>
>>
>>
>> Lastly, regardless of how many files should be there for 'ext1' and
>> 'ext2', the flow structure is simple as below.
>> Let's say there should be 8 files to start processing those.
>> 4 x ex1, and 4 ex2 in your case, but let's think it as 8 file types.
>> And I assume the types are known, meaning, static, not dynamically change.
>
>
> Correct, the format is . where:
>
> - groupId is unique for each set of 8
> - type has 4 variants (ab, cd, ef, gh), the same type-set for each ext
>
>> So, the rule is, "a set of files consists of 8 files, and a set should
>> wait to be processed until all 8 files are ready", that's all.
>
>
> For our use case it is important that we have positive identification that
> we have exact "positive identification" of each file.
>
>>
>> Then, the flow should be designed like below:
>> 1. List files, each file will be sent as a FlowFile
>
>
> Correct - we have several different listfiles for other sections of the
> flow, we are actually collecting many different sets, all variants of the
> above. However, those are far simpler (sets of 2 - ext1 and ext2 only)
>
>>
>> 2. Extract groupId and type from filename
>
>
> Correct
>
>>
>> 3. Route FlowFiles into two branches, let's call these 'Notify' branch
>> and 'Wait' branch, and pass only 1 type for a set to Wait-branch, and
>> the rest 7 types to Notify-branch
>
>
> OK, we currently split Notify branch to "all ext1" and wait branch to "all
> ext2"
>
>>
>> At Notify branch (for the rest 7 types FlowFile, e.g. type 2, 3, 4 ... 8)
>
>
> As mentioned, we only have 4 distinct types.
>
>>
>> 1. Notify that the type for a group has arrived.
>> 2. Discard the FlowFile, because there's nothing to do with it in this
>> branch
>
>
>
>>
>> At Wait branch (for the type 1 FlowFile)
>> 1. Wait for type 2 for the groupId.
>> 2. Wait for type 3 for the groupId, type 4, 5 and so on
>> 3. After passing Wait for type 8, it can guarantee that all 8 files
>> are available (unless there is any other program deleting those)
>> 4. Get actual file content using FetchFile, and process it
>
>
> Besides the "4 same types for each extension", this is configured as you
> describe.
>
>>
>> I hope it helps.
>>
>
> It does, thanks. I will extract this portion of the flow, sanitise, and send
> it along - easier to see than to describe :)
>
>
>>
>> Thanks,
>> Koji
>
>
> Thank you so much once again!
>
> Martijn
>
>
>
>>
>>
>> On Wed, May 30, 2018 at 6:10 PM, Martijn Dekkers 
>> wrote:
>> > Hey Pierre,
>> >
>> > Yes, we suspected as much, but we are only seeing this with the Wait
>> > processor. Possibly because that is the only 

Re: Only get file when a set exists.

[Martijn Dekkers]

Hi Koji, Many thanks for your continued assistance!


> - 1 file per second is relatively low in terms of traffic, it should
> be processed fine with 1 thread
> - A flow like this, which is stateful across different parts of the
> flow works at best with single thread, because using multiple threads
> would cause race condition or concurrency issue if there's any
> implementation error
>

Yes, we had similar thoughts.


> - Based on above, I strongly recommend to NOT increase "concurrent
> tasks". If you see FlowFiles staying in a wait queue, then there must
> be different issue
>

We don't see many flowfiles stuck in a wait queue, I ran a test over a few
hours yesterday that simulates the way in which these files would appear
(we would have 4 of "ext1" show up every second, and the "ext2" can show up
a few seconds later, and not always in the same order) and we found perhaps
6 flowfiles stuck in a wait queue.


> - Also, using concurrent tasks number like 400 is too much in general
> for all processors. I recommend to increment it as 2, 3, 4 .. up to 8
> or so, only if you see the clear benefit by doing so
>

Indeed, thanks for the suggestion. Once we have the logic finished and
tested we will have to optimise this Flow. The next step is to try to load
the required processors into MiNiFy, as this will be running on many
systems with limited capacity. If we don't manage with MiNiFy, we will
still be good, but we prefer to have the smaller footprint and ease of
management we can obtain with MiNiFy.


> - The important part of this flow is extracting 'groupId' and 'type'
> from file names. Regular Expression needs to be configured properly.
> - I recommend using https://regex101.com/ to test your Regular
> Expression to see whether it can extract correct groupId and type
>

Yes, we have tested our RegExes for this extensively


>
> Lastly, regardless of how many files should be there for 'ext1' and
> 'ext2', the flow structure is simple as below.
> Let's say there should be 8 files to start processing those.
> 4 x ex1, and 4 ex2 in your case, but let's think it as 8 file types.
> And I assume the types are known, meaning, static, not dynamically change.
>

Correct, the format is . where:

- groupId is unique for each set of 8
- type has 4 variants (ab, cd, ef, gh), the same type-set for each ext

So, the rule is, "a set of files consists of 8 files, and a set should
> wait to be processed until all 8 files are ready", that's all.
>

For our use case it is important that we have positive identification that
we have exact "positive identification" of each file.


> Then, the flow should be designed like below:
> 1. List files, each file will be sent as a FlowFile
>

Correct - we have several different listfiles for other sections of the
flow, we are actually collecting many different sets, all variants of the
above. However, those are far simpler (sets of 2 - ext1 and ext2 only)


> 2. Extract groupId and type from filename
>

Correct


> 3. Route FlowFiles into two branches, let's call these 'Notify' branch
> and 'Wait' branch, and pass only 1 type for a set to Wait-branch, and
> the rest 7 types to Notify-branch
>

OK, we currently split Notify branch to "all ext1" and wait branch to "all
ext2"


> At Notify branch (for the rest 7 types FlowFile, e.g. type 2, 3, 4 ... 8)
>

As mentioned, we only have 4 distinct types.


> 1. Notify that the type for a group has arrived.
> 2. Discard the FlowFile, because there's nothing to do with it in this
> branch
>



> At Wait branch (for the type 1 FlowFile)
> 1. Wait for type 2 for the groupId.
> 2. Wait for type 3 for the groupId, type 4, 5 and so on
> 3. After passing Wait for type 8, it can guarantee that all 8 files
> are available (unless there is any other program deleting those)
> 4. Get actual file content using FetchFile, and process it
>

Besides the "4 same types for each extension", this is configured as you
describe.


> I hope it helps.
>
>
It does, thanks. I will extract this portion of the flow, sanitise, and
send it along - easier to see than to describe :)



> Thanks,
> Koji


Thank you so much once again!

Martijn




>
> On Wed, May 30, 2018 at 6:10 PM, Martijn Dekkers 
> wrote:
> > Hey Pierre,
> >
> > Yes, we suspected as much, but we are only seeing this with the Wait
> > processor. Possibly because that is the only "blocking" we have in this
> > flow.
> >
> > Thanks for the clarification, much appreciated!
> >
> > Martijn
> >
> > On 30 May 2018 at 10:30, Pierre Villard 
> wrote:
> >>
> >> I'll let Koji give more information about the Wait/Notify, he is clearly
> >> the expert here.
> >>
> >> I'm just jumping in regarding your "and when viewing the queue, the
> dialog
> >> states that the queue is empty.". You're seeing this behavior because,
> even
> >> though the UI shows some flow files in the queue, the flow files are
> >> currently locked in the session of the running processor and you won't
> see
> >> flow files currently processed in a 

Re: Only get file when a set exists.

[Koji Kawamura]

Hi Martijn,

Thanks for elaborating your requirement.  Here are few comments:

- 1 file per second is relatively low in terms of traffic, it should
be processed fine with 1 thread
- A flow like this, which is stateful across different parts of the
flow works at best with single thread, because using multiple threads
would cause race condition or concurrency issue if there's any
implementation error
- Based on above, I strongly recommend to NOT increase "concurrent
tasks". If you see FlowFiles staying in a wait queue, then there must
be different issue
- Also, using concurrent tasks number like 400 is too much in general
for all processors. I recommend to increment it as 2, 3, 4 .. up to 8
or so, only if you see the clear benefit by doing so
- The important part of this flow is extracting 'groupId' and 'type'
from file names. Regular Expression needs to be configured properly.
- I recommend using https://regex101.com/ to test your Regular
Expression to see whether it can extract correct groupId and type

Lastly, regardless of how many files should be there for 'ext1' and
'ext2', the flow structure is simple as below.
Let's say there should be 8 files to start processing those.
4 x ex1, and 4 ex2 in your case, but let's think it as 8 file types.
And I assume the types are known, meaning, static, not dynamically change.
So, the rule is, "a set of files consists of 8 files, and a set should
wait to be processed until all 8 files are ready", that's all.

Then, the flow should be designed like below:
1. List files, each file will be sent as a FlowFile
2. Extract groupId and type from filename
3. Route FlowFiles into two branches, let's call these 'Notify' branch
and 'Wait' branch, and pass only 1 type for a set to Wait-branch, and
the rest 7 types to Notify-branch

At Notify branch (for the rest 7 types FlowFile, e.g. type 2, 3, 4 ... 8)
1. Notify that the type for a group has arrived.
2. Discard the FlowFile, because there's nothing to do with it in this branch

At Wait branch (for the type 1 FlowFile)
1. Wait for type 2 for the groupId.
2. Wait for type 3 for the groupId, type 4, 5 and so on
3. After passing Wait for type 8, it can guarantee that all 8 files
are available (unless there is any other program deleting those)
4. Get actual file content using FetchFile, and process it

I hope it helps.

Thanks,
Koji


On Wed, May 30, 2018 at 6:10 PM, Martijn Dekkers  wrote:
> Hey Pierre,
>
> Yes, we suspected as much, but we are only seeing this with the Wait
> processor. Possibly because that is the only "blocking" we have in this
> flow.
>
> Thanks for the clarification, much appreciated!
>
> Martijn
>
> On 30 May 2018 at 10:30, Pierre Villard  wrote:
>>
>> I'll let Koji give more information about the Wait/Notify, he is clearly
>> the expert here.
>>
>> I'm just jumping in regarding your "and when viewing the queue, the dialog
>> states that the queue is empty.". You're seeing this behavior because, even
>> though the UI shows some flow files in the queue, the flow files are
>> currently locked in the session of the running processor and you won't see
>> flow files currently processed in a session when listing a queue. If you
>> stop the processor, the session will be closed and you'll be able to list
>> the queue and see the flow files.
>>
>> I recall discussions in the past to improve the UX for this. Not sure we
>> have a JIRA for it though...
>>
>> Pierre
>>
>> 2018-05-30 8:26 GMT+02:00 Martijn Dekkers :
>>>
>>> Hi Koji,
>>>
>>> Thank you for responding. I had adjusted the run schedule to closely
>>> mimic our environment. We are expecting about 1 file per second or so.
>>> We are also seeing some random "orphans" sitting in a wait queue every
>>> now and again that don't trigger a debug message, and when viewing the
>>> queue, the dialog states that the queue is empty.
>>>
>>> We found the random "no signal found" issue to be significantly decreased
>>> when we increase the "concurrent tasks" to something large - currently set
>>> to 400 for all wait and notify processors.
>>>
>>> I do need to mention that our requirements had changed since you made the
>>> template, in that we are looking for a set of 8 files - 4 x "ext1" and 4 x
>>> "ext2" both with the same pattern: .ext1 or ext2
>>>
>>> We found that the best way to make this work was to add another
>>> wait/notify pair, each processor coming after the ones already there, with a
>>> simple counter against the groupID.
>>>
>>> I will export a template for you, many thanks for your help - I just need
>>> to spend some time sanitising the varies fields etc.
>>>
>>> Many thanks once again for your kind assistance.
>>>
>>> Martijn
>>>
>>> On 30 May 2018 at 08:14, Koji Kawamura  wrote:

 Hi Martjin,

 In my template, I was using 'Run Schedule' as '5 secs' for the Wait
 processors to avoid overusing CPU resource. However, if you expect
 more throughput, it should be lowered.
 Changed Run Schedule to 0 sec, and I passed 100 group of 

RE: User, Group in LDAP appear to be unknown to PutFile

[Shawn Weeks]

We’re using SSSD not LDAP  directly so ours look like this. You can use LDAP as 
the source for SSSD but  I’m not exactly sure why the behavior would be 
different.

passwd: files  sss
shadow: files  sss
group:files  sss

I’m looking back through the thread and I haven’t seen it mentioned, are you 
just using LDAP or is Kerberos in the mix, we’re using an integrated system 
with FreeIPA which might make some differences.

Can you post the exact error and jdk version? I’ll go take a look at the Oracle 
source to see exactly what Java is trying to do.

Thanks
Shawn

From: James McMahon 
Sent: Wednesday, May 30, 2018 1:41 PM
To: users@nifi.apache.org
Subject: Re: User, Group in LDAP appear to be unknown to PutFile

Yes sir - we are indeed able to create files with that group. By chance, are 
you using /etc/nsswitch.conf? Do your entries for passwd, shadow, and group 
look something like this?
passwd: files  ldap
shadow: files  ldap
group:files  ldap
(We did try to reverse that order to "ldap  files". The same warnings get 
thrown).

On Wed, May 30, 2018 at 11:11 AM, Shawn Weeks 
mailto:swe...@weeksconsulting.us>> wrote:
Logged in as the user NiFi is running as on the same host are you able to 
create files with that group? We use PutFile and none of our groups are local 
to the host.

Thanks
Shawn

From: James McMahon mailto:jsmcmah...@gmail.com>>
Sent: Wednesday, May 30, 2018 8:21 AM
To: users@nifi.apache.org
Subject: Re: User, Group in LDAP appear to be unknown to PutFile

I did indeed configure PutFile as follows:
Permissions . 775
Owner . nifi
Group . ext_dev
When nifi is in local /etc/passwd and ext_dev is in local /etc/group, the 
PutFile succeeds.
When neither exists in the local files, I get the Warning in both case and the 
file is output with nifi:nifi (the default, as you mentioned Pierre).
When one is put back in the local file and the other left in the ldap, the one 
in the local succeeds and the one only in the ldap still throws the Warning and 
defaults on the output file.
We have also tried to reorder in our nsswitch.conf. For instance, changing
group: files ldap
to
group: ldap files
This did not change the results.

From the NiFi processor, user and group resolution do not appear to be looking 
beyond a check in the local files.
I did double check that user nifi exists only in either the local file or the 
ldap. Likewise for group ext_dev.


On Wed, May 30, 2018 at 8:52 AM, Pierre Villard 
mailto:pierre.villard...@gmail.com>> wrote:
By default, PutFile will set the ownership of the file to the user running the 
NiFi instance (nifi if NiFi is running as nifi user). Then, if you configured a 
different ownership in the processor configuration it'll try to set the 
ownership using the username you configured in the processor. What did you set 
in the processor parameters for owner/group and permission?

2018-05-30 14:39 GMT+02:00 James McMahon 
mailto:jsmcmah...@gmail.com>>:
Yes sir, sure does. In this instance my user nifi does indeed resolve at the OS 
level - I think that gives us some confidence it does resolve. The 
lookupPrincipalByName(owner) within the PutFile is where I believe the failure 
is rooted, but I do not understand how that function executes its lookup. I'm 
trying to dig deeper into that now.

On Wed, May 30, 2018 at 8:30 AM, Pierre Villard 
mailto:pierre.villard...@gmail.com>> wrote:
I think we're saying the same :) Let me rephrase it differently: to set the 
owner of a file, the user needs to be resolved at OS level. If the user does 
not exist (from the OS point of view), NiFi won't be able to set the owner 
(even though the username is in the LDAP configured for NiFi authentication). 
Does it make more sense?

$ touch test.file
$ id toto
id: toto: no such user
$ chown toto test.file
chown: toto: illegal user name

Pierre




2018-05-30 14:13 GMT+02:00 James McMahon 
mailto:jsmcmah...@gmail.com>>:
I don't understand this:
"Until you can't resolve the user with OS commands, I don't think NiFi will be 
able to set the expected owner on the file"
Did you intend to say can there - don't we want to be able to resolve the user 
at the OS as an initial validation that we can get to the ldap and as 
prerequisite of Nifi being able to set the owner? And in this case I can indeed 
resolve at the OS level. That seems like a good thing to me.

On Wed, May 30, 2018 at 7:53 AM, Pierre Villard 
mailto:pierre.villard...@gmail.com>> wrote:
It depends how your OS is configured, you could leverage tools like SSSD to 
resolve users against your LDAP but that's something to be configured at OS 
level.
Until you can't resolve the user with OS commands, I don't think NiFi will be 
able to set the expected owner on the file.

2018-05-30 11:54 GMT+02:00 James McMahon 
mailto:jsmcmah...@gmail.com>>:
Hello Pierre, and thank you. The user in this case - nifi - is not in the local 
/etc/passwd and is in the ldap. I 

Re: User, Group in LDAP appear to be unknown to PutFile

[James McMahon]

Yes sir - we are indeed able to create files with that group. By chance,
are you using /etc/nsswitch.conf? Do your entries for passwd, shadow, and
group look something like this?
passwd: files  ldap
shadow: files  ldap
group:files  ldap
(We did try to reverse that order to "ldap  files". The same warnings get
thrown).

On Wed, May 30, 2018 at 11:11 AM, Shawn Weeks 
wrote:

> Logged in as the user NiFi is running as on the same host are you able to
> create files with that group? We use PutFile and none of our groups are
> local to the host.
>
>
>
> Thanks
>
> Shawn
>
>
>
> *From:* James McMahon 
> *Sent:* Wednesday, May 30, 2018 8:21 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: User, Group in LDAP appear to be unknown to PutFile
>
>
>
> I did indeed configure PutFile as follows:
>
> Permissions . 775
>
> Owner . nifi
>
> Group . ext_dev
>
> When nifi is in local /etc/passwd and ext_dev is in local /etc/group, the
> PutFile succeeds.
>
> When neither exists in the local files, I get the Warning in both case and
> the file is output with nifi:nifi (the default, as you mentioned Pierre).
>
> When one is put back in the local file and the other left in the ldap, the
> one in the local succeeds and the one only in the ldap still throws the
> Warning and defaults on the output file.
>
> We have also tried to reorder in our nsswitch.conf. For instance, changing
>
> group: files ldap
>
> to
>
> group: ldap files
>
> This did not change the results.
>
>
>
> From the NiFi processor, user and group resolution do not appear to be
> looking beyond a check in the local files.
>
> I did double check that user nifi exists only in either the local file or
> the ldap. Likewise for group ext_dev.
>
>
>
>
>
> On Wed, May 30, 2018 at 8:52 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
> By default, PutFile will set the ownership of the file to the user running
> the NiFi instance (nifi if NiFi is running as nifi user). Then, if you
> configured a different ownership in the processor configuration it'll try
> to set the ownership using the username you configured in the processor.
> What did you set in the processor parameters for owner/group and permission?
>
>
>
> 2018-05-30 14:39 GMT+02:00 James McMahon :
>
> Yes sir, sure does. In this instance my user nifi does indeed resolve at
> the OS level - I think that gives us some confidence it does resolve. The
> lookupPrincipalByName(owner) within the PutFile is where I believe the
> failure is rooted, but I do not understand how that function executes its
> lookup. I'm trying to dig deeper into that now.
>
>
>
> On Wed, May 30, 2018 at 8:30 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
> I think we're saying the same :) Let me rephrase it differently: to set
> the owner of a file, the user needs to be resolved at OS level. If the user
> does not exist (from the OS point of view), NiFi won't be able to set the
> owner (even though the username is in the LDAP configured for NiFi
> authentication). Does it make more sense?
>
>
>
> $ touch test.file
> $ id toto
>
> id: toto: no such user
>
> $ chown toto test.file
>
> chown: toto: illegal user name
>
>
>
> Pierre
>
>
>
>
>
>
>
>
>
> 2018-05-30 14:13 GMT+02:00 James McMahon :
>
> I don't understand this:
> "Until you *can't* resolve the user with OS commands, I don't think NiFi
> will be able to set the expected owner on the file"
>
> Did you intend to say can there - don't we want to be able to resolve the
> user at the OS as an initial validation that we can get to the ldap and as
> prerequisite of Nifi being able to set the owner? And in this case I can
> indeed resolve at the OS level. That seems like a good thing to me.
>
>
>
> On Wed, May 30, 2018 at 7:53 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
> It depends how your OS is configured, you could leverage tools like SSSD
> to resolve users against your LDAP but that's something to be configured at
> OS level.
>
> Until you can't resolve the user with OS commands, I don't think NiFi will
> be able to set the expected owner on the file.
>
>
>
> 2018-05-30 11:54 GMT+02:00 James McMahon :
>
> Hello Pierre, and thank you. The user in this case - nifi - is not in the
> local /etc/passwd and is in the ldap. I presume this will force the id
>  to resolve using the ldap, if it does resolve?  At the OS the id
> command returns the uid, the gid, and the groups to which user nifi has
> membership within the ldap.
>
>
>
> On Wed, May 30, 2018 at 4:37 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
> Hi Jim,
>
>
>
> LDAP for authentication and authorizations in NiFi has nothing to do with
> the processors.
>
> How processors are running/working is completely independent to the
> authN/authZ model you configure for NiFi.
>
>
>
> Regarding your error, I'd say that you get this error because user/group
> you're setting in the processor configuration cannot be resolved at OS
> level (even though 

Re: Hive connection Pool error

[Jeff]

Vishal Dutt,

Your issue relates to an existing JIRA [1] and as luck would have it, it's
already resolved! :) The fix for that JIRA [1] has been merged to master
and will be in the next NiFi release.

[1] https://issues.apache.org/jira/browse/NIFI-5134
[2] https://github.com/apache/nifi/pull/2667

On Wed, May 30, 2018 at 3:42 AM Pierre Villard 
wrote:

> Hi,
>
> Could you share additional details about the processor/CS configuration as
> well?
>
> Thanks
>
> 2018-05-30 7:03 GMT+02:00 Koji Kawamura :
>
>> Hello,
>>
>> Although I encountered various Kerberos related error, I haven't
>> encountered that one.
>> I tried to reproduce the same error by changing Kerberos related
>> configuration, but to no avail.
>> I recommend enabling Kerberos debug option for further debugging.
>>
>> You can add the option at nifi/conf/bootstrap.conf:
>> java.arg.19=-Dsun.security.krb5.debug=true
>>
>> Then debug logs are written to nifi/logs/nifi-bootstap.log
>>
>> Thanks,
>> Koji
>>
>> On Tue, May 29, 2018 at 10:31 PM, Vishal Dutt 
>> wrote:
>> > Hi ,
>> >
>> >
>> >
>> > We  are getting below error on randomly for few minutes and then goes
>> away,
>> > its coming in PUThiveql
>> >
>> >
>> >
>> >
>> >
>> > 2018-05-29 01:01:07,279 INFO [Timer-Driven Process Thread-95]
>> > org.apache.hive.jdbc.HiveConnection Will try to open client transport
>> with
>> > JDBC Uri:
>> > jdbc:hive2://
>> ctcl-hdpmaster2.msoit.com:1/default;principal=hive/_h...@msoit.com
>> >
>> > 2018-05-29 01:01:07,281 ERROR [Timer-Driven Process Thread-95]
>> > o.apache.thrift.transport.TSaslTransport SASL negotiation failure
>> >
>> > javax.security.sasl.SaslException: GSS initiate failed
>> >
>> > at
>> >
>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
>> >
>> > at
>> >
>> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
>> >
>> > at
>> > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>> >
>> > at
>> >
>> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>> >
>> > at
>> >
>> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>> >
>> > at
>> >
>> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>> >
>> > at java.security.AccessController.doPrivileged(Native Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:422)
>> >
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
>> >
>> > at
>> >
>> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
>> >
>> > at
>> >
>> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
>> >
>> > at
>> > org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)
>> >
>> > at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
>> >
>> > at
>> >
>> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
>> >
>> > at
>> >
>> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>> >
>> > at
>> >
>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
>> >
>> > at
>> >
>> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
>> >
>> > at
>> >
>> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
>> >
>> > at
>> >
>> org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(HiveConnectionPool.java:355)
>> >
>> > at java.security.AccessController.doPrivileged(Native Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:422)
>> >
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
>> >
>> > at
>> >
>> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:355)
>> >
>> > at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source)
>> >
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >
>> > at java.lang.reflect.Method.invoke(Method.java:498)
>> >
>> > at
>> >
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
>> >
>> > at com.sun.proxy.$Proxy97.getConnection(Unknown Source)
>> >
>> > at
>> >
>> org.apache.nifi.processors.hive.PutHiveQL.lambda$new$1(PutHiveQL.java:191)
>> >
>> > at
>> org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:96)
>> >
>> > at
>> >
>> org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:274)
>> >
>> > at
>> >
>> 

RE: User, Group in LDAP appear to be unknown to PutFile

[Shawn Weeks]

Logged in as the user NiFi is running as on the same host are you able to 
create files with that group? We use PutFile and none of our groups are local 
to the host.

Thanks
Shawn

From: James McMahon 
Sent: Wednesday, May 30, 2018 8:21 AM
To: users@nifi.apache.org
Subject: Re: User, Group in LDAP appear to be unknown to PutFile

I did indeed configure PutFile as follows:
Permissions . 775
Owner . nifi
Group . ext_dev
When nifi is in local /etc/passwd and ext_dev is in local /etc/group, the 
PutFile succeeds.
When neither exists in the local files, I get the Warning in both case and the 
file is output with nifi:nifi (the default, as you mentioned Pierre).
When one is put back in the local file and the other left in the ldap, the one 
in the local succeeds and the one only in the ldap still throws the Warning and 
defaults on the output file.
We have also tried to reorder in our nsswitch.conf. For instance, changing
group: files ldap
to
group: ldap files
This did not change the results.

From the NiFi processor, user and group resolution do not appear to be looking 
beyond a check in the local files.
I did double check that user nifi exists only in either the local file or the 
ldap. Likewise for group ext_dev.


On Wed, May 30, 2018 at 8:52 AM, Pierre Villard 
mailto:pierre.villard...@gmail.com>> wrote:
By default, PutFile will set the ownership of the file to the user running the 
NiFi instance (nifi if NiFi is running as nifi user). Then, if you configured a 
different ownership in the processor configuration it'll try to set the 
ownership using the username you configured in the processor. What did you set 
in the processor parameters for owner/group and permission?

2018-05-30 14:39 GMT+02:00 James McMahon 
mailto:jsmcmah...@gmail.com>>:
Yes sir, sure does. In this instance my user nifi does indeed resolve at the OS 
level - I think that gives us some confidence it does resolve. The 
lookupPrincipalByName(owner) within the PutFile is where I believe the failure 
is rooted, but I do not understand how that function executes its lookup. I'm 
trying to dig deeper into that now.

On Wed, May 30, 2018 at 8:30 AM, Pierre Villard 
mailto:pierre.villard...@gmail.com>> wrote:
I think we're saying the same :) Let me rephrase it differently: to set the 
owner of a file, the user needs to be resolved at OS level. If the user does 
not exist (from the OS point of view), NiFi won't be able to set the owner 
(even though the username is in the LDAP configured for NiFi authentication). 
Does it make more sense?

$ touch test.file
$ id toto
id: toto: no such user
$ chown toto test.file
chown: toto: illegal user name

Pierre




2018-05-30 14:13 GMT+02:00 James McMahon 
mailto:jsmcmah...@gmail.com>>:
I don't understand this:
"Until you can't resolve the user with OS commands, I don't think NiFi will be 
able to set the expected owner on the file"
Did you intend to say can there - don't we want to be able to resolve the user 
at the OS as an initial validation that we can get to the ldap and as 
prerequisite of Nifi being able to set the owner? And in this case I can indeed 
resolve at the OS level. That seems like a good thing to me.

On Wed, May 30, 2018 at 7:53 AM, Pierre Villard 
mailto:pierre.villard...@gmail.com>> wrote:
It depends how your OS is configured, you could leverage tools like SSSD to 
resolve users against your LDAP but that's something to be configured at OS 
level.
Until you can't resolve the user with OS commands, I don't think NiFi will be 
able to set the expected owner on the file.

2018-05-30 11:54 GMT+02:00 James McMahon 
mailto:jsmcmah...@gmail.com>>:
Hello Pierre, and thank you. The user in this case - nifi - is not in the local 
/etc/passwd and is in the ldap. I presume this will force the id  to 
resolve using the ldap, if it does resolve?  At the OS the id command returns 
the uid, the gid, and the groups to which user nifi has membership within the 
ldap.

On Wed, May 30, 2018 at 4:37 AM, Pierre Villard 
mailto:pierre.villard...@gmail.com>> wrote:
Hi Jim,

LDAP for authentication and authorizations in NiFi has nothing to do with the 
processors.
How processors are running/working is completely independent to the authN/authZ 
model you configure for NiFi.

Regarding your error, I'd say that you get this error because user/group you're 
setting in the processor configuration cannot be resolved at OS level (even 
though they exist in the LDAP, but again, that's totally unrelated). Something 
you can quickly check: can you resolve the username/group on the host where 
you're using PutFile processor? What do you get if you execute the following 
command: id ?

Pierre

2018-05-30 1:14 GMT+02:00 Joe Witt 
mailto:joe.w...@gmail.com>>:
jim

please only post to one list.

users is good for this.

thanks
joe

On Tue, May 29, 2018, 3:54 PM James McMahon 
mailto:jsmcmah...@gmail.com>> wrote:
Good evening. I have recently migrated my nifi service host server from local 

Re: User, Group in LDAP appear to be unknown to PutFile

[James McMahon]

I did indeed configure PutFile as follows:

Permissions . 775
Owner . nifi
Group . ext_dev

When nifi is in local /etc/passwd and ext_dev is in local /etc/group, the
PutFile succeeds.
When neither exists in the local files, I get the Warning in both case and
the file is output with nifi:nifi (the default, as you mentioned Pierre).
When one is put back in the local file and the other left in the ldap, the
one in the local succeeds and the one only in the ldap still throws the
Warning and defaults on the output file.

We have also tried to reorder in our nsswitch.conf. For instance, changing
group: files ldap
to
group: ldap files
This did not change the results.

>From the NiFi processor, user and group resolution do not appear to be
looking beyond a check in the local files.

I did double check that user nifi exists only in either the local file or
the ldap. Likewise for group ext_dev.


On Wed, May 30, 2018 at 8:52 AM, Pierre Villard  wrote:

> By default, PutFile will set the ownership of the file to the user running
> the NiFi instance (nifi if NiFi is running as nifi user). Then, if you
> configured a different ownership in the processor configuration it'll try
> to set the ownership using the username you configured in the processor.
> What did you set in the processor parameters for owner/group and permission?
>
> 2018-05-30 14:39 GMT+02:00 James McMahon :
>
>> Yes sir, sure does. In this instance my user nifi does indeed resolve at
>> the OS level - I think that gives us some confidence it does resolve. The
>> lookupPrincipalByName(owner) within the PutFile is where I believe the
>> failure is rooted, but I do not understand how that function executes its
>> lookup. I'm trying to dig deeper into that now.
>>
>> On Wed, May 30, 2018 at 8:30 AM, Pierre Villard <
>> pierre.villard...@gmail.com> wrote:
>>
>>> I think we're saying the same :) Let me rephrase it differently: to set
>>> the owner of a file, the user needs to be resolved at OS level. If the user
>>> does not exist (from the OS point of view), NiFi won't be able to set the
>>> owner (even though the username is in the LDAP configured for NiFi
>>> authentication). Does it make more sense?
>>>
>>> $ touch test.file
>>> $ id toto
>>> id: toto: no such user
>>> $ chown toto test.file
>>> chown: toto: illegal user name
>>>
>>> Pierre
>>>
>>>
>>>
>>>
>>> 2018-05-30 14:13 GMT+02:00 James McMahon :
>>>
 I don't understand this:
 "Until you *can't* resolve the user with OS commands, I don't think
 NiFi will be able to set the expected owner on the file"
 Did you intend to say can there - don't we want to be able to resolve
 the user at the OS as an initial validation that we can get to the ldap and
 as prerequisite of Nifi being able to set the owner? And in this case I can
 indeed resolve at the OS level. That seems like a good thing to me.

 On Wed, May 30, 2018 at 7:53 AM, Pierre Villard <
 pierre.villard...@gmail.com> wrote:

> It depends how your OS is configured, you could leverage tools like
> SSSD to resolve users against your LDAP but that's something to be
> configured at OS level.
> Until you can't resolve the user with OS commands, I don't think NiFi
> will be able to set the expected owner on the file.
>
> 2018-05-30 11:54 GMT+02:00 James McMahon :
>
>> Hello Pierre, and thank you. The user in this case - nifi - is not in
>> the local /etc/passwd and is in the ldap. I presume this will force the 
>> id
>>  to resolve using the ldap, if it does resolve?  At the OS the 
>> id
>> command returns the uid, the gid, and the groups to which user nifi has
>> membership within the ldap.
>>
>> On Wed, May 30, 2018 at 4:37 AM, Pierre Villard <
>> pierre.villard...@gmail.com> wrote:
>>
>>> Hi Jim,
>>>
>>> LDAP for authentication and authorizations in NiFi has nothing to do
>>> with the processors.
>>> How processors are running/working is completely independent to the
>>> authN/authZ model you configure for NiFi.
>>>
>>> Regarding your error, I'd say that you get this error because
>>> user/group you're setting in the processor configuration cannot be 
>>> resolved
>>> at OS level (even though they exist in the LDAP, but again, that's 
>>> totally
>>> unrelated). Something you can quickly check: can you resolve the
>>> username/group on the host where you're using PutFile processor? What do
>>> you get if you execute the following command: id ?
>>>
>>> Pierre
>>>
>>> 2018-05-30 1:14 GMT+02:00 Joe Witt :
>>>
 jim

 please only post to one list.

 users is good for this.

 thanks
 joe

 On Tue, May 29, 2018, 3:54 PM James McMahon 
 wrote:

> Good evening. I have recently migrated my nifi service host server
> from local 

Re: NiFi 1.6.0 fails to start

[Mike Thomsen]

Sounds like a pretty safe plan. You said your development instance is at
1.1 and your prod on 1.3. Is that right? If it is, you might want to
shutdown, backup NiFi and just do a clean installation without bringing
over your security settings until you have NiFi 1.6 up and running because
a lot has changed between 1.1 and 1.6 from what I understand.

On Wed, May 30, 2018 at 8:57 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
alexander.s...@nih.gov> wrote:

> Hi Mike,
>
>
>
> Thanks for the warning. In prod, we are still running NiFi 1.3 with Java
> 8. I guess for now, I will just install the JRE 8 onto the dev server, and
> experiment with NiFi 1.7 at a later stage.
>
>
>
> *From:* Mike Thomsen 
> *Sent:* Wednesday, May 30, 2018 8:49 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi 1.6.0 fails to start
>
>
>
> Alexander,
>
>
>
> You're welcome to try the patch in a dev environment and send us feedback
> on what you find. The more the merrier in that respect, however, just to be
> clear... this is absolutely not recommended as an immediate production fix
> for your issue (assuming this environment you're emailing about is a
> production one). If it is a prod environment, you have to go the Java 8
> route for now to resolve this for your client.
>
>
>
> Thanks,
>
>
>
> Mike
>
>
>
> On Wed, May 30, 2018 at 8:14 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.s...@nih.gov> wrote:
>
> Excellent, thanks everyone! Let me try the patch first. Plan B would be to
> install Java 8 and bootstrap NiFi to it.
>
>
>
> *From:* Mike Thomsen 
> *Sent:* Wednesday, May 30, 2018 7:59 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi 1.6.0 fails to start
>
>
>
> We added a patch for 1.7.0-SNAPSHOT that should handle that issue going
> forward. You're welcome to download the code and try a build (in your dev
> environment obviously) to see if that solves your problem.
>
>
>
> On Wed, May 30, 2018 at 7:43 AM Bryan Bende  wrote:
>
> Hello,
>
>
>
> There is some work being done to support Java 9/10, but at this time NiFi
> currently requires Java 8.
>
>
>
> There is an issue when using Java 9/10 where some javax.xml classes are
> now part of the JDK, but before NiFi had to bundle them, so it creates
> conflicting classes.
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
> On May 30, 2018, at 7:20 AM, Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.s...@nih.gov> wrote:
>
>
>
> Hello,
>
>
>
> We have recently tried to upgrade the development instance of NiFi running
> on Windows 2012 from 1.1.1 to 1.6.0, following the process described here
> .
> Both versions are very much vanilla in terms of configuration (no
> authentications), but v.1.6.0 fails to start – the content of the
> nifi-app.log file follows:
>
>
>
> 2018-05-25 14:57:14,032 INFO [main] org.eclipse.jetty.util.log Logging
> initialized @17556ms to org.eclipse.jetty.util.log.Slf4jLog
>
> 2018-05-25 14:57:14,404 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.util.ServiceConfigurationError:
> org.apache.nifi.authorization.UserGroupProvider:
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public
> no-arg constructor
>
> java.util.ServiceConfigurationError:
> org.apache.nifi.authorization.UserGroupProvider:
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public
> no-arg constructor
>
>at java.base/java.util.ServiceLoader.fail(Unknown Source)
>
>at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
>
>at java.base/java.util.ServiceLoader.access$1000(Unknown Source)
>
>at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
> Source)
>
>at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
> Source)
>
>at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
>
>at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
>
>at
> org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148)
>
>at
> org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123)
>
>at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771)
>
>at org.apache.nifi.NiFi.(NiFi.java:157)
>
>at org.apache.nifi.NiFi.(NiFi.java:71)
>
>at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
>
>at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
>
>at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown
> Source)
>
>at java.base/java.lang.Class.getConstructor0(Unknown Source)
>
>at java.base/java.lang.Class.getConstructor(Unknown Source)
>
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>
>at 

RE: NiFi 1.6.0 fails to start

[Saip, Alexander (NIH/CC/BTRIS) [C]]

Hi Mike,

Thanks for the warning. In prod, we are still running NiFi 1.3 with Java 8. I 
guess for now, I will just install the JRE 8 onto the dev server, and 
experiment with NiFi 1.7 at a later stage.

From: Mike Thomsen 
Sent: Wednesday, May 30, 2018 8:49 AM
To: users@nifi.apache.org
Subject: Re: NiFi 1.6.0 fails to start

Alexander,

You're welcome to try the patch in a dev environment and send us feedback on 
what you find. The more the merrier in that respect, however, just to be 
clear... this is absolutely not recommended as an immediate production fix for 
your issue (assuming this environment you're emailing about is a production 
one). If it is a prod environment, you have to go the Java 8 route for now to 
resolve this for your client.

Thanks,

Mike

On Wed, May 30, 2018 at 8:14 AM Saip, Alexander (NIH/CC/BTRIS) [C] 
mailto:alexander.s...@nih.gov>> wrote:
Excellent, thanks everyone! Let me try the patch first. Plan B would be to 
install Java 8 and bootstrap NiFi to it.

From: Mike Thomsen mailto:mikerthom...@gmail.com>>
Sent: Wednesday, May 30, 2018 7:59 AM
To: users@nifi.apache.org
Subject: Re: NiFi 1.6.0 fails to start

We added a patch for 1.7.0-SNAPSHOT that should handle that issue going 
forward. You're welcome to download the code and try a build (in your dev 
environment obviously) to see if that solves your problem.

On Wed, May 30, 2018 at 7:43 AM Bryan Bende 
mailto:bbe...@gmail.com>> wrote:
Hello,

There is some work being done to support Java 9/10, but at this time NiFi 
currently requires Java 8.

There is an issue when using Java 9/10 where some javax.xml classes are now 
part of the JDK, but before NiFi had to bundle them, so it creates conflicting 
classes.

Thanks,

Bryan

On May 30, 2018, at 7:20 AM, Saip, Alexander (NIH/CC/BTRIS) [C] 
mailto:alexander.s...@nih.gov>> wrote:

Hello,

We have recently tried to upgrade the development instance of NiFi running on 
Windows 2012 from 1.1.1 to 1.6.0, following the process described 
here. 
Both versions are very much vanilla in terms of configuration (no 
authentications), but v.1.6.0 fails to start – the content of the nifi-app.log 
file follows:

2018-05-25 14:57:14,032 INFO [main] org.eclipse.jetty.util.log Logging 
initialized @17556ms to org.eclipse.jetty.util.log.Slf4jLog
2018-05-25 14:57:14,404 ERROR [main] org.apache.nifi.NiFi Failure to launch 
NiFi due to java.util.ServiceConfigurationError: 
org.apache.nifi.authorization.UserGroupProvider: 
org.apache.nifi.authorization.FileUserGroupProvider Unable to get public no-arg 
constructor
java.util.ServiceConfigurationError: 
org.apache.nifi.authorization.UserGroupProvider: 
org.apache.nifi.authorization.FileUserGroupProvider Unable to get public no-arg 
constructor
   at java.base/java.util.ServiceLoader.fail(Unknown Source)
   at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
   at java.base/java.util.ServiceLoader.access$1000(Unknown Source)
   at 
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
 Source)
   at 
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown 
Source)
   at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
   at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
   at 
org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148)
   at 
org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123)
   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771)
   at org.apache.nifi.NiFi.(NiFi.java:157)
   at org.apache.nifi.NiFi.(NiFi.java:71)
   at org.apache.nifi.NiFi.main(NiFi.java:292)
Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
   at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
   at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown 
Source)
   at java.base/java.lang.Class.getConstructor0(Unknown Source)
   at java.base/java.lang.Class.getConstructor(Unknown Source)
   at java.base/java.util.ServiceLoader$1.run(Unknown Source)
   at java.base/java.util.ServiceLoader$1.run(Unknown Source)
   at java.base/java.security.AccessController.doPrivileged(Native Method)
   ... 12 common frames omitted
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
   at java.base/java.net.URLClassLoader.findClass(Unknown Source)
   at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
   at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
   ... 19 common frames omitted

As for the JVM on that server, here is its version:

>java -version
java version "10.0.1" 2018-04-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)

I have searched this mailing list 

Re: User, Group in LDAP appear to be unknown to PutFile

[Mike Thomsen]

Shot in the dark, if you have a user named nifi in the LDAP and one in the
OS it might not actually be treated as the same unless the OS is using LDAP
to provide the user listing. Something as simple as /etc/users having a
password for "nifi" and the LDAP not having it or it being a different hash
could be throwing it off. Might want to look for something like that.

On Wed, May 30, 2018 at 8:40 AM James McMahon  wrote:

> Yes sir, sure does. In this instance my user nifi does indeed resolve at
> the OS level - I think that gives us some confidence it does resolve. The
> lookupPrincipalByName(owner) within the PutFile is where I believe the
> failure is rooted, but I do not understand how that function executes its
> lookup. I'm trying to dig deeper into that now.
>
> On Wed, May 30, 2018 at 8:30 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> I think we're saying the same :) Let me rephrase it differently: to set
>> the owner of a file, the user needs to be resolved at OS level. If the user
>> does not exist (from the OS point of view), NiFi won't be able to set the
>> owner (even though the username is in the LDAP configured for NiFi
>> authentication). Does it make more sense?
>>
>> $ touch test.file
>> $ id toto
>> id: toto: no such user
>> $ chown toto test.file
>> chown: toto: illegal user name
>>
>> Pierre
>>
>>
>>
>>
>> 2018-05-30 14:13 GMT+02:00 James McMahon :
>>
>>> I don't understand this:
>>> "Until you *can't* resolve the user with OS commands, I don't think
>>> NiFi will be able to set the expected owner on the file"
>>> Did you intend to say can there - don't we want to be able to resolve
>>> the user at the OS as an initial validation that we can get to the ldap and
>>> as prerequisite of Nifi being able to set the owner? And in this case I can
>>> indeed resolve at the OS level. That seems like a good thing to me.
>>>
>>> On Wed, May 30, 2018 at 7:53 AM, Pierre Villard <
>>> pierre.villard...@gmail.com> wrote:
>>>
 It depends how your OS is configured, you could leverage tools like
 SSSD to resolve users against your LDAP but that's something to be
 configured at OS level.
 Until you can't resolve the user with OS commands, I don't think NiFi
 will be able to set the expected owner on the file.

 2018-05-30 11:54 GMT+02:00 James McMahon :

> Hello Pierre, and thank you. The user in this case - nifi - is not in
> the local /etc/passwd and is in the ldap. I presume this will force the id
>  to resolve using the ldap, if it does resolve?  At the OS the 
> id
> command returns the uid, the gid, and the groups to which user nifi has
> membership within the ldap.
>
> On Wed, May 30, 2018 at 4:37 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> Hi Jim,
>>
>> LDAP for authentication and authorizations in NiFi has nothing to do
>> with the processors.
>> How processors are running/working is completely independent to the
>> authN/authZ model you configure for NiFi.
>>
>> Regarding your error, I'd say that you get this error because
>> user/group you're setting in the processor configuration cannot be 
>> resolved
>> at OS level (even though they exist in the LDAP, but again, that's 
>> totally
>> unrelated). Something you can quickly check: can you resolve the
>> username/group on the host where you're using PutFile processor? What do
>> you get if you execute the following command: id ?
>>
>> Pierre
>>
>> 2018-05-30 1:14 GMT+02:00 Joe Witt :
>>
>>> jim
>>>
>>> please only post to one list.
>>>
>>> users is good for this.
>>>
>>> thanks
>>> joe
>>>
>>> On Tue, May 29, 2018, 3:54 PM James McMahon 
>>> wrote:
>>>
 Good evening. I have recently migrated my nifi service host server
 from local resolution of users and groups to use an LDAP server. I
 configured login-identity-providers.xml and
 nifi.security.user.login.identity.provider. I verified my 
 configuration is
 known to NiFi by first restarting my nifi service and then attempting a
 login to the URL by a user without a cert, forcing it to resolve using
 LDAP. This appeared to work.

 I then attempted to set my file owner and file group in a PutFile
 to a user and a group that are each in the LDAP. The PutFile throws a
 Warning for both owner and group:
 java.nio.file.attribute.UserPrincipalNotFoundException. The file is 
 still
 output by the processor. It appears to default the user and owner to 
 nifi.

 A cursory review of the PutFile source shows that PutFile employs
 getUserPrincipalLookupService() when it seemingly tries to validate the
 user and group.

 How can I get this to resolve through the LDAP for the PutFile?

 

Re: User, Group in LDAP appear to be unknown to PutFile

[Pierre Villard]

By default, PutFile will set the ownership of the file to the user running
the NiFi instance (nifi if NiFi is running as nifi user). Then, if you
configured a different ownership in the processor configuration it'll try
to set the ownership using the username you configured in the processor.
What did you set in the processor parameters for owner/group and permission?

2018-05-30 14:39 GMT+02:00 James McMahon :

> Yes sir, sure does. In this instance my user nifi does indeed resolve at
> the OS level - I think that gives us some confidence it does resolve. The
> lookupPrincipalByName(owner) within the PutFile is where I believe the
> failure is rooted, but I do not understand how that function executes its
> lookup. I'm trying to dig deeper into that now.
>
> On Wed, May 30, 2018 at 8:30 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> I think we're saying the same :) Let me rephrase it differently: to set
>> the owner of a file, the user needs to be resolved at OS level. If the user
>> does not exist (from the OS point of view), NiFi won't be able to set the
>> owner (even though the username is in the LDAP configured for NiFi
>> authentication). Does it make more sense?
>>
>> $ touch test.file
>> $ id toto
>> id: toto: no such user
>> $ chown toto test.file
>> chown: toto: illegal user name
>>
>> Pierre
>>
>>
>>
>>
>> 2018-05-30 14:13 GMT+02:00 James McMahon :
>>
>>> I don't understand this:
>>> "Until you *can't* resolve the user with OS commands, I don't think
>>> NiFi will be able to set the expected owner on the file"
>>> Did you intend to say can there - don't we want to be able to resolve
>>> the user at the OS as an initial validation that we can get to the ldap and
>>> as prerequisite of Nifi being able to set the owner? And in this case I can
>>> indeed resolve at the OS level. That seems like a good thing to me.
>>>
>>> On Wed, May 30, 2018 at 7:53 AM, Pierre Villard <
>>> pierre.villard...@gmail.com> wrote:
>>>
 It depends how your OS is configured, you could leverage tools like
 SSSD to resolve users against your LDAP but that's something to be
 configured at OS level.
 Until you can't resolve the user with OS commands, I don't think NiFi
 will be able to set the expected owner on the file.

 2018-05-30 11:54 GMT+02:00 James McMahon :

> Hello Pierre, and thank you. The user in this case - nifi - is not in
> the local /etc/passwd and is in the ldap. I presume this will force the id
>  to resolve using the ldap, if it does resolve?  At the OS the 
> id
> command returns the uid, the gid, and the groups to which user nifi has
> membership within the ldap.
>
> On Wed, May 30, 2018 at 4:37 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> Hi Jim,
>>
>> LDAP for authentication and authorizations in NiFi has nothing to do
>> with the processors.
>> How processors are running/working is completely independent to the
>> authN/authZ model you configure for NiFi.
>>
>> Regarding your error, I'd say that you get this error because
>> user/group you're setting in the processor configuration cannot be 
>> resolved
>> at OS level (even though they exist in the LDAP, but again, that's 
>> totally
>> unrelated). Something you can quickly check: can you resolve the
>> username/group on the host where you're using PutFile processor? What do
>> you get if you execute the following command: id ?
>>
>> Pierre
>>
>> 2018-05-30 1:14 GMT+02:00 Joe Witt :
>>
>>> jim
>>>
>>> please only post to one list.
>>>
>>> users is good for this.
>>>
>>> thanks
>>> joe
>>>
>>> On Tue, May 29, 2018, 3:54 PM James McMahon 
>>> wrote:
>>>
 Good evening. I have recently migrated my nifi service host server
 from local resolution of users and groups to use an LDAP server. I
 configured login-identity-providers.xml and 
 nifi.security.user.login.identity.provider.
 I verified my configuration is known to NiFi by first restarting my 
 nifi
 service and then attempting a login to the URL by a user without a 
 cert,
 forcing it to resolve using LDAP. This appeared to work.

 I then attempted to set my file owner and file group in a PutFile
 to a user and a group that are each in the LDAP. The PutFile throws a
 Warning for both owner and group: 
 java.nio.file.attribute.UserPrincipalNotFoundException.
 The file is still output by the processor. It appears to default the 
 user
 and owner to nifi.

 A cursory review of the PutFile source shows that PutFile employs
 getUserPrincipalLookupService() when it seemingly tries to
 validate the user and group.

 How can I get this to resolve through the LDAP for the PutFile?

 

Re: NiFi 1.6.0 fails to start

[Mike Thomsen]

Alexander,

You're welcome to try the patch in a dev environment and send us feedback
on what you find. The more the merrier in that respect, however, just to be
clear... this is absolutely not recommended as an immediate production fix
for your issue (assuming this environment you're emailing about is a
production one). If it is a prod environment, you have to go the Java 8
route for now to resolve this for your client.

Thanks,

Mike

On Wed, May 30, 2018 at 8:14 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
alexander.s...@nih.gov> wrote:

> Excellent, thanks everyone! Let me try the patch first. Plan B would be to
> install Java 8 and bootstrap NiFi to it.
>
>
>
> *From:* Mike Thomsen 
> *Sent:* Wednesday, May 30, 2018 7:59 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi 1.6.0 fails to start
>
>
>
> We added a patch for 1.7.0-SNAPSHOT that should handle that issue going
> forward. You're welcome to download the code and try a build (in your dev
> environment obviously) to see if that solves your problem.
>
>
>
> On Wed, May 30, 2018 at 7:43 AM Bryan Bende  wrote:
>
> Hello,
>
>
>
> There is some work being done to support Java 9/10, but at this time NiFi
> currently requires Java 8.
>
>
>
> There is an issue when using Java 9/10 where some javax.xml classes are
> now part of the JDK, but before NiFi had to bundle them, so it creates
> conflicting classes.
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
> On May 30, 2018, at 7:20 AM, Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.s...@nih.gov> wrote:
>
>
>
> Hello,
>
>
>
> We have recently tried to upgrade the development instance of NiFi running
> on Windows 2012 from 1.1.1 to 1.6.0, following the process described here
> .
> Both versions are very much vanilla in terms of configuration (no
> authentications), but v.1.6.0 fails to start – the content of the
> nifi-app.log file follows:
>
>
>
> 2018-05-25 14:57:14,032 INFO [main] org.eclipse.jetty.util.log Logging
> initialized @17556ms to org.eclipse.jetty.util.log.Slf4jLog
>
> 2018-05-25 14:57:14,404 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.util.ServiceConfigurationError:
> org.apache.nifi.authorization.UserGroupProvider:
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public
> no-arg constructor
>
> java.util.ServiceConfigurationError:
> org.apache.nifi.authorization.UserGroupProvider:
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public
> no-arg constructor
>
>at java.base/java.util.ServiceLoader.fail(Unknown Source)
>
>at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
>
>at java.base/java.util.ServiceLoader.access$1000(Unknown Source)
>
>at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
> Source)
>
>at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
> Source)
>
>at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
>
>at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
>
>at
> org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148)
>
>at
> org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123)
>
>at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771)
>
>at org.apache.nifi.NiFi.(NiFi.java:157)
>
>at org.apache.nifi.NiFi.(NiFi.java:71)
>
>at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
>
>at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
>
>at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown
> Source)
>
>at java.base/java.lang.Class.getConstructor0(Unknown Source)
>
>at java.base/java.lang.Class.getConstructor(Unknown Source)
>
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>
>at java.base/java.security.AccessController.doPrivileged(Native
> Method)
>
>... 12 common frames omitted
>
> Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
>
>at java.base/java.net.URLClassLoader.findClass(Unknown Source)
>
>at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
>
>at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
>
>... 19 common frames omitted
>
>
>
> As for the JVM on that server, here is its version:
>
>
>
> >java -version
>
> java version "10.0.1" 2018-04-17
>
> Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
>
> Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)
>
>
>
> I have searched this mailing list archive and Google for a clue, but was
> unsuccessful so far.
>
>
>
> Thanks in advance,
>
>
>
> Alexander
>
>
>
>

Re: User, Group in LDAP appear to be unknown to PutFile

[James McMahon]

Yes sir, sure does. In this instance my user nifi does indeed resolve at
the OS level - I think that gives us some confidence it does resolve. The
lookupPrincipalByName(owner) within the PutFile is where I believe the
failure is rooted, but I do not understand how that function executes its
lookup. I'm trying to dig deeper into that now.

On Wed, May 30, 2018 at 8:30 AM, Pierre Villard  wrote:

> I think we're saying the same :) Let me rephrase it differently: to set
> the owner of a file, the user needs to be resolved at OS level. If the user
> does not exist (from the OS point of view), NiFi won't be able to set the
> owner (even though the username is in the LDAP configured for NiFi
> authentication). Does it make more sense?
>
> $ touch test.file
> $ id toto
> id: toto: no such user
> $ chown toto test.file
> chown: toto: illegal user name
>
> Pierre
>
>
>
>
> 2018-05-30 14:13 GMT+02:00 James McMahon :
>
>> I don't understand this:
>> "Until you *can't* resolve the user with OS commands, I don't think NiFi
>> will be able to set the expected owner on the file"
>> Did you intend to say can there - don't we want to be able to resolve the
>> user at the OS as an initial validation that we can get to the ldap and as
>> prerequisite of Nifi being able to set the owner? And in this case I can
>> indeed resolve at the OS level. That seems like a good thing to me.
>>
>> On Wed, May 30, 2018 at 7:53 AM, Pierre Villard <
>> pierre.villard...@gmail.com> wrote:
>>
>>> It depends how your OS is configured, you could leverage tools like SSSD
>>> to resolve users against your LDAP but that's something to be configured at
>>> OS level.
>>> Until you can't resolve the user with OS commands, I don't think NiFi
>>> will be able to set the expected owner on the file.
>>>
>>> 2018-05-30 11:54 GMT+02:00 James McMahon :
>>>
 Hello Pierre, and thank you. The user in this case - nifi - is not in
 the local /etc/passwd and is in the ldap. I presume this will force the id
  to resolve using the ldap, if it does resolve?  At the OS the id
 command returns the uid, the gid, and the groups to which user nifi has
 membership within the ldap.

 On Wed, May 30, 2018 at 4:37 AM, Pierre Villard <
 pierre.villard...@gmail.com> wrote:

> Hi Jim,
>
> LDAP for authentication and authorizations in NiFi has nothing to do
> with the processors.
> How processors are running/working is completely independent to the
> authN/authZ model you configure for NiFi.
>
> Regarding your error, I'd say that you get this error because
> user/group you're setting in the processor configuration cannot be 
> resolved
> at OS level (even though they exist in the LDAP, but again, that's totally
> unrelated). Something you can quickly check: can you resolve the
> username/group on the host where you're using PutFile processor? What do
> you get if you execute the following command: id ?
>
> Pierre
>
> 2018-05-30 1:14 GMT+02:00 Joe Witt :
>
>> jim
>>
>> please only post to one list.
>>
>> users is good for this.
>>
>> thanks
>> joe
>>
>> On Tue, May 29, 2018, 3:54 PM James McMahon 
>> wrote:
>>
>>> Good evening. I have recently migrated my nifi service host server
>>> from local resolution of users and groups to use an LDAP server. I
>>> configured login-identity-providers.xml and 
>>> nifi.security.user.login.identity.provider.
>>> I verified my configuration is known to NiFi by first restarting my nifi
>>> service and then attempting a login to the URL by a user without a cert,
>>> forcing it to resolve using LDAP. This appeared to work.
>>>
>>> I then attempted to set my file owner and file group in a PutFile to
>>> a user and a group that are each in the LDAP. The PutFile throws a 
>>> Warning
>>> for both owner and group: 
>>> java.nio.file.attribute.UserPrincipalNotFoundException.
>>> The file is still output by the processor. It appears to default the 
>>> user
>>> and owner to nifi.
>>>
>>> A cursory review of the PutFile source shows that PutFile employs
>>> getUserPrincipalLookupService() when it seemingly tries to validate
>>> the user and group.
>>>
>>> How can I get this to resolve through the LDAP for the PutFile?
>>>
>>> Thanks for any insights.  -Jim
>>>
>>
>

>>>
>>
>

Re: User, Group in LDAP appear to be unknown to PutFile

[Pierre Villard]

I think we're saying the same :) Let me rephrase it differently: to set the
owner of a file, the user needs to be resolved at OS level. If the user
does not exist (from the OS point of view), NiFi won't be able to set the
owner (even though the username is in the LDAP configured for NiFi
authentication). Does it make more sense?

$ touch test.file
$ id toto
id: toto: no such user
$ chown toto test.file
chown: toto: illegal user name

Pierre




2018-05-30 14:13 GMT+02:00 James McMahon :

> I don't understand this:
> "Until you *can't* resolve the user with OS commands, I don't think NiFi
> will be able to set the expected owner on the file"
> Did you intend to say can there - don't we want to be able to resolve the
> user at the OS as an initial validation that we can get to the ldap and as
> prerequisite of Nifi being able to set the owner? And in this case I can
> indeed resolve at the OS level. That seems like a good thing to me.
>
> On Wed, May 30, 2018 at 7:53 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> It depends how your OS is configured, you could leverage tools like SSSD
>> to resolve users against your LDAP but that's something to be configured at
>> OS level.
>> Until you can't resolve the user with OS commands, I don't think NiFi
>> will be able to set the expected owner on the file.
>>
>> 2018-05-30 11:54 GMT+02:00 James McMahon :
>>
>>> Hello Pierre, and thank you. The user in this case - nifi - is not in
>>> the local /etc/passwd and is in the ldap. I presume this will force the id
>>>  to resolve using the ldap, if it does resolve?  At the OS the id
>>> command returns the uid, the gid, and the groups to which user nifi has
>>> membership within the ldap.
>>>
>>> On Wed, May 30, 2018 at 4:37 AM, Pierre Villard <
>>> pierre.villard...@gmail.com> wrote:
>>>
 Hi Jim,

 LDAP for authentication and authorizations in NiFi has nothing to do
 with the processors.
 How processors are running/working is completely independent to the
 authN/authZ model you configure for NiFi.

 Regarding your error, I'd say that you get this error because
 user/group you're setting in the processor configuration cannot be resolved
 at OS level (even though they exist in the LDAP, but again, that's totally
 unrelated). Something you can quickly check: can you resolve the
 username/group on the host where you're using PutFile processor? What do
 you get if you execute the following command: id ?

 Pierre

 2018-05-30 1:14 GMT+02:00 Joe Witt :

> jim
>
> please only post to one list.
>
> users is good for this.
>
> thanks
> joe
>
> On Tue, May 29, 2018, 3:54 PM James McMahon 
> wrote:
>
>> Good evening. I have recently migrated my nifi service host server
>> from local resolution of users and groups to use an LDAP server. I
>> configured login-identity-providers.xml and 
>> nifi.security.user.login.identity.provider.
>> I verified my configuration is known to NiFi by first restarting my nifi
>> service and then attempting a login to the URL by a user without a cert,
>> forcing it to resolve using LDAP. This appeared to work.
>>
>> I then attempted to set my file owner and file group in a PutFile to
>> a user and a group that are each in the LDAP. The PutFile throws a 
>> Warning
>> for both owner and group: 
>> java.nio.file.attribute.UserPrincipalNotFoundException.
>> The file is still output by the processor. It appears to default the user
>> and owner to nifi.
>>
>> A cursory review of the PutFile source shows that PutFile employs
>> getUserPrincipalLookupService() when it seemingly tries to validate
>> the user and group.
>>
>> How can I get this to resolve through the LDAP for the PutFile?
>>
>> Thanks for any insights.  -Jim
>>
>

>>>
>>
>

RE: NiFi 1.6.0 fails to start

[Saip, Alexander (NIH/CC/BTRIS) [C]]

Excellent, thanks everyone! Let me try the patch first. Plan B would be to 
install Java 8 and bootstrap NiFi to it.

From: Mike Thomsen 
Sent: Wednesday, May 30, 2018 7:59 AM
To: users@nifi.apache.org
Subject: Re: NiFi 1.6.0 fails to start

We added a patch for 1.7.0-SNAPSHOT that should handle that issue going 
forward. You're welcome to download the code and try a build (in your dev 
environment obviously) to see if that solves your problem.

On Wed, May 30, 2018 at 7:43 AM Bryan Bende 
mailto:bbe...@gmail.com>> wrote:
Hello,

There is some work being done to support Java 9/10, but at this time NiFi 
currently requires Java 8.

There is an issue when using Java 9/10 where some javax.xml classes are now 
part of the JDK, but before NiFi had to bundle them, so it creates conflicting 
classes.

Thanks,

Bryan


On May 30, 2018, at 7:20 AM, Saip, Alexander (NIH/CC/BTRIS) [C] 
mailto:alexander.s...@nih.gov>> wrote:

Hello,

We have recently tried to upgrade the development instance of NiFi running on 
Windows 2012 from 1.1.1 to 1.6.0, following the process described 
here. 
Both versions are very much vanilla in terms of configuration (no 
authentications), but v.1.6.0 fails to start – the content of the nifi-app.log 
file follows:

2018-05-25 14:57:14,032 INFO [main] org.eclipse.jetty.util.log Logging 
initialized @17556ms to org.eclipse.jetty.util.log.Slf4jLog
2018-05-25 14:57:14,404 ERROR [main] org.apache.nifi.NiFi Failure to launch 
NiFi due to java.util.ServiceConfigurationError: 
org.apache.nifi.authorization.UserGroupProvider: 
org.apache.nifi.authorization.FileUserGroupProvider Unable to get public no-arg 
constructor
java.util.ServiceConfigurationError: 
org.apache.nifi.authorization.UserGroupProvider: 
org.apache.nifi.authorization.FileUserGroupProvider Unable to get public no-arg 
constructor
   at java.base/java.util.ServiceLoader.fail(Unknown Source)
   at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
   at java.base/java.util.ServiceLoader.access$1000(Unknown Source)
   at 
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
 Source)
   at 
java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown 
Source)
   at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
   at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
   at 
org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148)
   at 
org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123)
   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771)
   at org.apache.nifi.NiFi.(NiFi.java:157)
   at org.apache.nifi.NiFi.(NiFi.java:71)
   at org.apache.nifi.NiFi.main(NiFi.java:292)
Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
   at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
   at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown 
Source)
   at java.base/java.lang.Class.getConstructor0(Unknown Source)
   at java.base/java.lang.Class.getConstructor(Unknown Source)
   at java.base/java.util.ServiceLoader$1.run(Unknown Source)
   at java.base/java.util.ServiceLoader$1.run(Unknown Source)
   at java.base/java.security.AccessController.doPrivileged(Native Method)
   ... 12 common frames omitted
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
   at java.base/java.net.URLClassLoader.findClass(Unknown Source)
   at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
   at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
   ... 19 common frames omitted

As for the JVM on that server, here is its version:

>java -version
java version "10.0.1" 2018-04-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)

I have searched this mailing list archive and Google for a clue, but was 
unsuccessful so far.

Thanks in advance,

Alexander

Re: User, Group in LDAP appear to be unknown to PutFile

[James McMahon]

I don't understand this:
"Until you *can't* resolve the user with OS commands, I don't think NiFi
will be able to set the expected owner on the file"
Did you intend to say can there - don't we want to be able to resolve the
user at the OS as an initial validation that we can get to the ldap and as
prerequisite of Nifi being able to set the owner? And in this case I can
indeed resolve at the OS level. That seems like a good thing to me.

On Wed, May 30, 2018 at 7:53 AM, Pierre Villard  wrote:

> It depends how your OS is configured, you could leverage tools like SSSD
> to resolve users against your LDAP but that's something to be configured at
> OS level.
> Until you can't resolve the user with OS commands, I don't think NiFi will
> be able to set the expected owner on the file.
>
> 2018-05-30 11:54 GMT+02:00 James McMahon :
>
>> Hello Pierre, and thank you. The user in this case - nifi - is not in the
>> local /etc/passwd and is in the ldap. I presume this will force the id
>>  to resolve using the ldap, if it does resolve?  At the OS the id
>> command returns the uid, the gid, and the groups to which user nifi has
>> membership within the ldap.
>>
>> On Wed, May 30, 2018 at 4:37 AM, Pierre Villard <
>> pierre.villard...@gmail.com> wrote:
>>
>>> Hi Jim,
>>>
>>> LDAP for authentication and authorizations in NiFi has nothing to do
>>> with the processors.
>>> How processors are running/working is completely independent to the
>>> authN/authZ model you configure for NiFi.
>>>
>>> Regarding your error, I'd say that you get this error because user/group
>>> you're setting in the processor configuration cannot be resolved at OS
>>> level (even though they exist in the LDAP, but again, that's totally
>>> unrelated). Something you can quickly check: can you resolve the
>>> username/group on the host where you're using PutFile processor? What do
>>> you get if you execute the following command: id ?
>>>
>>> Pierre
>>>
>>> 2018-05-30 1:14 GMT+02:00 Joe Witt :
>>>
 jim

 please only post to one list.

 users is good for this.

 thanks
 joe

 On Tue, May 29, 2018, 3:54 PM James McMahon 
 wrote:

> Good evening. I have recently migrated my nifi service host server
> from local resolution of users and groups to use an LDAP server. I
> configured login-identity-providers.xml and 
> nifi.security.user.login.identity.provider.
> I verified my configuration is known to NiFi by first restarting my nifi
> service and then attempting a login to the URL by a user without a cert,
> forcing it to resolve using LDAP. This appeared to work.
>
> I then attempted to set my file owner and file group in a PutFile to a
> user and a group that are each in the LDAP. The PutFile throws a Warning
> for both owner and group: 
> java.nio.file.attribute.UserPrincipalNotFoundException.
> The file is still output by the processor. It appears to default the user
> and owner to nifi.
>
> A cursory review of the PutFile source shows that PutFile employs
> getUserPrincipalLookupService() when it seemingly tries to validate
> the user and group.
>
> How can I get this to resolve through the LDAP for the PutFile?
>
> Thanks for any insights.  -Jim
>

>>>
>>
>

Re: NiFi 1.6.0 fails to start

[Mike Thomsen]

We added a patch for 1.7.0-SNAPSHOT that should handle that issue going
forward. You're welcome to download the code and try a build (in your dev
environment obviously) to see if that solves your problem.

On Wed, May 30, 2018 at 7:43 AM Bryan Bende  wrote:

> Hello,
>
> There is some work being done to support Java 9/10, but at this time NiFi
> currently requires Java 8.
>
> There is an issue when using Java 9/10 where some javax.xml classes are
> now part of the JDK, but before NiFi had to bundle them, so it creates
> conflicting classes.
>
> Thanks,
>
> Bryan
>
> On May 30, 2018, at 7:20 AM, Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.s...@nih.gov> wrote:
>
> Hello,
>
> We have recently tried to upgrade the development instance of NiFi running
> on Windows 2012 from 1.1.1 to 1.6.0, following the process described here
> .
> Both versions are very much vanilla in terms of configuration (no
> authentications), but v.1.6.0 fails to start – the content of the
> nifi-app.log file follows:
>
> 2018-05-25 14:57:14,032 INFO [main] org.eclipse.jetty.util.log Logging
> initialized @17556ms to org.eclipse.jetty.util.log.Slf4jLog
> 2018-05-25 14:57:14,404 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.util.ServiceConfigurationError:
> org.apache.nifi.authorization.UserGroupProvider:
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public
> no-arg constructor
> java.util.ServiceConfigurationError:
> org.apache.nifi.authorization.UserGroupProvider:
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public
> no-arg constructor
>at java.base/java.util.ServiceLoader.fail(Unknown Source)
>at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
>at java.base/java.util.ServiceLoader.access$1000(Unknown Source)
>at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
> Source)
>at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
> Source)
>at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
>at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
>at
> org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148)
>at
> org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123)
>at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771)
>at org.apache.nifi.NiFi.(NiFi.java:157)
>at org.apache.nifi.NiFi.(NiFi.java:71)
>at org.apache.nifi.NiFi.main(NiFi.java:292)
> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
>at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
>at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown
> Source)
>at java.base/java.lang.Class.getConstructor0(Unknown Source)
>at java.base/java.lang.Class.getConstructor(Unknown Source)
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>at java.base/java.security.AccessController.doPrivileged(Native
> Method)
>... 12 common frames omitted
> Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
>at java.base/java.net.URLClassLoader.findClass(Unknown Source)
>at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
>at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
>... 19 common frames omitted
>
> As for the JVM on that server, here is its version:
>
> >java -version
> java version "10.0.1" 2018-04-17
> Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
> Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)
>
> I have searched this mailing list archive and Google for a clue, but was
> unsuccessful so far.
>
> Thanks in advance,
>
> Alexander
>
>
>

Re: Unable to read the Parquet file written by NiFi through Spark when Logical Data Type is set to true.

[Mike Thomsen]

What's the error from Spark? Logical data types are just a variant on
existing data types in Avro 1.8.

On Wed, May 30, 2018 at 7:54 AM Mohit 
wrote:

> Hi all,
>
>
>
> I’m fetching the data from RDBMS and writing it to parquet using
> PutParquet processor. I’m not able to read the data from Spark when Logical
> Data Type is true. I’m able to read it from Hive.
>
> Do I have to set some specific properties in the PutParquet processor to
> make it readable from spark as well?
>
>
>
> Regards,
>
> Mohit
>

Unable to read the Parquet file written by NiFi through Spark when Logical Data Type is set to true.

[Mohit]

Hi all,

 

I'm fetching the data from RDBMS and writing it to parquet using PutParquet
processor. I'm not able to read the data from Spark when Logical Data Type
is true. I'm able to read it from Hive.

Do I have to set some specific properties in the PutParquet processor to
make it readable from spark as well?

 

Regards,

Mohit

Re: User, Group in LDAP appear to be unknown to PutFile

[Pierre Villard]

It depends how your OS is configured, you could leverage tools like SSSD to
resolve users against your LDAP but that's something to be configured at OS
level.
Until you can't resolve the user with OS commands, I don't think NiFi will
be able to set the expected owner on the file.

2018-05-30 11:54 GMT+02:00 James McMahon :

> Hello Pierre, and thank you. The user in this case - nifi - is not in the
> local /etc/passwd and is in the ldap. I presume this will force the id
>  to resolve using the ldap, if it does resolve?  At the OS the id
> command returns the uid, the gid, and the groups to which user nifi has
> membership within the ldap.
>
> On Wed, May 30, 2018 at 4:37 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> Hi Jim,
>>
>> LDAP for authentication and authorizations in NiFi has nothing to do with
>> the processors.
>> How processors are running/working is completely independent to the
>> authN/authZ model you configure for NiFi.
>>
>> Regarding your error, I'd say that you get this error because user/group
>> you're setting in the processor configuration cannot be resolved at OS
>> level (even though they exist in the LDAP, but again, that's totally
>> unrelated). Something you can quickly check: can you resolve the
>> username/group on the host where you're using PutFile processor? What do
>> you get if you execute the following command: id ?
>>
>> Pierre
>>
>> 2018-05-30 1:14 GMT+02:00 Joe Witt :
>>
>>> jim
>>>
>>> please only post to one list.
>>>
>>> users is good for this.
>>>
>>> thanks
>>> joe
>>>
>>> On Tue, May 29, 2018, 3:54 PM James McMahon 
>>> wrote:
>>>
 Good evening. I have recently migrated my nifi service host server from
 local resolution of users and groups to use an LDAP server. I configured
 login-identity-providers.xml and 
 nifi.security.user.login.identity.provider.
 I verified my configuration is known to NiFi by first restarting my nifi
 service and then attempting a login to the URL by a user without a cert,
 forcing it to resolve using LDAP. This appeared to work.

 I then attempted to set my file owner and file group in a PutFile to a
 user and a group that are each in the LDAP. The PutFile throws a Warning
 for both owner and group: 
 java.nio.file.attribute.UserPrincipalNotFoundException.
 The file is still output by the processor. It appears to default the user
 and owner to nifi.

 A cursory review of the PutFile source shows that PutFile employs
 getUserPrincipalLookupService() when it seemingly tries to validate
 the user and group.

 How can I get this to resolve through the LDAP for the PutFile?

 Thanks for any insights.  -Jim

>>>
>>
>

Re: NiFi 1.6.0 fails to start

[Bryan Bende]

Hello,

There is some work being done to support Java 9/10, but at this time NiFi 
currently requires Java 8.

There is an issue when using Java 9/10 where some javax.xml classes are now 
part of the JDK, but before NiFi had to bundle them, so it creates conflicting 
classes.

Thanks,

Bryan

> On May 30, 2018, at 7:20 AM, Saip, Alexander (NIH/CC/BTRIS) [C] 
>  wrote:
> 
> Hello,
> 
> We have recently tried to upgrade the development instance of NiFi running on 
> Windows 2012 from 1.1.1 to 1.6.0, following the process described here 
> . 
> Both versions are very much vanilla in terms of configuration (no 
> authentications), but v.1.6.0 fails to start – the content of the 
> nifi-app.log file follows:
> 
> 2018-05-25 14:57:14,032 INFO [main] org.eclipse.jetty.util.log Logging 
> initialized @17556ms to org.eclipse.jetty.util.log.Slf4jLog
> 2018-05-25 14:57:14,404 ERROR [main] org.apache.nifi.NiFi Failure to launch 
> NiFi due to java.util.ServiceConfigurationError: 
> org.apache.nifi.authorization.UserGroupProvider: 
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public 
> no-arg constructor
> java.util.ServiceConfigurationError: 
> org.apache.nifi.authorization.UserGroupProvider: 
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public 
> no-arg constructor
>at java.base/java.util.ServiceLoader.fail(Unknown Source)
>at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
>at java.base/java.util.ServiceLoader.access$1000(Unknown Source)
>at 
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
>  Source)
>at 
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown 
> Source)
>at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
>at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
>at 
> org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148)
>at 
> org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123)
>at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771)
>at org.apache.nifi.NiFi.(NiFi.java:157)
>at org.apache.nifi.NiFi.(NiFi.java:71)
>at org.apache.nifi.NiFi.main(NiFi.java:292)
> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
>at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
>at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown 
> Source)
>at java.base/java.lang.Class.getConstructor0(Unknown Source)
>at java.base/java.lang.Class.getConstructor(Unknown Source)
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>at java.base/java.security.AccessController.doPrivileged(Native Method)
>... 12 common frames omitted
> Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
>at java.base/java.net.URLClassLoader.findClass(Unknown Source)
>at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
>at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
>... 19 common frames omitted
> 
> As for the JVM on that server, here is its version:
> 
> >java -version
> java version "10.0.1" 2018-04-17
> Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
> Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)
> 
> I have searched this mailing list archive and Google for a clue, but was 
> unsuccessful so far.
> 
> Thanks in advance,
> 
> Alexander



signature.asc
Description: Message signed with OpenPGP

Re: NiFi 1.6.0 fails to start

[Martijn Dekkers]

Try java 8, and set JAVA_HOME etc.

On 30 May 2018 at 13:20, Saip, Alexander (NIH/CC/BTRIS) [C] <
alexander.s...@nih.gov> wrote:

> Hello,
>
>
>
> We have recently tried to upgrade the development instance of NiFi running
> on Windows 2012 from 1.1.1 to 1.6.0, following the process described here
> .
> Both versions are very much vanilla in terms of configuration (no
> authentications), but v.1.6.0 fails to start – the content of the
> nifi-app.log file follows:
>
>
>
> 2018-05-25 14:57:14,032 INFO [main] org.eclipse.jetty.util.log Logging
> initialized @17556ms to org.eclipse.jetty.util.log.Slf4jLog
>
> 2018-05-25 14:57:14,404 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.util.ServiceConfigurationError:
> org.apache.nifi.authorization.UserGroupProvider:
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public
> no-arg constructor
>
> java.util.ServiceConfigurationError: 
> org.apache.nifi.authorization.UserGroupProvider:
> org.apache.nifi.authorization.FileUserGroupProvider Unable to get public
> no-arg constructor
>
>at java.base/java.util.ServiceLoader.fail(Unknown Source)
>
>at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
>
>at java.base/java.util.ServiceLoader.access$1000(Unknown Source)
>
>at 
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
> Source)
>
>at 
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
> Source)
>
>at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
>
>at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
>
>at org.apache.nifi.nar.ExtensionManager.loadExtensions(
> ExtensionManager.java:148)
>
>at org.apache.nifi.nar.ExtensionManager.discoverExtensions(
> ExtensionManager.java:123)
>
>at org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:771)
>
>at org.apache.nifi.NiFi.(NiFi.java:157)
>
>at org.apache.nifi.NiFi.(NiFi.java:71)
>
>at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
>
>at java.base/java.lang.Class.getDeclaredConstructors0(Native
> Method)
>
>at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown
> Source)
>
>at java.base/java.lang.Class.getConstructor0(Unknown Source)
>
>at java.base/java.lang.Class.getConstructor(Unknown Source)
>
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>
>at java.base/java.util.ServiceLoader$1.run(Unknown Source)
>
>at java.base/java.security.AccessController.doPrivileged(Native
> Method)
>
>... 12 common frames omitted
>
> Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
>
>at java.base/java.net.URLClassLoader.findClass(Unknown Source)
>
>at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
>
>at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
>
>... 19 common frames omitted
>
>
>
> As for the JVM on that server, here is its version:
>
>
>
> >java -version
>
> java version "10.0.1" 2018-04-17
>
> Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
>
> Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)
>
>
>
> I have searched this mailing list archive and Google for a clue, but was
> unsuccessful so far.
>
>
>
> Thanks in advance,
>
>
>
> Alexander
>

Re: User, Group in LDAP appear to be unknown to PutFile

[James McMahon]

My apologies. It is sometimes difficult to decide if the root cause of a
challenge is related to code or related to a user level configuration
issue. In this case I had included the developer group because it seemed it
might relate to the PutFile after I had eliminated the possible
configuration omissions.

On Tue, May 29, 2018 at 7:14 PM, Joe Witt  wrote:

> jim
>
> please only post to one list.
>
> users is good for this.
>
> thanks
> joe
>
> On Tue, May 29, 2018, 3:54 PM James McMahon  wrote:
>
>> Good evening. I have recently migrated my nifi service host server from
>> local resolution of users and groups to use an LDAP server. I configured
>> login-identity-providers.xml and nifi.security.user.login.identity.provider.
>> I verified my configuration is known to NiFi by first restarting my nifi
>> service and then attempting a login to the URL by a user without a cert,
>> forcing it to resolve using LDAP. This appeared to work.
>>
>> I then attempted to set my file owner and file group in a PutFile to a
>> user and a group that are each in the LDAP. The PutFile throws a Warning
>> for both owner and group: java.nio.file.attribute.
>> UserPrincipalNotFoundException. The file is still output by the
>> processor. It appears to default the user and owner to nifi.
>>
>> A cursory review of the PutFile source shows that PutFile employs
>> getUserPrincipalLookupService() when it seemingly tries to validate the
>> user and group.
>>
>> How can I get this to resolve through the LDAP for the PutFile?
>>
>> Thanks for any insights.  -Jim
>>
>

Re: Only get file when a set exists.

[Pierre Villard]

I'll let Koji give more information about the Wait/Notify, he is clearly
the expert here.

I'm just jumping in regarding your "and when viewing the queue, the dialog
states that the queue is empty.". You're seeing this behavior because, even
though the UI shows some flow files in the queue, the flow files are
currently locked in the session of the running processor and you won't see
flow files currently processed in a session when listing a queue. If you
stop the processor, the session will be closed and you'll be able to list
the queue and see the flow files.

I recall discussions in the past to improve the UX for this. Not sure we
have a JIRA for it though...

Pierre

2018-05-30 8:26 GMT+02:00 Martijn Dekkers :

> Hi Koji,
>
> Thank you for responding. I had adjusted the run schedule to closely mimic
> our environment. We are expecting about 1 file per second or so.
> We are also seeing some random "orphans" sitting in a wait queue every now
> and again that don't trigger a debug message, and when viewing the queue,
> the dialog states that the queue is empty.
>
> We found the random "no signal found" issue to be significantly decreased
> when we increase the "concurrent tasks" to something large - currently set
> to 400 for all wait and notify processors.
>
> I do need to mention that our requirements had changed since you made the
> template, in that we are looking for a set of 8 files - 4 x "ext1" and 4 x
> "ext2" both with the same pattern: .ext1 or ext2
>
> We found that the best way to make this work was to add another
> wait/notify pair, each processor coming after the ones already there, with
> a simple counter against the groupID.
>
> I will export a template for you, many thanks for your help - I just need
> to spend some time sanitising the varies fields etc.
>
> Many thanks once again for your kind assistance.
>
> Martijn
>
> On 30 May 2018 at 08:14, Koji Kawamura  wrote:
>
>> Hi Martjin,
>>
>> In my template, I was using 'Run Schedule' as '5 secs' for the Wait
>> processors to avoid overusing CPU resource. However, if you expect
>> more throughput, it should be lowered.
>> Changed Run Schedule to 0 sec, and I passed 100 group of files (400
>> files because 4 files are 1 set in my example), they reached to the
>> expected goal of the flow without issue.
>>
>> If you can share your flow and example input file volume (hundreds of
>> files were fine in my flow), I may be able to provide more useful
>> comment.
>>
>> Thanks,
>> Koji
>>
>> On Wed, May 30, 2018 at 2:08 PM, Martijn Dekkers 
>> wrote:
>> > Hi Koji,
>> >
>> > I am seeing many issues to get this to run reliably. When running this
>> with
>> > a few flowfiles at a time, and stepping through by switching processors
>> on
>> > and off it works mostly fine, but running this at volume I receive many
>> > errors about "no release signal found"
>> >
>> > I have tried to fix this in a few different ways, but the issue keeps
>> coming
>> > back. This is also not consistent at all - different wait processors
>> will
>> > block different flowfiles at different times, without changing any
>> > configuration. Stop/Start the flow, and different queues will fill up.
>> Do
>> > you have any ideas what could be causing this behavior? I checked the
>> > DistributedMapCache Server/Client components, and they all appear to be
>> > working OK.
>> >
>> > Thanks,
>> >
>> > Martijn
>> >
>> > On 28 May 2018 at 05:11, Koji Kawamura  wrote:
>> >>
>> >> Hi Martin,
>> >>
>> >> Alternative approach is using Wait/Notify processors.
>> >> I have developed similar flow using those before, and it will work
>> >> with your case I believe.
>> >> A NiFi flow template is available here.
>> >> https://gist.github.com/ijokarumawak/06b3b071eeb4d10d8a27507981422edd
>> >>
>> >> Hope this helps,
>> >> Koji
>> >>
>> >>
>> >> On Sun, May 27, 2018 at 11:48 PM, Andrew Grande 
>> >> wrote:
>> >> > Martijn,
>> >> >
>> >> > Here's an idea you could explore. Have the ListFile processor work as
>> >> > usual
>> >> > and create a custom component (start with a scripting one to
>> prototype)
>> >> > grouping the filenames as needed. I don't know of the number of
>> files in
>> >> > a
>> >> > set is different every time, so trying to be more robust.
>> >> >
>> >> > Once you group and count the set, you can transfer the names to the
>> >> > success
>> >> > relationship. Ignore otherwise and wait until the set is full.
>> >> >
>> >> > Andrew
>> >> >
>> >> >
>> >> > On Sun, May 27, 2018, 7:29 AM Martijn Dekkers <
>> mart...@dekkers.org.uk>
>> >> > wrote:
>> >> >>
>> >> >> Hello all,
>> >> >>
>> >> >> I am trying to work out an issue with little success.
>> >> >>
>> >> >> I need to ingest files generated by some application. I can only
>> ingest
>> >> >> these files when a specific set exists. For example:
>> >> >>
>> >> >> file_123_456_ab.ex1
>> >> >> file_123_456_cd.ex1
>> >> >> file_123_456_ef.ex1
>> >> >> file_123_456_gh.ex1
>> >> >> file_123_456.ex2
>> >> >>
>> >> >> Only when a set like 

Re: Hive connection Pool error

[Pierre Villard]

Hi,

Could you share additional details about the processor/CS configuration as
well?

Thanks

2018-05-30 7:03 GMT+02:00 Koji Kawamura :

> Hello,
>
> Although I encountered various Kerberos related error, I haven't
> encountered that one.
> I tried to reproduce the same error by changing Kerberos related
> configuration, but to no avail.
> I recommend enabling Kerberos debug option for further debugging.
>
> You can add the option at nifi/conf/bootstrap.conf:
> java.arg.19=-Dsun.security.krb5.debug=true
>
> Then debug logs are written to nifi/logs/nifi-bootstap.log
>
> Thanks,
> Koji
>
> On Tue, May 29, 2018 at 10:31 PM, Vishal Dutt 
> wrote:
> > Hi ,
> >
> >
> >
> > We  are getting below error on randomly for few minutes and then goes
> away,
> > its coming in PUThiveql
> >
> >
> >
> >
> >
> > 2018-05-29 01:01:07,279 INFO [Timer-Driven Process Thread-95]
> > org.apache.hive.jdbc.HiveConnection Will try to open client transport
> with
> > JDBC Uri:
> > jdbc:hive2://ctcl-hdpmaster2.msoit.com:1/default;
> principal=hive/_h...@msoit.com
> >
> > 2018-05-29 01:01:07,281 ERROR [Timer-Driven Process Thread-95]
> > o.apache.thrift.transport.TSaslTransport SASL negotiation failure
> >
> > javax.security.sasl.SaslException: GSS initiate failed
> >
> > at
> > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(
> GssKrb5Client.java:211)
> >
> > at
> > org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(
> TSaslClientTransport.java:94)
> >
> > at
> > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> >
> > at
> > org.apache.thrift.transport.TSaslClientTransport.open(
> TSaslClientTransport.java:37)
> >
> > at
> > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$
> 1.run(TUGIAssumingTransport.java:52)
> >
> > at
> > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$
> 1.run(TUGIAssumingTransport.java:49)
> >
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> >
> > at
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1656)
> >
> > at
> > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.
> open(TUGIAssumingTransport.java:49)
> >
> > at
> > org.apache.hive.jdbc.HiveConnection.openTransport(
> HiveConnection.java:204)
> >
> > at
> > org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:176)
> >
> > at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
> >
> > at
> > org.apache.commons.dbcp.DriverConnectionFactory.createConnection(
> DriverConnectionFactory.java:38)
> >
> > at
> > org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(
> PoolableConnectionFactory.java:582)
> >
> > at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(
> GenericObjectPool.java:1148)
> >
> > at
> > org.apache.commons.dbcp.PoolingDataSource.getConnection(
> PoolingDataSource.java:106)
> >
> > at
> > org.apache.commons.dbcp.BasicDataSource.getConnection(
> BasicDataSource.java:1044)
> >
> > at
> > org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(
> HiveConnectionPool.java:355)
> >
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> >
> > at
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1656)
> >
> > at
> > org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(
> HiveConnectionPool.java:355)
> >
> > at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source)
> >
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> >
> > at java.lang.reflect.Method.invoke(Method.java:498)
> >
> > at
> > org.apache.nifi.controller.service.StandardControllerServiceInvoc
> ationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
> >
> > at com.sun.proxy.$Proxy97.getConnection(Unknown Source)
> >
> > at
> > org.apache.nifi.processors.hive.PutHiveQL.lambda$new$1(
> PutHiveQL.java:191)
> >
> > at org.apache.nifi.processor.util.pattern.Put.onTrigger(
> Put.java:96)
> >
> > at
> > org.apache.nifi.processors.hive.PutHiveQL.lambda$
> onTrigger$6(PutHiveQL.java:274)
> >
> > at
> > org.apache.nifi.processor.util.pattern.PartialFunctions.
> onTrigger(PartialFunctions.java:114)
> >
> > at
> > org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(
> RollbackOnFailure.java:184)
> >
> > at
> > org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:274)
> >
> > at
> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1147)
> >
> > at
> > org.apache.nifi.controller.tasks.ConnectableTask.invoke(
> 

Re: Only get file when a set exists.

[Martijn Dekkers]

Hi Koji,

Thank you for responding. I had adjusted the run schedule to closely mimic
our environment. We are expecting about 1 file per second or so.
We are also seeing some random "orphans" sitting in a wait queue every now
and again that don't trigger a debug message, and when viewing the queue,
the dialog states that the queue is empty.

We found the random "no signal found" issue to be significantly decreased
when we increase the "concurrent tasks" to something large - currently set
to 400 for all wait and notify processors.

I do need to mention that our requirements had changed since you made the
template, in that we are looking for a set of 8 files - 4 x "ext1" and 4 x
"ext2" both with the same pattern: .ext1 or ext2

We found that the best way to make this work was to add another wait/notify
pair, each processor coming after the ones already there, with a simple
counter against the groupID.

I will export a template for you, many thanks for your help - I just need
to spend some time sanitising the varies fields etc.

Many thanks once again for your kind assistance.

Martijn

On 30 May 2018 at 08:14, Koji Kawamura  wrote:

> Hi Martjin,
>
> In my template, I was using 'Run Schedule' as '5 secs' for the Wait
> processors to avoid overusing CPU resource. However, if you expect
> more throughput, it should be lowered.
> Changed Run Schedule to 0 sec, and I passed 100 group of files (400
> files because 4 files are 1 set in my example), they reached to the
> expected goal of the flow without issue.
>
> If you can share your flow and example input file volume (hundreds of
> files were fine in my flow), I may be able to provide more useful
> comment.
>
> Thanks,
> Koji
>
> On Wed, May 30, 2018 at 2:08 PM, Martijn Dekkers 
> wrote:
> > Hi Koji,
> >
> > I am seeing many issues to get this to run reliably. When running this
> with
> > a few flowfiles at a time, and stepping through by switching processors
> on
> > and off it works mostly fine, but running this at volume I receive many
> > errors about "no release signal found"
> >
> > I have tried to fix this in a few different ways, but the issue keeps
> coming
> > back. This is also not consistent at all - different wait processors will
> > block different flowfiles at different times, without changing any
> > configuration. Stop/Start the flow, and different queues will fill up. Do
> > you have any ideas what could be causing this behavior? I checked the
> > DistributedMapCache Server/Client components, and they all appear to be
> > working OK.
> >
> > Thanks,
> >
> > Martijn
> >
> > On 28 May 2018 at 05:11, Koji Kawamura  wrote:
> >>
> >> Hi Martin,
> >>
> >> Alternative approach is using Wait/Notify processors.
> >> I have developed similar flow using those before, and it will work
> >> with your case I believe.
> >> A NiFi flow template is available here.
> >> https://gist.github.com/ijokarumawak/06b3b071eeb4d10d8a27507981422edd
> >>
> >> Hope this helps,
> >> Koji
> >>
> >>
> >> On Sun, May 27, 2018 at 11:48 PM, Andrew Grande 
> >> wrote:
> >> > Martijn,
> >> >
> >> > Here's an idea you could explore. Have the ListFile processor work as
> >> > usual
> >> > and create a custom component (start with a scripting one to
> prototype)
> >> > grouping the filenames as needed. I don't know of the number of files
> in
> >> > a
> >> > set is different every time, so trying to be more robust.
> >> >
> >> > Once you group and count the set, you can transfer the names to the
> >> > success
> >> > relationship. Ignore otherwise and wait until the set is full.
> >> >
> >> > Andrew
> >> >
> >> >
> >> > On Sun, May 27, 2018, 7:29 AM Martijn Dekkers  >
> >> > wrote:
> >> >>
> >> >> Hello all,
> >> >>
> >> >> I am trying to work out an issue with little success.
> >> >>
> >> >> I need to ingest files generated by some application. I can only
> ingest
> >> >> these files when a specific set exists. For example:
> >> >>
> >> >> file_123_456_ab.ex1
> >> >> file_123_456_cd.ex1
> >> >> file_123_456_ef.ex1
> >> >> file_123_456_gh.ex1
> >> >> file_123_456.ex2
> >> >>
> >> >> Only when a set like that exists should I pick them up into the Flow.
> >> >> The
> >> >> parts I am looking for to "group" would "ab.ex1", "cd.ex1", "ef.ex1",
> >> >> "gh.ex1", ".ex2".
> >> >>
> >> >> I tried to do this with some expression, but couldn't work it out.
> >> >>
> >> >> What would be the best way to achieve this?
> >> >>
> >> >> Many thanks!
> >
> >
>