Re: nifi-toolkit docker images

[Peter Wilcsinszky]

Hi,

also the Toolkit is available in the NiFi image as well, preconfigured to
be used against the locally running instance. Example:

docker run -d apache/nifi:1.8.0
389997393ef93496b875277a3667bca483c8ccee0f99513c61d7d447042df09a

docker exec
389997393ef93496b875277a3667bca483c8ccee0f99513c61d7d447042df09a
/opt/nifi/nifi-toolkit-current/bin/cli.sh nifi current-user
anonymous

Peter

On Tue, Oct 30, 2018 at 3:32 PM Aldrin Piri  wrote:

> Hi Charlie,
>
> One will be pushed, please check in about 30 minutes.  While NiFi is
> driven via automated builds, some of our Docker Hub repos require manual
> steps that can only be performed by certain folks on the PMC.
>
> Thanks for pointing out this was missing!
> --aldrin
>
> On Tue, Oct 30, 2018 at 9:58 AM Charlie Meyer <
> charlie.me...@civitaslearning.com> wrote:
>
>> Hi,
>>
>> Are docker images for nifi-toolkit regularly published still? The latest
>> tag I see at https://hub.docker.com/r/apache/nifi-toolkit/tags/
>> references 1.7.0. Will a 1.8.0 image be pushed or is the recommended that
>> users roll their own?
>>
>> Thanks!
>>
>

Re: nifi-toolkit docker images

[Aldrin Piri]

Hi Charlie,

One will be pushed, please check in about 30 minutes.  While NiFi is driven
via automated builds, some of our Docker Hub repos require manual steps
that can only be performed by certain folks on the PMC.

Thanks for pointing out this was missing!
--aldrin

On Tue, Oct 30, 2018 at 9:58 AM Charlie Meyer <
charlie.me...@civitaslearning.com> wrote:

> Hi,
>
> Are docker images for nifi-toolkit regularly published still? The latest
> tag I see at https://hub.docker.com/r/apache/nifi-toolkit/tags/
> references 1.7.0. Will a 1.8.0 image be pushed or is the recommended that
> users roll their own?
>
> Thanks!
>

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[ara m.]

I started to test my container locally and saw the speed was drastically
faster. 
So I limited my docker cpu on my laptop to half a core instead of the 4 or 8
my mac has.. and the script went from running in 13 seconds to 1+ minute. 

On the cluster i changed half core to be using 1 whole core, and saw the
same script that took 1:15 before to execute to drop to 36seconds. So
perhaps thats the biggest limiting factor..



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Andrew Grande]

Guessing - if it's a new container on the first invocation, maybe the JVM
is generating the binary class cache? This operation is performed only once
ever, but with a clean environment every time I can see it being invoked
again and again.

Other than that, you'd need to connect a profiler and see where it spends
most of the time.

Andrew

On Mon, Oct 29, 2018, 1:22 PM ara m.  wrote:

> Hi Andrew - yes its a container environment. Everything I work with is in
> effect 'dockerized'. I created the container environment with 128min memory
> and 256 max memory and 500m cpu which is half a core (2.40GHz per core). I
> think that should be fine for the CLI but you tell me otherwise.
>
> Hi Andy - I was hoping the TLS handshake was the reason but that parameter
> for the JVM did not solve it. The speed remained the same, slow. I found
> out
> the jvm i was using already had /dev/urandom it was using.
>
> Calling cli.sh   is still 10+ seconds.
>
> Here's the odd thing.. Using the CLI, the very first command is again 10+
> seconds, but subsequent commands are 1 second if that.
> The only one that takes 2-3 seconds is the pg-import one but thats because
> its a hefty flow (100+ processors there).
>
> I also tried setting '-XX:+UseAES -XX:+UseAESIntrinsics' as well. ("TLS/SSL
> Performance Improvements for Java"
>
> https://docs.hazelcast.org/docs/latest-development/manual/html/Security/TLS-SSL.html
> ).
>
> I'm puzzled why using the CLI, the first command is fast, and rest are
> fast,
> while calling the cli from outside is always same 10+ seconds slowness...
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[ara m.]

Hi Andrew - yes its a container environment. Everything I work with is in
effect 'dockerized'. I created the container environment with 128min memory
and 256 max memory and 500m cpu which is half a core (2.40GHz per core). I
think that should be fine for the CLI but you tell me otherwise.

Hi Andy - I was hoping the TLS handshake was the reason but that parameter
for the JVM did not solve it. The speed remained the same, slow. I found out
the jvm i was using already had /dev/urandom it was using.

Calling cli.sh   is still 10+ seconds. 

Here's the odd thing.. Using the CLI, the very first command is again 10+
seconds, but subsequent commands are 1 second if that. 
The only one that takes 2-3 seconds is the pg-import one but thats because
its a hefty flow (100+ processors there).

I also tried setting '-XX:+UseAES -XX:+UseAESIntrinsics' as well. ("TLS/SSL
Performance Improvements for Java"
https://docs.hazelcast.org/docs/latest-development/manual/html/Security/TLS-SSL.html).

I'm puzzled why using the CLI, the first command is fast, and rest are fast,
while calling the cli from outside is always same 10+ seconds slowness...



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Andy LoPresto]

If you’re running in a containerized environment and this is the first task run 
that requires cryptographic operations (i.e. performing TLS handshake 
negotiations), there is a chance that the issue is that the OS doesn’t feel it 
has sufficient entropy to provide cryptographically-secure random values, and 
it is blocking until it receives enough input. 

As an experiment, please modify the cli.sh script and add this JVM argument 
(similar to Bryan’s suggestion below about adding TLS debugging):

-Djava.security.egd=file:/dev/urandom

This sets the entropy provider for Java’s SecureRandom class to /dev/urandom, 
which is non-blocking. If you are concerned because you have read that 
/dev/urandom is “not as secure” as /dev/random, please read [1][2] and rest 
assured that it provides a strong security stance and is the preferred source 
of cryptographically-secure random numbers on *nix systems. 

If adding this parameter solves the delay at startup, please let us know and we 
can open a Jira and add this into the command by default to handle this. There 
is no ill-effect for a normal deployment, and it will drastically improve 
performance on containerized/virtualized environments. 

As for REST calls or scripting for batch commands, I do think it would be 
valuable but as Bryan indicated, there probably needs to be a larger discussion 
around the requirements here. Please feel free to list out your expectations 
and any suggestions for implementation. 

Hope this helps; sorry, I am traveling and didn’t see this thread the other 
day. 

[1] https://www.2uo.de/myths-about-urandom/
[2] https://security.stackexchange.com/a/3939/16485


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 29, 2018, at 1:39 AM, Andrew Grande  wrote:
> 
> Are you running in some container environment? It should never take 15 secs 
> and there's no caching performed by cli. I would review the container 
> environment and see why it's taking forever to start.
> 
> Andrew
> 
> On Sun, Oct 28, 2018, 10:11 AM ara m.  > wrote:
> The Keystore user had "view processor" and "modify processor" permissions.
> Still did not work. Its odd since NiFi UI can make imports, so the CLI
> should be able to.
> 
> Speed wise.. JVM takes a few seconds to start. The command to list buckets
> takes ~13 seconds. Its kind of rough. I noticed however from inside the cli,
> the first command is slow, but then things start getting cached and running
> faster. You dont get the same benefits when executing many commands by
> calling ./cli.sh  .
> 
> I think I'm going to have a problem in deployment with setting variables..
> since every operation takes 12-15 seconds.
> 
> You can imagine a standard deployment goes like this
> 
> bucket_id = registry list-buckets with this $bucket_name
> flow_id = registry list-flows get flow_id with this bucket_id and this
> $flow_name
> pg_id = nifi pg-import with this bucket_id, flow_id, and flow_version
> 
> now lets say you have 20-30 variables to set
> nifi pg-set-var -pgid $PGID -var someVar -val someVal
> 
> nifi pg-start
> 
> That is a typical deployment. But if you have 20-30 variables to set and
> each one takes ~12-13 seconds to set, youre looking at ~5 minutes just
> setting variables.
> 
> Thoughts? Why isnt there an equivalent Registry REST call for it? Perhaps
> that will be quicker.. although i like using the CLI only. 
> 
> I noticed there is a rest api for nifi itself, but it seems to want to
> replace the whole contents of variable registry rather then just set 1
> variable which is what the CLI provides
> (/process-groups/{id}/variable-registry). 
> 
> 
> 
> 
> 
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/ 
> 

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Andrew Grande]

Are you running in some container environment? It should never take 15 secs
and there's no caching performed by cli. I would review the container
environment and see why it's taking forever to start.

Andrew

On Sun, Oct 28, 2018, 10:11 AM ara m.  wrote:

> The Keystore user had "view processor" and "modify processor" permissions.
> Still did not work. Its odd since NiFi UI can make imports, so the CLI
> should be able to.
>
> Speed wise.. JVM takes a few seconds to start. The command to list buckets
> takes ~13 seconds. Its kind of rough. I noticed however from inside the
> cli,
> the first command is slow, but then things start getting cached and running
> faster. You dont get the same benefits when executing many commands by
> calling ./cli.sh  .
>
> I think I'm going to have a problem in deployment with setting variables..
> since every operation takes 12-15 seconds.
>
> You can imagine a standard deployment goes like this
>
> bucket_id = registry list-buckets with this $bucket_name
> flow_id = registry list-flows get flow_id with this bucket_id and this
> $flow_name
> pg_id = nifi pg-import with this bucket_id, flow_id, and flow_version
>
> now lets say you have 20-30 variables to set
> nifi pg-set-var -pgid $PGID -var someVar -val someVal
>
> nifi pg-start
>
> That is a typical deployment. But if you have 20-30 variables to set and
> each one takes ~12-13 seconds to set, youre looking at ~5 minutes just
> setting variables.
>
> Thoughts? Why isnt there an equivalent Registry REST call for it? Perhaps
> that will be quicker.. although i like using the CLI only.
>
> I noticed there is a rest api for nifi itself, but it seems to want to
> replace the whole contents of variable registry rather then just set 1
> variable which is what the CLI provides
> (/process-groups/{id}/variable-registry).
>
>
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[ara m.]

The Keystore user had "view processor" and "modify processor" permissions.
Still did not work. Its odd since NiFi UI can make imports, so the CLI
should be able to.

Speed wise.. JVM takes a few seconds to start. The command to list buckets
takes ~13 seconds. Its kind of rough. I noticed however from inside the cli,
the first command is slow, but then things start getting cached and running
faster. You dont get the same benefits when executing many commands by
calling ./cli.sh  .

I think I'm going to have a problem in deployment with setting variables..
since every operation takes 12-15 seconds.

You can imagine a standard deployment goes like this

bucket_id = registry list-buckets with this $bucket_name
flow_id = registry list-flows get flow_id with this bucket_id and this
$flow_name
pg_id = nifi pg-import with this bucket_id, flow_id, and flow_version

now lets say you have 20-30 variables to set
nifi pg-set-var -pgid $PGID -var someVar -val someVal

nifi pg-start

That is a typical deployment. But if you have 20-30 variables to set and
each one takes ~12-13 seconds to set, youre looking at ~5 minutes just
setting variables.

Thoughts? Why isnt there an equivalent Registry REST call for it? Perhaps
that will be quicker.. although i like using the CLI only. 

I noticed there is a rest api for nifi itself, but it seems to want to
replace the whole contents of variable registry rather then just set 1
variable which is what the CLI provides
(/process-groups/{id}/variable-registry). 





--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Bryan Bende]

Glad you were able to get it working.

Regarding the super user comment... I don't think it has to be a super
user, but it has to be a user that has permissions to perform the
action. For your example of "nifi pg-import" it would have to be a
user that has write permission to the parent process group where you
are importing. If you don't use a proxied entity and just the
keystore, the NiFi user by default does not have write to any process
groups, the servers only get /proxy and /controller, so you could go
in and grant the server nodes permissions to other things, but by
default it wouldn't work.

Regarding the slow execution... Currently you can only run 1 command
or use interactive mode, but it could be a nice improvement to execute
a series of commands using the one JVM instance. I would be curious to
know which part is slow though, is it really that slow spinning up the
JVM, or is the actual call to retrieve the buckets slow?
On Wed, Oct 24, 2018 at 9:24 PM ara m.  wrote:
>
> D'oh. Thanks.. I forgot I'm spinning up jvm in the CLI. You'd think after a
> week of troubleshooting NiFi<->Reg ssl I'd have thought of this.
>
> Let's just say I had some jvm flags to set of my own besides the debugging
> one...
> -Dcom.ibm.jsse2.overrideDefaultTLS=true
>
> Then I hit a few stumbles and worked through them. I am not sure why I
> needed a super-user as a proxiedUser to get anything done. I should have
> been able to use the user in the keystore, as it has permissions in NiFI,
> but for some reason I was getting errors with "> nifi pg-import" code.
>
> The next thing to fix it seems like spinning up the JVM to run the tasks is
> quite slow. Takes almost 20+ seconds to run one command like list-buckets.
> Maybe I've to increase the memory for my container.
>
> There's no way to pass the cli several lines of commands to run? I know
> there is benefits by using back-references for bucket and flow, which I kind
> of lose when executing CLI commands from the outside.
>
> Still thank you very much for getting my head straight!
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Bryan Bende]

Also one more note, if you want to add that same debugging on the NiFi
or NiFi Registry side then it would go in the bootstrap.conf as a
java.arg like:

java.arg.17=-Djavax.net.debug=ssl:handshake

Then restart the application and the SSL handshake debug should go to
nifi-boostrap.log, or nifi-registry-bootstrap.log
On Wed, Oct 24, 2018 at 5:24 PM Bryan Bende  wrote:
>
> Can you try editing cli.sh and near the end where it actually executes
> the CLIMain, edit the line to add the -D for SSL handshake debugging
> so it looks like this:
>
> exec "${JAVA}" -cp "${CLASSPATH}" ${JAVA_OPTS:--Xms128m -Xmx256m}
> -Djavax.net.debug=ssl:handshake  org.apache.nifi.toolkit.cli.CLIMain
> "$@"
>
> Then run your command and you should see all the debugging info print
> out to the console. Curious to see what that shows.
> On Wed, Oct 24, 2018 at 5:03 PM ara m.  wrote:
> >
> > I tried the ssldump tool, didnt have much luck with it no matter which
> > parameters i passed in. Not sure if its because my cluster everything is
> > dockerized and hides the communication so it cannot parse the communication
> > then.
> >
> > Still stuck without being able to use CLI from inside NiFi container using
> > NiFi keystore+truststore..
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Bryan Bende]

Can you try editing cli.sh and near the end where it actually executes
the CLIMain, edit the line to add the -D for SSL handshake debugging
so it looks like this:

exec "${JAVA}" -cp "${CLASSPATH}" ${JAVA_OPTS:--Xms128m -Xmx256m}
-Djavax.net.debug=ssl:handshake  org.apache.nifi.toolkit.cli.CLIMain
"$@"

Then run your command and you should see all the debugging info print
out to the console. Curious to see what that shows.
On Wed, Oct 24, 2018 at 5:03 PM ara m.  wrote:
>
> I tried the ssldump tool, didnt have much luck with it no matter which
> parameters i passed in. Not sure if its because my cluster everything is
> dockerized and hides the communication so it cannot parse the communication
> then.
>
> Still stuck without being able to use CLI from inside NiFi container using
> NiFi keystore+truststore..
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[ara m.]

I tried the ssldump tool, didnt have much luck with it no matter which
parameters i passed in. Not sure if its because my cluster everything is
dockerized and hides the communication so it cannot parse the communication
then.

Still stuck without being able to use CLI from inside NiFi container using
NiFi keystore+truststore..



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[ara m.]

Everything is 1.7.1.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Peter Wilcsinszky]

Hi Ara,

I can see that the Toolkit version is 1.7.1 but what versions of NiFi and
NiFi Registry are you using?

Can you perform an ssldump [1] on the Toolkit or the Registry side to see
the TLS handshake error (supposing that is the issue) more closely?

[1]
https://devcentral.f5.com/articles/troubleshooting-tls-problems-with-ssldump

On Wed, Oct 24, 2018 at 4:21 PM Bryan Bende  wrote:

> With the CLI using the keystore/truststore from NiFi you shouldn't get
> an SSL handshake error at all, regardless of whether it was a command
> against NiFi or NIFi Registry.
>
> After the SSL handshake the command could still fail based on whether
> the identity of the CLI has permissions to execute the command in the
> target system.
>
> For example, using NiFi's own identity to make a call against NiFi
> like "nifi pg-start ..." will probably fail with an unauthorized error
> unless you granted the NiFi user permission to modify components, by
> default the server users only have one or two permissions like /proxy
> and /controller.
>
> Alternatively you can also pass a proxiedEntity specifying a user that
> already has permission to modify components and then it should work.
>
> On Wed, Oct 24, 2018 at 10:04 AM ara m.  wrote:
> >
> > Thats right they are identical, and registry-dev.properties has 2 more
> > fields, one that is baseUrl https:// registry:port, and the other
> > proxiedEntity is left blank..
> >
> > /baseUrl=https://nifi-registry.xx.local:18443
> > proxiedEntity=/
> >
> > So using those NiFi properties the CLI can only talk to the Registry, is
> > that right?
> >
> > Meaning these commands we expect to succeed
> > /> registry list-buckets -p registry-dev.properties/
> >
> > but any command that calls to 'nifi' from the CLI, we expect to fail?
> > /> nifi pg-start -pgid .../
> >
> > From inside NiFi container I am able to ping that Registry address, and
> the
> > port is indeed listening in Registry.. The url I used for Registry is the
> > same URL that I specified in NiFi UI for Registry and I was able to get
> > buckets && version flow (of my processor group).
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Bryan Bende]

With the CLI using the keystore/truststore from NiFi you shouldn't get
an SSL handshake error at all, regardless of whether it was a command
against NiFi or NIFi Registry.

After the SSL handshake the command could still fail based on whether
the identity of the CLI has permissions to execute the command in the
target system.

For example, using NiFi's own identity to make a call against NiFi
like "nifi pg-start ..." will probably fail with an unauthorized error
unless you granted the NiFi user permission to modify components, by
default the server users only have one or two permissions like /proxy
and /controller.

Alternatively you can also pass a proxiedEntity specifying a user that
already has permission to modify components and then it should work.

On Wed, Oct 24, 2018 at 10:04 AM ara m.  wrote:
>
> Thats right they are identical, and registry-dev.properties has 2 more
> fields, one that is baseUrl https:// registry:port, and the other
> proxiedEntity is left blank..
>
> /baseUrl=https://nifi-registry.xx.local:18443
> proxiedEntity=/
>
> So using those NiFi properties the CLI can only talk to the Registry, is
> that right?
>
> Meaning these commands we expect to succeed
> /> registry list-buckets -p registry-dev.properties/
>
> but any command that calls to 'nifi' from the CLI, we expect to fail?
> /> nifi pg-start -pgid .../
>
> From inside NiFi container I am able to ping that Registry address, and the
> port is indeed listening in Registry.. The url I used for Registry is the
> same URL that I specified in NiFi UI for Registry and I was able to get
> buckets && version flow (of my processor group).
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[ara m.]

Thats right they are identical, and registry-dev.properties has 2 more
fields, one that is baseUrl https:// registry:port, and the other
proxiedEntity is left blank..

/baseUrl=https://nifi-registry.xx.local:18443
proxiedEntity=/

So using those NiFi properties the CLI can only talk to the Registry, is
that right?

Meaning these commands we expect to succeed
/> registry list-buckets -p registry-dev.properties/

but any command that calls to 'nifi' from the CLI, we expect to fail?
/> nifi pg-start -pgid .../

>From inside NiFi container I am able to ping that Registry address, and the
port is indeed listening in Registry.. The url I used for Registry is the
same URL that I specified in NiFi UI for Registry and I was able to get
buckets && version flow (of my processor group).



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Bryan Bende]

So you're saying the CLI properties for the keystore and truststore
match exactly what is in your nifi.properties for the nifi.security
properties?

Basically these should be identical...

nifi.properties

nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=

CLI properties

keystore=
keystoreType=
keystorePasswd=
keyPasswd=
truststore=
truststoreType=
truststorePasswd=

I don't know how that could not work since the keystore/truststore in
nifi.properties is what NiFi is using to securely talk to NiFi
Registry.

On Wed, Oct 24, 2018 at 9:06 AM ara m.  wrote:
>
> Since NiFi is already talking to the Registry, the CLI inside the NiFi
> container should be able to connect right?
>
> What else can I try?
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[ara m.]

Since NiFi is already talking to the Registry, the CLI inside the NiFi
container should be able to connect right? 

What else can I try?



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[ara m.]

I tried this >>> You could also just use the keystore from NiFi since NiFi is
already 
successfully talking to registry. 

I used the CLI from inside NiFi and provided the truststore  as well as the
keystore that NiFi uses, still same error message.

/ERROR: Error executing command 'list-buckets' : Received fatal alert:
handshake_failure/



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi Toolkit CLI issues with NiFi/Registry SSL handshake

[Bryan Bende]

In order to perform any action against a secure registry or NiFi you
will need to authenticate, just like if you went to one of the UI's in
your browser and you provided a client cert in your browser, or logged
in with LDAP/Kerberos/etc.

By specifying a truststore you are only verifying the server's
identity, in this case it allows CLI to verify that NiFi Registry
presented a good cert, but it is not providing any identity for the
CLI to present to NiFi or NiFi Registry.

You can authenticate by specifying a keystore and an optional proxied entity:
https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli#security-configuration

The keystore could be a certificate representing a user that you made
specifically for the CLI, and you would have add that use to NiFi and
NiFi registry and grant that user appropriate permissions.

You could also just use the keystore from NiFi since NiFi is already
successfully talking to registry.
On Tue, Oct 23, 2018 at 3:34 PM ara m.  wrote:
>
> Hi I've setup NiFi and NiFi registry to talk securely with 2-way SSL auth
> (tls), and so have 2 separate keystores and truststores. The truststores of
> each service include the cert from the other service. I am able to go to
> https:// NiFI url and https:// Registry url, and see both UI's, and I am
> able to create buckets, users, set proxy users in the Registry. From NiFi I
> am able to version a flow, and see that flow-name appear in NiFi registry
> under that bucket. So here's my question..
>
> Now I am having issues using the CLI to talk from inside the container to
> NiFi and the Registry. Both of my containers (I should mention I have
> containerized everything), have the NiFi Toolkit installed, and both are
> passing the truststore+truststore_type+truststore_pass to the NiFi / Reg,
> and are getting a 1 liner exception that I am trying to debug.
>
> Don't have much to go on, except I know its an SSL issue. Usually a line
> like this would indicate that I don't have the cert of the server that I am
> trying to communicate with, but since I am inside the container of NiFi Reg,
> I would think I do have the correct truststore.. so i am perplexed.
>
> Any directions of help?
> (I have tried adding javax.net.debug to logback in NiFi nifi-user.log but
> haven't seen any ssl exceptions come in, nor any java logging, so perhaps I
> have a mistake there.. but this is what my next steps will be - to try to
> produce more logging)
>
> I have tried 4 different ways to talk using the CLI (and from both
> containers - NiFi and Reg).
>
> Calling from outside providing vars:
> //opt/nifi-toolkit/nifi-toolkit-1.7.1/bin/cli.sh registry list-buckets -u
> $BASE_URL -ts $TRUSTSTORE -tst $TRUSTSTORE_TYPE
> /
> Doing the same call but from inside the cli:
> /> registry list-buckets -u $BASE_URL -ts $TRUSTSTORE -tst $TRUSTSTORE_TYPE
> /
> Filled out truststore (as well as keystore info) in a properties file and
> calling it from outside cli and from inside: registry-dev.properties.
>
> /./bin/cli.sh registry list-buckets -p registry-dev.properties
> /
> /> registry list-buckets -p registry-dev.properties
> /
> I always get the same error message.
>
> /ERROR: Error executing command 'list-buckets' : Received fatal alert:
> handshake_failure
> /
> Before you ask, i do use https:// and https port, and I do have the right
> truststore path and password, otherwise if i provide the wrong password the
> CLI complains, and also the truststore is already being used by the Registry
> container to make the NiFi <-> NiFi Registry connection work.
>
> Thoughts?
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: nifi toolkit

[Juan Sequeiros]

Andy thanks for pointing out configJsonIn, that will work just as well.
Thanks for all the hard work with the toolkit its very cool.

On Thu, Sep 20, 2018 at 9:32 PM Andy LoPresto  wrote:

> Juan,
>
> The client/server mode does not expose as many command-line options as
> standalone mode, but you can provide a config.json file which contains
> values to use and use --configJsonIn to specify the path. See [1] for more
> details.
>
> I am currently working on NIFI-5485 [2] and can try to improve the
> command-line flags for client/server mode as an add-on ticket [3].
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#client
> [2] https://issues.apache.org/jira/browse/NIFI-5485
> [3] https://issues.apache.org/jira/browse/NIFI-5620
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Sep 20, 2018, at 3:41 PM, Juan Sequeiros  wrote:
>
> Hello all,
>
> Using nifitoolkit bin/tls-toolkit.sh
>
> While using client / server mode ( CA issuer )
> Is there any way to pass it parameters like the ones using standalone mode?
>
> Want to be able to set the password through the command line
> Similar to how I can on standalone mode
> EX:
>
> Do this equivalent through toolkit with client option:
>
> Thanks!
>
> tls-toolkit.sh standalone -n 'SomeHost' -K 'KeyStorePassword' -P
> 'TrustPassword' -S 'KeyPassword'
>
>
>

-- 
Juan Carlos Sequeiros

Re: nifi toolkit

[Andy LoPresto]

Juan,

The client/server mode does not expose as many command-line options as 
standalone mode, but you can provide a config.json file which contains values 
to use and use --configJsonIn to specify the path. See [1] for more details.

I am currently working on NIFI-5485 [2] and can try to improve the command-line 
flags for client/server mode as an add-on ticket [3].

[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#client
[2] https://issues.apache.org/jira/browse/NIFI-5485
[3] https://issues.apache.org/jira/browse/NIFI-5620

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Sep 20, 2018, at 3:41 PM, Juan Sequeiros  wrote:
> 
> Hello all,
> 
> Using nifitoolkit bin/tls-toolkit.sh
> 
> While using client / server mode ( CA issuer )
> Is there any way to pass it parameters like the ones using standalone mode?
> 
> Want to be able to set the password through the command line
> Similar to how I can on standalone mode
> EX:
> 
> Do this equivalent through toolkit with client option:
> 
> Thanks!
> 
> tls-toolkit.sh standalone -n 'SomeHost' -K 'KeyStorePassword' -P 
> 'TrustPassword' -S 'KeyPassword'



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Nifi toolkit 1.7 additional question

[Juan Sequeiros]

All,

Disregard the previous email.
I was running the toolkit for my user cert in same directory that already
had previous server certs in there.
Once I removed them I was able to generate my "user" cert with -D

On Thu, Sep 20, 2018 at 7:29 PM Juan Sequeiros  wrote:

> Hello,
>
> When using nifi toolkit 1.7.1 to get user cert I run this:
>
> ./tls-toolkit.sh client -c CAhost -t AlongTokenString -p AportNumber -D
> "CN=Juan Carlos Sequeiros,OU=NIFI" -T pksc12
>
> And it fails with this return:
>
> 2018/09/20 23:23:57 INFO [main]
> org.apache.nifi.toolkit.tls.commandLine.BaseTlsToolkitCommandLine: Command
> line argument --keyStoreType=pkcs12 only applies to keystore, recommended
> truststore type of JKS unaffected.
> Service client error: null
>
> Any tips?
>
> Juan Carlos Sequeiros
>
>
>

-- 
Juan Carlos Sequeiros

Re: Nifi toolkit grape dependecies

[Pierre Villard]

Hey Dan,

I think it's worth filing a JIRA for it.

Pierre

2018-05-22 19:27 GMT+02:00 dan young :

> Hello,
>
> I'm trying to run the Nifi Expression testing groovy tool with 1.6.0 and
> am getting the following dep errorNifi 1.4 and 1.5 works fine.  I tried
> to also download via grape in the groovy cli the jackson-core and it
> doesn;t work for the 2.9.4 version  2.9.5 works fineany ideas?
>
> I can use the 1.5 w/o  issue
>
>
>
> λ ~/software/nifi-tools groovy testEL-1.6.groovy
> '${now():format("MMdd")}'
>
> org.codehaus.groovy.control.MultipleCompilationErrorsException: startup
> failed:
>
> General error during conversion: Error grabbing Grapes -- [download
> failed: com.fasterxml.jackson.core#jackson-core;2.9.4!jackson-
> core.jar(bundle)]
>
>
> java.lang.RuntimeException: Error grabbing Grapes -- [download failed:
> com.fasterxml.jackson.core#jackson-core;2.9.4!jackson-core.jar(bundle)]
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:62)
>
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
> at org.codehaus.groovy.reflection.CachedConstructor.
> invoke(CachedConstructor.java:83)
>
> at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(
> CachedConstructor.java:77)
>
> at org.codehaus.groovy.runtime.callsite.ConstructorSite$
> ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
>
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.
> defaultCallConstructor(CallSiteArray.java:59)
>
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(
> AbstractCallSite.java:238)
>
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(
> AbstractCallSite.java:250)
>
> at groovy.grape.GrapeIvy.getDependencies(GrapeIvy.groovy:424)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$
> PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
>
> at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(
> PogoMetaMethodSite.java:59)
>
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(
> CallSiteArray.java:51)
>
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.
> callCurrent(AbstractCallSite.java:157)
>
> at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:571)
>
> at groovy.grape.GrapeIvy$resolve$1.callCurrent(Unknown Source)
>
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(
> CallSiteArray.java:51)
>
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.
> callCurrent(AbstractCallSite.java:157)
>
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.
> callCurrent(AbstractCallSite.java:193)
>
> at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:538)
>
> at groovy.grape.GrapeIvy$resolve$0.callCurrent(Unknown Source)
>
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(
> CallSiteArray.java:51)
>
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.
> callCurrent(AbstractCallSite.java:157)
>
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.
> callCurrent(AbstractCallSite.java:185)
>
> at groovy.grape.GrapeIvy.grab(GrapeIvy.groovy:256)
>
> at groovy.grape.Grape.grab(Grape.java:167)
>
> at groovy.grape.GrabAnnotationTransformation.visit(
> GrabAnnotationTransformation.java:376)
>
> at org.codehaus.groovy.transform.ASTTransformationVisitor$3.
> call(ASTTransformationVisitor.java:346)
>
> at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(
> CompilationUnit.java:966)
>
> at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(
> CompilationUnit.java:626)
>
> at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(
> CompilationUnit.java:602)
>
> at org.codehaus.groovy.control.CompilationUnit.compile(
> CompilationUnit.java:579)
>
> at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
>
> at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
>
> at groovy.lang.GroovyShell.parseClass(GroovyShell.java:677)
>
> at groovy.lang.GroovyShell.run(GroovyShell.java:506)
>
> at groovy.lang.GroovyShell.run(GroovyShell.java:496)
>
> at groovy.ui.GroovyMain.processOnce(GroovyMain.java:597)
>
> at groovy.ui.GroovyMain.run(GroovyMain.java:329)
>
> at groovy.ui.GroovyMain.process(GroovyMain.java:315)
>
> at groovy.ui.GroovyMain.processArgs(GroovyMain.java:134)
>
> at groovy.ui.GroovyMain.main(GroovyMain.java:114)
>
> at