Hey Henk! Could you make any progress remote debugging your issue? Although not guaranteed to help, you may try and use strace to look into what's happening on the syscall level.
For example I've ran this command after starting the server: strace -fp <PID_OF_JAVA_PROCESS> -s 100 2>&1 | grep -v clock_gettime | grep -v futex Also it can help if you strace the first invocation of the server, because then at least we can see what classes are loaded from the stat calls and hopefully guess where the exception actually came from. On Tue, Jun 5, 2018 at 8:25 PM Andy LoPresto <[email protected]> wrote: > Henk, > > I understand if you don’t want to share the value of your CA hostname, but > can you please double check that it doesn’t contain any spaces or other > characters that would need to be escaped? Using double quotes is usually > recommended for these arguments, especially as the single quotes are only > wrapping the internal shell command to be executed as the “linux-user”. I > see in your original client log output that it appears to be successful in > parsing, but it doesn’t hurt to be sure. > > Frustrating though it may be, I would recommend enabling remote debugging > on the toolkit server application and putting a breakpoint > at TlsCertificateAuthorityServiceHandler.java line 99 where you can catch > the actual exception that is being thrown and examine it. Once we have more > information (or we have sufficient input & scenario such that we can > reproduce this issue locally), we can open a Jira to improve the error > handling here. Sorry the tool is not more helpful. > > To enable remote debugging: > > Modify tis-toolkit.sh line 114 to look like the following: > > exec "${JAVA}" -cp "${CLASSPATH}" ${JAVA_OPTS:--Xms12m -Xmx24m} > -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y > org.apache.nifi.toolkit.tls.TlsToolkitMain "$@" > > That will pause the running application until you can start debugging in > your IDE, and then you can resume operation on the shell. > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jun 5, 2018, at 9:39 AM, Henk Reder <[email protected]> wrote: > > Hey Peter! > > in response to this > < > http://mail-archives.apache.org/mod_mbox/nifi-dev/201806.mbox/raw/%3ccalj3mba0ny7zmh66vrmsyp+7fax490yajkiwv5-vc5cb9hj...@mail.gmail.com%3E/1 > > > . > > Thanks for being quick on your response. I added that -c <ca-hostname> > parameter to the CA server setup, same issue occurs. > That "&" is actually an artifact of linux, it just means "run whatever this > line is on its own thread." I verified that it wasn't polluting the > arguments. > I also removed the --dn option from the server configuration and the issue > was consistent, response is still a 500. > > I'm thinking a rollback to 1.5.0 should fix this, but it would appear that > whatever changes happened in the transition between versions broke that > existing configuration. > > Any additional ideas or should I go ahead and roll it back? > > Thanks again, > Henk > > > On Mon, Jun 4, 2018 at 8:34 PM, Henk Reder <[email protected]> wrote: > > Hello! > > My names Henk. I'm a web developer working on getting a nifi instance > setup for some integrations. > > Long story short, I setup this nifi deployment configuration back when > 1.5.0 was the latest version. Now that its 1.6.0 everything works! > ...except one thing. > > I'm currently using tls-toolkit to setup each nodes identity in the > cluster. In order to do this, I've followed the guidelines from this post > < > https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/> > back > from 2016. heres a brief overview of my previously working configuration. > > *CA serverside:* > > <download tls-toolkit> > <setup a linux-user> > runuser -l <linux-user> -c '/home/nifica/bin/tls-toolkit.sh server -t > <secret-token> -D CN=<ca-hostname>,OU=NIFI&' > > *Nifi-client side:* > > <download nifi> > <download tls-toolkit> > ./nifi-toolkit/bin/tls-toolkit.sh client -c <ca-hostname> -t > <secret-token> --subjectAlternativeNames '<node-ip>,<node-hostname>' -D > 'CN=<node-hostname>,OU=NIFI' -T PKCS12 > > I expect the actual certs and json config to comeback after the call to > the tls-toolkit server. I can verify the request is going through but I get > these obtuse messages that tell me very little. > > > > *Nifi-client side: * > 2018/06/05 00:23:43 INFO [main] org.apache.nifi.toolkit.tls. > service.client.TlsCertificateAuthorityClient: Requesting new certificate > from <ca-hostname>:8443 > 2018/06/05 00:23:44 INFO [main] org.apache.nifi.toolkit.tls. > service.client.TlsCertificateSigningRequestPerformer: Requesting > certificate with dn CN=<hostname>,OU=NIFI from <ca-hostname>:8443 > Service client error: Received response code 500 with payload > > *CA serverside: * > > 2018/06/04 22:55:50 WARN [qtp1108924067-13] > org.eclipse.jetty.server.HttpChannel: > / > javax.servlet.ServletException: Server error > at org.apache.nifi.toolkit.tls.service.server. > TlsCertificateAuthorityServiceHandler.handle( > TlsCertificateAuthorityServiceHandler.java:99) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle( > HandlerWrapper.java:132) > at org.eclipse.jetty.server.Server.handle(Server.java:564) > at org.eclipse.jetty.server.HttpChannel.handle( > HttpChannel.java:369) > at org.eclipse.jetty.server.HttpConnection.onFillable( > HttpConnection.java:251) > at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( > AbstractConnection.java:279) > at org.eclipse.jetty.io.FillInterest.fillable( > FillInterest.java:110) > at org.eclipse.jetty.io.ssl.SslConnection.onFillable( > SslConnection.java:258) > at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded( > SslConnection.java:147) > at org.eclipse.jetty.io.FillInterest.fillable( > FillInterest.java:110) > at org.eclipse.jetty.io.ChannelEndPoint$2.run( > ChannelEndPoint.java:124) > at org.eclipse.jetty.util.thread.Invocable.invokePreferred( > Invocable.java:122) > at org.eclipse.jetty.util.thread.strategy. > ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58) > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume. > produceConsume(ExecuteProduceConsume.java:201) > at org.eclipse.jetty.util.thread.strategy. > ExecuteProduceConsume.run(ExecuteProduceConsume.java:133) > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > QueuedThreadPool.java:672) > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( > QueuedThreadPool.java:590) > at java.lang.Thread.run(Thread.java:748) > 2018/06/04 22:55:50 WARN [qtp1108924067-13] org.eclipse.jetty.server. > HttpChannel: > java.lang.IllegalStateException: Error already set > at org.eclipse.jetty.server.HttpChannelState.onError( > HttpChannelState.java:743) > at org.eclipse.jetty.server.HttpChannel.handleException( > HttpChannel.java:514) > at org.eclipse.jetty.server.HttpChannelOverHttp.handleException( > HttpChannelOverHttp.java:463) > at org.eclipse.jetty.server.HttpChannel.handle( > HttpChannel.java:448) > at org.eclipse.jetty.server.HttpConnection.onFillable( > HttpConnection.java:251) > at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( > AbstractConnection.java:279) > at org.eclipse.jetty.io.FillInterest.fillable( > FillInterest.java:110) > at org.eclipse.jetty.io.ssl.SslConnection.onFillable( > SslConnection.java:258) > at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded( > SslConnection.java:147) > at org.eclipse.jetty.io.FillInterest.fillable( > FillInterest.java:110) > at org.eclipse.jetty.io.ChannelEndPoint$2.run( > ChannelEndPoint.java:124) > at org.eclipse.jetty.util.thread.Invocable.invokePreferred( > Invocable.java:122) > at org.eclipse.jetty.util.thread.strategy. > ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58) > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume. > produceConsume(ExecuteProduceConsume.java:201) > at org.eclipse.jetty.util.thread.strategy. > ExecuteProduceConsume.run(ExecuteProduceConsume.java:133) > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > QueuedThreadPool.java:672) > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( > QueuedThreadPool.java:590) > at java.lang.Thread.run(Thread.java:748) > > > Seeing as this code was working before, my first assumption is the problem > has something to do with some new undocumented behaviors. I fooled around > with the parameters for a bit and got very similar results, and a lot of > research and digging has left me scratching my head. I know its a 500, and > the CA server is the one throwing the exception, but I can't quite figure > out what changed and why its behaving this way. There's also no clear > indicator of what's causing the issue. I can see a callback being passed > down the stack but no message hinting at bad parameters or anything like > that. > > If it helps, this is the java version I'm running on my machine: > > > openjdk version "1.8.0_171" > OpenJDK Runtime Environment (build 1.8.0_171-b10) > OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode) > > If you have any further questions, please don't hesitate to ask. Thanks > for your time! > Henk > > > >
