Hi Alexey, thanks for looking into it.
I'd like to share the result from tls_handshake-test since that seems to be getting closer to the root cause: [ RUN ] TestTlsHandshake.TestHandshakeSequence /home/cvaliente/git/kudu/src/kudu/security/tls_handshake-test.cc:206: Failure Value of: client.Continue(buf1, &buf2).IsIncomplete() Actual: false Expected: true it fails at this position: // Client receives server Hello and sends client Finished ASSERT_TRUE(client.Continue(buf1, &buf2).IsIncomplete()); So for some reason the client believes it has not finished yet. Cheers Clemens ________________________________ From: Alexey Serbin <[email protected]> Sent: 19 April 2019 18:10:51 To: dev Subject: Re: java unit tests failing when starting MiniKuduCluster Hi Clemens, Thank you for the information. Yes, to me it looks like an issue with IO over TLS-encrypted channels. I don't have access to Fedora29 and newer, so I didn't do any troubleshooting/debugging this time. However, if you really want to run on Fedora29, you can disable authentication and encryption (--rpc_authentication=disabled --rpc_encryption=disabled) and it seems that should work fine on Fedora29. Other options are running on Linux platforms that are currently listed at https://kudu.apache.org/docs/installation.html#_prerequisites_and_requirements or going forward with troubleshooting on your own, where good starting points might be tls_handshake-test and tls_socket-test in addition to negotiation-test. I'm also planning to take a look at the issue soon. I'll let you know once I find anything of particular interest there. Kind regards, Alexey On Thu, Apr 18, 2019 at 2:14 AM Clemens Valiente < [email protected]> wrote: > Hi Alexey, > > > find the tests attached. > > The pattern that I can see is that the successful tests are those without > a signed certificate (server: {pki: NONE) and the ones with disabled > encryption on client- or serverside (encryption: DISABLED) - but those > last ones are not supposed to establish a successful connection anyway. > > The failed tests all require some signed key on the server's side and > encryption supported by both sides. So it might hint at an SSL issue. > > I am using a FIPS enabled version of openssl: > > OpenSSL 1.1.1b FIPS 26 Feb 2019 > > > Cheers > > Clemens > ------------------------------ > *From:* Alexey Serbin <[email protected]> > *Sent:* 17 April 2019 20:49:29 > *To:* dev > *Subject:* Re: java unit tests failing when starting MiniKuduCluster > > Hi Clemens, > > Could you run the negotiation-test (built without your patch) and attach > the output? The test loops through various combinations of client/server > configurations, so it would be much easier to see what works and what not. > I suspect there might be some issues related to the combination of newer > glibc and OpenSSL libraries. > > > Kind regards, > > Alexey > > On Wed, Apr 17, 2019 at 3:07 AM Clemens Valiente < > [email protected]> wrote: > > > Hi Adar, > > > > thanks for looking into it. > > > > I had to apply the patches mentioned in KUDU-2770 to be able to build > kudu > > at all. As mentioned in the ticket, I am using fedora 29 which comes with > > glibc 2.28. > > > > I made sure that everything built correctly and ran the c++ tests and got > > many failing tests. They all seem to occur when trying to connect, the > > first stacktrace usually leads to > > > > > > 0417 12:01:53.737010 (+ 3212us) negotiation.cc:304] Negotiation > complete: > > IO error: Server connection negotiation failed: server connection from > > 127.4.196.1:38781: received invalid message of size 386073344 which > > exceeds the rpc_max_message_size of 52428800 bytes > > > > > > This message size of 386073344 bytes is the same across all failed tests > > so I suspect that to be part of the issue, but I haven't been able to > > figure out what is inside that message. Is there any debug logging I > could > > activate on the tests? I can now easily reproduce a failure with > > > > > > bin/rpc_line_item_dao-test --gtest_filter=RpcLineItemDAOTest.TestInsert > > > > > > Thanks a lot for the help > > > > Clemens > > > > > > ------------------------------ > > *From:* Adar Lieber-Dembo <[email protected]> > > *Sent:* 16 April 2019 21:15:23 > > *To:* [email protected] > > *Subject:* Re: java unit tests failing when starting MiniKuduCluster > > > > I'm guessing it's something to do with your local system. You had > > previously filed KUDU-2770 regarding incompatibilities with newer > > versions of glibc; what system/distro are you running on? Are you > > using _any_ Kudu patches whatsoever, even just to get the build (or > > thirdparty build) working? > > > > FWIW, these tests pass locally for me (running against master) and > > they're passing in Kudu precommit tests regularly. Could you try to > > run the C++ test suite? They're easier to debug when they fail. > > > > On Tue, Apr 16, 2019 at 12:53 AM Clemens Valiente > > <[email protected]> wrote: > > > > > > Hi, > > > > > > > > > I added these flags after the initial runs that complained about > > rpc_max_message_size exceeded in the tests. That got me one step further > to > > the connection failed error. > > > > > > > > > I tried running the tests both on master and on the kudu 1.9.0 release > > (with the appropriate kudu build), with and without my patch. > > > > > > ________________________________ > > > From: Adar Lieber-Dembo <[email protected]> > > > Sent: 14 April 2019 19:59:45 > > > To: [email protected] > > > Subject: Re: java unit tests failing when starting MiniKuduCluster > > > > > > It might be related to your local changes. This caught my eye: > > > > > > extra_master_flags: "--rpc_max_message_size=3860733440" > > > extra_tserver_flags: "--rpc_max_message_size=3860733440" > > > > > > Why did you have to add this configuration? What happens if you remove > > > it? On a related note, do the tests still fail if you rebuild from a > > > clean working tree (i.e. without your patches)? > > > > > > On Sat, Apr 13, 2019 at 12:52 PM Clemens Valiente > > > <[email protected]> wrote: > > > > > > > > Hi, > > > > > > > > I have a few fixes for the kudu-mapreduce package and wanted to add > > unit tests but neither the existing nor the newly added unit tests can be > > run on my system. > > > > > > > > I managed to build kudu and theMiniKuduCluster starts up but then > > fails with connection/timeout issues. > > > > > > > > I think this might point at the problem: > > > > > > > > 16:42:05.780 [INFO - cluster stderr printer] > > (MiniKuduCluster.java:543) 0410 16:42:05.779084 (+3053866us) > > negotiation.cc:304] Negotiation complete: Timed out: Server connection > > negotiation failed: server connection from 127.0.0.1:60698 > > > > > > > > Though I have no explanation as to why kudu would not be able to make > > local connections within my machine. (fedora 29, OpenSSL 1.1.1b FIPS 26 > > Feb 2019) > > > > I attached the full test report. Can someone help me figure out how > to > > further debug this, or is there any additional information I can provide? > > > > > > > > Best Regards > > > > Clemens Valiente > > >
