Hi Flavio, When I tested the TLS setup on a single machine, I also had issues, java errors. It turns out those error were completely unrelated to the problem, namely I didn't have the localhost setup in my hostname file. If I remember correctly, I needed to add localhost 127.0.0.1, because I only had the machine's name setup there. So it couldn't find "localhost".
Not entirely sure the solution anymore, but it was definitely something with localhost. After that, it worked fine for me, but I also used a single key for all instances on the machine. Regards, Norbert On Tue, Apr 30, 2019 at 1:29 PM Andor Molnar <[email protected]> wrote: > Hi Flavio, > > Works for me on a single machine with the following keystores. > Aliases are the hostname of the machine (all the same). > > keystore1.jks > ~~~~~~~~~~ > Keystore type: JKS > Keystore provider: SUN > > Your keystore contains 1 entry > > andors-macbook-pro.local, Apr 30, 2019, PrivateKeyEntry, > Certificate fingerprint (SHA1): > 14:46:5F:92:D9:03:88:7E:C9:0A:95:9E:F5:74:08:F4:27:89:36:9D > > keystore2.jks > ~~~~~~~~~~ > Keystore type: JKS > Keystore provider: SUN > > Your keystore contains 1 entry > > andors-macbook-pro.local, Apr 30, 2019, PrivateKeyEntry, > Certificate fingerprint (SHA1): > 61:11:F3:FC:97:B1:3D:DB:6C:65:11:AE:FB:26:39:C0:4F:8E:A7:F7 > > keystore3.jks > ~~~~~~~~~~ > Keystore type: JKS > Keystore provider: SUN > > Your keystore contains 1 entry > > andors-macbook-pro.local, Apr 30, 2019, PrivateKeyEntry, > Certificate fingerprint (SHA1): > E0:84:2A:37:A0:8E:22:67:B3:50:21:43:34:D0:FD:E8:A4:50:C4:3F > > > …and a single truststore: > > $ keytool -list -keystore truststore.jks > Enter keystore password: > Keystore type: JKS > Keystore provider: SUN > > Your keystore contains 4 entries > > mycert3, Apr 30, 2019, trustedCertEntry, > Certificate fingerprint (SHA1): > E0:84:2A:37:A0:8E:22:67:B3:50:21:43:34:D0:FD:E8:A4:50:C4:3F > mycert2, Apr 30, 2019, trustedCertEntry, > Certificate fingerprint (SHA1): > 61:11:F3:FC:97:B1:3D:DB:6C:65:11:AE:FB:26:39:C0:4F:8E:A7:F7 > mycert1, Apr 30, 2019, trustedCertEntry, > Certificate fingerprint (SHA1): > 14:46:5F:92:D9:03:88:7E:C9:0A:95:9E:F5:74:08:F4:27:89:36:9D > > Aliases (mycert1, mycert2, mycert3) doesn’t matter here, ZooKeeper only > checks if the given certificate is included in the truststore or not. > > Regards, > Andor > > > > > On 2019. Apr 30., at 12:48, Andor Molnar <[email protected]> wrote: > > > > Makes sense. > > > > I’ve tested it on a single machine with the same cert/key for all > instances: keystore / truststore only contained a single entry and it > worked fine. > > We’ve also tested on multiple instances with multiple keys / instances > which also worked fine. > > > > Let me give it another go with single machine / multiple certs combo. > > > > I might need to modify the docs to emphasize keys must be generated on a > per machine basis, not ZK instance. > > > > Regards, > > Andor > > > > > > > >> On 2019. Apr 29., at 16:53, Flavio Junqueira <[email protected]> wrote: > >> > >> I'm also +1 for adding a comment to the release notes (thanks for the > suggestion, Ted). Updating the readme makes sense, but the release notes > will be the main source to indicate that we require a specific or later > version of Java from that particular release. My preference would be to > update the release notes. > >> > >> As for running TLS on a single node, have you been able to do it? I > haven't had a chance to look further into it throughout my day, so if > anyone has successfully done it and can share some instructions, it would > help me. Otherwise, I'll keep investigating once I have a chance. To be > specific, I created the keystore, certificate and truststore files > according to instructions, but the instructions assume that the aliases are > different when it comes to populating the truststore. At that point, I had > to get creative and I have tried a couple of options that didn't work. > Either way, I think that being able to run locally and documenting is > desirable, although not a blocker. If I can get it right, then I can write > a gist describing it that we can use as a reference until we properly > document it. > >> > >> -Flavio > >> > >>> On 29 Apr 2019, at 15:42, Andor Molnar <[email protected]> wrote: > >>> > >>> Thanks Flavio for the investigation. I’ll update the README file to > include instructions on supported Java 8 versions. > >>> I’m wondering if I have to update the admin docs based on your > problems running TLS quorum on a single machine. > >>> > >>> Andor > >>> > >>> > >>> > >>>> On 2019. Apr 29., at 15:06, Enrico Olivelli <[email protected]> > wrote: > >>>> > >>>> Il lun 29 apr 2019, 13:44 Ted Dunning <[email protected]> ha > scritto: > >>>> > >>>>> Other changes in u211+ substantially improve how Java 8 applications > behave > >>>>> in containers. I am seeing this more and more with customers. > >>>>> > >>>>> Combined with the crypto issues, it might be worth a release note > >>>>> suggesting that if you are going to compile with Java 1.8, you > should use a > >>>>> recent release at u211 (u212?) Or above. > >>>>> > >>>> > >>>> +1 for a note on release docs > >>>> > >>>> > >>>> Enrico > >>>> > >>>> > >>>> > >>>> > >>>>> On Mon, Apr 29, 2019, 11:43 AM Flavio Junqueira <[email protected]> > wrote: > >>>>> > >>>>>> I did a bit more research and it turns out that the crypto.policy > option > >>>>>> was introduced u151: > >>>>>> > >>>>>> > >>>>> > https://www.oracle.com/technetwork/java/javase/8u151-relnotes-3850493.html > >>>>>> < > >>>>>> > >>>>> > https://www.oracle.com/technetwork/java/javase/8u151-relnotes-3850493.html > >>>>>>> > >>>>>> > >>>>>> And started being defined by default with the "unlimited" option in > u161: > >>>>>> > >>>>>> > >>>>> > https://www.oracle.com/technetwork/java/javase/8u161-relnotes-4021379.html > >>>>>> < > >>>>>> > >>>>> > https://www.oracle.com/technetwork/java/javase/8u161-relnotes-4021379.html > >>>>>>> > >>>>>> > >>>>>> I have installed a more recent version, 1.8.0_211, and it builds > fine > >>>>> (all > >>>>>> tests pass consistently for me). > >>>>>> > >>>>>> > >>>>>> I'm now trying to start an ensemble with ssl enabled locally, but > it is > >>>>>> failing for me. It looks like the instructions in the admin doc > assumes > >>>>>> different hosts. I need to look more closely into it to determine > what > >>>>> not > >>>>>> is that I'm doing wrong, but in any case, the instructions do not > make it > >>>>>> very clear whether one can run locally. > >>>>>> > >>>>>> -Flavio > >>>>>> > >>>>>>> On 27 Apr 2019, at 19:28, Patrick Hunt <[email protected]> wrote: > >>>>>>> > >>>>>>> Odd. I had done my testing on jdk11/macos which is fine. > >>>>>>> > >>>>>>> I just tried jdk8 and 3 times in a row it's failing with: > >>>>>>> [ERROR] > SaslAuthTest.testZKOperationsAfterClientSaslAuthFailure:176 » > >>>>>>> Timeout Failed t... > >>>>>>> > >>>>>>> I don't see the error Flavio is seeing. I have never installed > special > >>>>>>> crypto libraries, etc... just vanilla jdk. > >>>>>>> > >>>>>>> ⌂102% [phunt:~/Downloads/z/apache-zookeeper-3.5.5] 3s $ mvn > --version > >>>>>>> Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; > >>>>>>> 2019-04-04T12:00:29-07:00) > >>>>>>> Maven home: /usr/local/Cellar/maven/3.6.1/libexec > >>>>>>> Java version: 1.8.0_201, vendor: Oracle Corporation, runtime: > >>>>>>> > /Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home/jre > >>>>>>> Default locale: en_US, platform encoding: UTF-8 > >>>>>>> OS name: "mac os x", version: "10.14.4", arch: "x86_64", family: > "mac" > >>>>>>> > >>>>>>> > >>>>>>> 2019-04-27 10:11:51,635 [myid:] - INFO > >>>>>>> [NIOWorkerThread-6:SaslServerCallbackHandler@136] - Setting > >>>>>> authorizedID: > >>>>>>> super > >>>>>>> 2019-04-27 10:11:51,636 [myid:] - INFO > >>>>>>> [NIOWorkerThread-6:ZooKeeperServer@1170] - adding SASL > authorization > >>>>> for > >>>>>>> authorizationID: super > >>>>>>> 2019-04-27 10:11:51,813 [myid:] - INFO > >>>>>>> [SessionTracker:SessionTrackerImpl@163] - SessionTrackerImpl > exited > >>>>>> loop! > >>>>>>> 2019-04-27 10:12:21,596 [myid:] - INFO > >>>>>>> [main:JUnit4ZKTestRunner$LoggedInvokeMethod@98] - TEST METHOD > FAILED > >>>>>>> testZKOperationsAfterClientSaslAuthFailure > >>>>>>> java.util.concurrent.TimeoutException: Failed to connect to > ZooKeeper > >>>>>>> server. > >>>>>>> at > >>>>>>> > >>>>>> > >>>>> > org.apache.zookeeper.test.ClientBase$CountdownWatcher.waitForConnected(ClientBase.java:150) > >>>>>>> at > >>>>>>> > >>>>>> > >>>>> > org.apache.zookeeper.SaslAuthTest.testZKOperationsAfterClientSaslAuthFailure(SaslAuthTest.java:176) > >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>>>>>> at > >>>>>>> > >>>>>> > >>>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>>>>>> at > >>>>>>> > >>>>>> > >>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) > >>>>>>> at > >>>>>>> > >>>>>> > >>>>> > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > >>>>>>> at > >>>>>>> > >>>>>> > >>>>> > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > >>>>>>> > >>>>>>> > >>>>>>> Patrick > >>>>>>> > >>>>>>> > >>>>>>> On Sat, Apr 27, 2019 at 8:54 AM Enrico Olivelli < > [email protected]> > >>>>>> wrote: > >>>>>>> > >>>>>>>> On my local tests I usually don't get the error because I am using > >>>>>>>> jdk11 and unlimited strength cryptography is enabled by default > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > https://www.oracle.com/technetwork/java/javase/documentation/jdk11-readme-5097204.html#jce > >>>>>>>> > >>>>>>>> In production it depends on the configuration of SSL, if you > require > >>>>>>>> strong ciphers/big keys you will have problems, but the server > won't > >>>>>>>> start so you will find soon the problem. > >>>>>>>> I think this is not a real issue (for production I mean). > >>>>>>>> I see these ways: > >>>>>>>> 1) adapt the tests in order to make default jdk8 happy > >>>>>>>> 2) tweak the tests enabling "unlimited strenght cryptography" > using > >>>>>>>> reflection > >>>>>>>> 3) just write a line in documentation that says that in order to > make > >>>>>>>> tests pass you have to enable the flag > >>>>>>>> > >>>>>>>> That flag is deprecated and enabled by default in modern JREs, so > I > >>>>>>>> would go for 2) or 3) > >>>>>>>> > >>>>>>>> I guess that on ASF Jenkins if the JDK8 we are using has the flag > >>>>>> turned > >>>>>>>> on > >>>>>>>> > >>>>>>>> Enrico > >>>>>>>> > >>>>>>>> Il giorno sab 27 apr 2019 alle ore 17:48 Andor Molnar > >>>>>>>> <[email protected]> ha scritto: > >>>>>>>>> > >>>>>>>>> I’m running the tests fine without setting the policy to > unlimited: > >>>>>>>>> > >>>>>>>>> java version "1.8.0_161" > >>>>>>>>> Java(TM) SE Runtime Environment (build 1.8.0_161-b12) > >>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode) > >>>>>>>>> > >>>>>>>>> Have you tried to run it with a more recent version of Java? > >>>>>>>>> > >>>>>>>>> Andor > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> On 2019. Apr 27., at 17:33, Andor Molnar <[email protected]> > wrote: > >>>>>>>>>> > >>>>>>>>>> Good catch, thanks Flavio for reporting this. We need to double > >>>>> check > >>>>>>>> the tests with Ilya I believe. > >>>>>>>>>> > >>>>>>>>>> Having tests failure means that you were actually able to > _build_ > >>>>>>>> ZooKeeper successfully without changing the crypto policy setting. > >>>>> Have > >>>>>> you > >>>>>>>> tried to start an ensemble with Quorum TLS by any chance? That > would > >>>>> add > >>>>>>>> some more color to this issue. > >>>>>>>>>> > >>>>>>>>>> This might be just a testing issue. > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> Andor > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On 2019. Apr 27., at 16:09, Flavio Junqueira <[email protected]> > >>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi Enrico, > >>>>>>>>>>> > >>>>>>>>>>> Here is the info you are requesting: > >>>>>>>>>>> > >>>>>>>>>>> *Java version* > >>>>>>>>>>> > >>>>>>>>>>> $ java -version > >>>>>>>>>>> java version "1.8.0_152" > >>>>>>>>>>> Java(TM) SE Runtime Environment (build 1.8.0_152-b16) > >>>>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed > mode) > >>>>>>>>>>> > >>>>>>>>>>> *Test case errors* > >>>>>>>>>>> > >>>>>>>>>>> I won’t post all of them, I get a good number of errors: > >>>>>>>>>>> > >>>>>>>>>>> ================================ > >>>>>>>>>>> [ERROR] Tests run: 64, Failures: 0, Errors: 16, Skipped: 0, > Time > >>>>>>>> elapsed: 9.21 s <<< FAILURE! - in > >>>>>> org.apache.zookeeper.util.PemReaderTest > >>>>>>>>>>> [ERROR] > >>>>>>>> > >>>>>> > >>>>> > testLoadCertificateFromKeyStore[1](org.apache.zookeeper.util.PemReaderTest) > >>>>>>>> Time elapsed: 1.593 s <<< ERROR! > >>>>>>>>>>> java.io.IOException: > >>>>>>>> org.bouncycastle.operator.OperatorCreationException: Illegal key > size > >>>>> or > >>>>>>>> default parameters > >>>>>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> > org.apache.zookeeper.util.PemReaderTest.testLoadCertificateFromKeyStore(PemReaderTest.java:125) > >>>>>>>>>>> Caused by: org.bouncycastle.operator.OperatorCreationException: > >>>>>>>> Illegal key size or default parameters > >>>>>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> > org.apache.zookeeper.util.PemReaderTest.testLoadCertificateFromKeyStore(PemReaderTest.java:125) > >>>>>>>>>>> Caused by: java.security.InvalidKeyException: Illegal key size > or > >>>>>>>> default parameters > >>>>>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> > org.apache.zookeeper.util.PemReaderTest.testLoadCertificateFromKeyStore(PemReaderTest.java:125) > >>>>>>>>>>> > >>>>>>>>>>> [ERROR] > >>>>>>>> > >>>>>> > >>>>> > testLoadEncryptedPrivateKeyFromKeyStoreWithWrongPassword[1](org.apache.zookeeper.util.PemReaderTest) > >>>>>>>> Time elapsed: 0.004 s <<< ERROR! > >>>>>>>>>>> java.lang.Exception: Unexpected exception, > >>>>>>>> expected<java.security.GeneralSecurityException> but > >>>>>>>> was<java.io.IOException> > >>>>>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> > org.apache.zookeeper.util.PemReaderTest.testLoadEncryptedPrivateKeyFromKeyStoreWithWrongPassword(PemReaderTest.java:93) > >>>>>>>>>>> Caused by: org.bouncycastle.operator.OperatorCreationException: > >>>>>>>> Illegal key size or default parameters > >>>>>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> > org.apache.zookeeper.util.PemReaderTest.testLoadEncryptedPrivateKeyFromKeyStoreWithWrongPassword(PemReaderTest.java:93) > >>>>>>>>>>> Caused by: java.security.InvalidKeyException: Illegal key size > or > >>>>>>>> default parameters > >>>>>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> > org.apache.zookeeper.util.PemReaderTest.testLoadEncryptedPrivateKeyFromKeyStoreWithWrongPassword(PemReaderTest.java:93) > >>>>>>>>>>> ... > >>>>>>>>>>> ================================ > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> *Crypto policy* > >>>>>>>>>>> If I uncomment this configuration option: > >>>>>>>>>>> > >>>>>>>>>>> # Please see the JCA documentation for additional information > on > >>>>>> these > >>>>>>>>>>> # files and formats. > >>>>>>>>>>> # crypto.policy=unlimited > >>>>>>>>>>> > >>>>>>>>>>> in: > >>>>>>>>>>> > >>>>>>>>>>> $JAVA_HOME/jre/lib/security/java.security > >>>>>>>>>>> > >>>>>>>>>>> then it all works and I get no error at all. This option > controls > >>>>>>>> cryptographic strengths according to the documentation, and is > present > >>>>>>>> because of crypto regulations in different countries. > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> -Flavio > >>>>>>>>>>> > >>>>>>>>>>>> On 27 Apr 2019, at 15:52, Enrico Olivelli < > [email protected]> > >>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Il sab 27 apr 2019, 14:18 Flavio Junqueira <[email protected]> > ha > >>>>>>>> scritto: > >>>>>>>>>>>> > >>>>>>>>>>>>> I have a clarification question about the RC. To build the > RC, I > >>>>>>>> had to > >>>>>>>>>>>>> enable crypto.policy unlimited in the jre (I'm using build > >>>>>>>> 1.8.0_152-b16). > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Flavio > >>>>>>>>>>>> What do you mean with 'build' ? > >>>>>>>>>>>> Make tests pass? > >>>>>>>>>>>> AFAIK we are not using tweaked jdks in CI builds, so in theory > >>>>> there > >>>>>>>> is no > >>>>>>>>>>>> need. > >>>>>>>>>>>> > >>>>>>>>>>>> Can you please share your error? > >>>>>>>>>>>> > >>>>>>>>>>>> Enrico > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> I'm wondering if this is going to be an issue for some users > as > >>>>> this > >>>>>>>> option > >>>>>>>>>>>>> is related to import/export regulation. Has anyone looked > into it > >>>>>>>> and could > >>>>>>>>>>>>> clarify it to me, please? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> -Flavio > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On 25 Apr 2019, at 15:10, Andor Molnar <[email protected]> > >>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This is the first stable release of 3.5 branch: 3.5.5. It > >>>>> resolves > >>>>>>>> 117 > >>>>>>>>>>>>> issues, including Maven migration, Quorum TLS, TTL nodes and > lots > >>>>>>>> of other > >>>>>>>>>>>>> performance and stability improvements. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The full release notes is available at: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12343268 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> *** Please download, test and vote by May 3rd 2019, 23:59 > UTC+0. > >>>>>>>> *** > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Source files: > >>>>>>>>>>>>>> > >>>>>>>> > https://dist.apache.org/repos/dist/dev/zookeeper/zookeeper-3.5.5-rc5/ > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Maven staging repos: > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > https://repository.apache.org/content/groups/staging/org/apache/zookeeper/parent/3.5.5/ > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > https://repository.apache.org/content/groups/staging/org/apache/zookeeper/zookeeper-jute/3.5.5/ > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > https://repository.apache.org/content/groups/staging/org/apache/zookeeper/zookeeper/3.5.5/ > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The release candidate tag in git to be voted upon: > >>>>>>>> release-3.5.5-rc5 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> ZooKeeper's KEYS file containing PGP keys we use to sign the > >>>>>>>> release: > >>>>>>>>>>>>>> http://www.apache.org/dist/zookeeper/KEYS > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Should we release this candidate? > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>> > >>> > >> > > > >
