Hi Flavio, Works for me on a single machine with the following keystores. Aliases are the hostname of the machine (all the same).
keystore1.jks ~~~~~~~~~~ Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry andors-macbook-pro.local, Apr 30, 2019, PrivateKeyEntry, Certificate fingerprint (SHA1): 14:46:5F:92:D9:03:88:7E:C9:0A:95:9E:F5:74:08:F4:27:89:36:9D keystore2.jks ~~~~~~~~~~ Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry andors-macbook-pro.local, Apr 30, 2019, PrivateKeyEntry, Certificate fingerprint (SHA1): 61:11:F3:FC:97:B1:3D:DB:6C:65:11:AE:FB:26:39:C0:4F:8E:A7:F7 keystore3.jks ~~~~~~~~~~ Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry andors-macbook-pro.local, Apr 30, 2019, PrivateKeyEntry, Certificate fingerprint (SHA1): E0:84:2A:37:A0:8E:22:67:B3:50:21:43:34:D0:FD:E8:A4:50:C4:3F …and a single truststore: $ keytool -list -keystore truststore.jks Enter keystore password: Keystore type: JKS Keystore provider: SUN Your keystore contains 4 entries mycert3, Apr 30, 2019, trustedCertEntry, Certificate fingerprint (SHA1): E0:84:2A:37:A0:8E:22:67:B3:50:21:43:34:D0:FD:E8:A4:50:C4:3F mycert2, Apr 30, 2019, trustedCertEntry, Certificate fingerprint (SHA1): 61:11:F3:FC:97:B1:3D:DB:6C:65:11:AE:FB:26:39:C0:4F:8E:A7:F7 mycert1, Apr 30, 2019, trustedCertEntry, Certificate fingerprint (SHA1): 14:46:5F:92:D9:03:88:7E:C9:0A:95:9E:F5:74:08:F4:27:89:36:9D Aliases (mycert1, mycert2, mycert3) doesn’t matter here, ZooKeeper only checks if the given certificate is included in the truststore or not. Regards, Andor > On 2019. Apr 30., at 12:48, Andor Molnar <[email protected]> wrote: > > Makes sense. > > I’ve tested it on a single machine with the same cert/key for all instances: > keystore / truststore only contained a single entry and it worked fine. > We’ve also tested on multiple instances with multiple keys / instances which > also worked fine. > > Let me give it another go with single machine / multiple certs combo. > > I might need to modify the docs to emphasize keys must be generated on a per > machine basis, not ZK instance. > > Regards, > Andor > > > >> On 2019. Apr 29., at 16:53, Flavio Junqueira <[email protected]> wrote: >> >> I'm also +1 for adding a comment to the release notes (thanks for the >> suggestion, Ted). Updating the readme makes sense, but the release notes >> will be the main source to indicate that we require a specific or later >> version of Java from that particular release. My preference would be to >> update the release notes. >> >> As for running TLS on a single node, have you been able to do it? I haven't >> had a chance to look further into it throughout my day, so if anyone has >> successfully done it and can share some instructions, it would help me. >> Otherwise, I'll keep investigating once I have a chance. To be specific, I >> created the keystore, certificate and truststore files according to >> instructions, but the instructions assume that the aliases are different >> when it comes to populating the truststore. At that point, I had to get >> creative and I have tried a couple of options that didn't work. Either way, >> I think that being able to run locally and documenting is desirable, >> although not a blocker. If I can get it right, then I can write a gist >> describing it that we can use as a reference until we properly document it. >> >> -Flavio >> >>> On 29 Apr 2019, at 15:42, Andor Molnar <[email protected]> wrote: >>> >>> Thanks Flavio for the investigation. I’ll update the README file to include >>> instructions on supported Java 8 versions. >>> I’m wondering if I have to update the admin docs based on your problems >>> running TLS quorum on a single machine. >>> >>> Andor >>> >>> >>> >>>> On 2019. Apr 29., at 15:06, Enrico Olivelli <[email protected]> wrote: >>>> >>>> Il lun 29 apr 2019, 13:44 Ted Dunning <[email protected]> ha scritto: >>>> >>>>> Other changes in u211+ substantially improve how Java 8 applications >>>>> behave >>>>> in containers. I am seeing this more and more with customers. >>>>> >>>>> Combined with the crypto issues, it might be worth a release note >>>>> suggesting that if you are going to compile with Java 1.8, you should use >>>>> a >>>>> recent release at u211 (u212?) Or above. >>>>> >>>> >>>> +1 for a note on release docs >>>> >>>> >>>> Enrico >>>> >>>> >>>> >>>> >>>>> On Mon, Apr 29, 2019, 11:43 AM Flavio Junqueira <[email protected]> wrote: >>>>> >>>>>> I did a bit more research and it turns out that the crypto.policy option >>>>>> was introduced u151: >>>>>> >>>>>> >>>>> https://www.oracle.com/technetwork/java/javase/8u151-relnotes-3850493.html >>>>>> < >>>>>> >>>>> https://www.oracle.com/technetwork/java/javase/8u151-relnotes-3850493.html >>>>>>> >>>>>> >>>>>> And started being defined by default with the "unlimited" option in u161: >>>>>> >>>>>> >>>>> https://www.oracle.com/technetwork/java/javase/8u161-relnotes-4021379.html >>>>>> < >>>>>> >>>>> https://www.oracle.com/technetwork/java/javase/8u161-relnotes-4021379.html >>>>>>> >>>>>> >>>>>> I have installed a more recent version, 1.8.0_211, and it builds fine >>>>> (all >>>>>> tests pass consistently for me). >>>>>> >>>>>> >>>>>> I'm now trying to start an ensemble with ssl enabled locally, but it is >>>>>> failing for me. It looks like the instructions in the admin doc assumes >>>>>> different hosts. I need to look more closely into it to determine what >>>>> not >>>>>> is that I'm doing wrong, but in any case, the instructions do not make it >>>>>> very clear whether one can run locally. >>>>>> >>>>>> -Flavio >>>>>> >>>>>>> On 27 Apr 2019, at 19:28, Patrick Hunt <[email protected]> wrote: >>>>>>> >>>>>>> Odd. I had done my testing on jdk11/macos which is fine. >>>>>>> >>>>>>> I just tried jdk8 and 3 times in a row it's failing with: >>>>>>> [ERROR] SaslAuthTest.testZKOperationsAfterClientSaslAuthFailure:176 » >>>>>>> Timeout Failed t... >>>>>>> >>>>>>> I don't see the error Flavio is seeing. I have never installed special >>>>>>> crypto libraries, etc... just vanilla jdk. >>>>>>> >>>>>>> ⌂102% [phunt:~/Downloads/z/apache-zookeeper-3.5.5] 3s $ mvn --version >>>>>>> Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; >>>>>>> 2019-04-04T12:00:29-07:00) >>>>>>> Maven home: /usr/local/Cellar/maven/3.6.1/libexec >>>>>>> Java version: 1.8.0_201, vendor: Oracle Corporation, runtime: >>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home/jre >>>>>>> Default locale: en_US, platform encoding: UTF-8 >>>>>>> OS name: "mac os x", version: "10.14.4", arch: "x86_64", family: "mac" >>>>>>> >>>>>>> >>>>>>> 2019-04-27 10:11:51,635 [myid:] - INFO >>>>>>> [NIOWorkerThread-6:SaslServerCallbackHandler@136] - Setting >>>>>> authorizedID: >>>>>>> super >>>>>>> 2019-04-27 10:11:51,636 [myid:] - INFO >>>>>>> [NIOWorkerThread-6:ZooKeeperServer@1170] - adding SASL authorization >>>>> for >>>>>>> authorizationID: super >>>>>>> 2019-04-27 10:11:51,813 [myid:] - INFO >>>>>>> [SessionTracker:SessionTrackerImpl@163] - SessionTrackerImpl exited >>>>>> loop! >>>>>>> 2019-04-27 10:12:21,596 [myid:] - INFO >>>>>>> [main:JUnit4ZKTestRunner$LoggedInvokeMethod@98] - TEST METHOD FAILED >>>>>>> testZKOperationsAfterClientSaslAuthFailure >>>>>>> java.util.concurrent.TimeoutException: Failed to connect to ZooKeeper >>>>>>> server. >>>>>>> at >>>>>>> >>>>>> >>>>> org.apache.zookeeper.test.ClientBase$CountdownWatcher.waitForConnected(ClientBase.java:150) >>>>>>> at >>>>>>> >>>>>> >>>>> org.apache.zookeeper.SaslAuthTest.testZKOperationsAfterClientSaslAuthFailure(SaslAuthTest.java:176) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>> at >>>>>>> >>>>>> >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>> at >>>>>>> >>>>>> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>>> at >>>>>>> >>>>>> >>>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) >>>>>>> at >>>>>>> >>>>>> >>>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) >>>>>>> >>>>>>> >>>>>>> Patrick >>>>>>> >>>>>>> >>>>>>> On Sat, Apr 27, 2019 at 8:54 AM Enrico Olivelli <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>>> On my local tests I usually don't get the error because I am using >>>>>>>> jdk11 and unlimited strength cryptography is enabled by default >>>>>>>> >>>>>>>> >>>>>> >>>>> https://www.oracle.com/technetwork/java/javase/documentation/jdk11-readme-5097204.html#jce >>>>>>>> >>>>>>>> In production it depends on the configuration of SSL, if you require >>>>>>>> strong ciphers/big keys you will have problems, but the server won't >>>>>>>> start so you will find soon the problem. >>>>>>>> I think this is not a real issue (for production I mean). >>>>>>>> I see these ways: >>>>>>>> 1) adapt the tests in order to make default jdk8 happy >>>>>>>> 2) tweak the tests enabling "unlimited strenght cryptography" using >>>>>>>> reflection >>>>>>>> 3) just write a line in documentation that says that in order to make >>>>>>>> tests pass you have to enable the flag >>>>>>>> >>>>>>>> That flag is deprecated and enabled by default in modern JREs, so I >>>>>>>> would go for 2) or 3) >>>>>>>> >>>>>>>> I guess that on ASF Jenkins if the JDK8 we are using has the flag >>>>>> turned >>>>>>>> on >>>>>>>> >>>>>>>> Enrico >>>>>>>> >>>>>>>> Il giorno sab 27 apr 2019 alle ore 17:48 Andor Molnar >>>>>>>> <[email protected]> ha scritto: >>>>>>>>> >>>>>>>>> I’m running the tests fine without setting the policy to unlimited: >>>>>>>>> >>>>>>>>> java version "1.8.0_161" >>>>>>>>> Java(TM) SE Runtime Environment (build 1.8.0_161-b12) >>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode) >>>>>>>>> >>>>>>>>> Have you tried to run it with a more recent version of Java? >>>>>>>>> >>>>>>>>> Andor >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 2019. Apr 27., at 17:33, Andor Molnar <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> Good catch, thanks Flavio for reporting this. We need to double >>>>> check >>>>>>>> the tests with Ilya I believe. >>>>>>>>>> >>>>>>>>>> Having tests failure means that you were actually able to _build_ >>>>>>>> ZooKeeper successfully without changing the crypto policy setting. >>>>> Have >>>>>> you >>>>>>>> tried to start an ensemble with Quorum TLS by any chance? That would >>>>> add >>>>>>>> some more color to this issue. >>>>>>>>>> >>>>>>>>>> This might be just a testing issue. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Andor >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 2019. Apr 27., at 16:09, Flavio Junqueira <[email protected]> >>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Enrico, >>>>>>>>>>> >>>>>>>>>>> Here is the info you are requesting: >>>>>>>>>>> >>>>>>>>>>> *Java version* >>>>>>>>>>> >>>>>>>>>>> $ java -version >>>>>>>>>>> java version "1.8.0_152" >>>>>>>>>>> Java(TM) SE Runtime Environment (build 1.8.0_152-b16) >>>>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode) >>>>>>>>>>> >>>>>>>>>>> *Test case errors* >>>>>>>>>>> >>>>>>>>>>> I won’t post all of them, I get a good number of errors: >>>>>>>>>>> >>>>>>>>>>> ================================ >>>>>>>>>>> [ERROR] Tests run: 64, Failures: 0, Errors: 16, Skipped: 0, Time >>>>>>>> elapsed: 9.21 s <<< FAILURE! - in >>>>>> org.apache.zookeeper.util.PemReaderTest >>>>>>>>>>> [ERROR] >>>>>>>> >>>>>> >>>>> testLoadCertificateFromKeyStore[1](org.apache.zookeeper.util.PemReaderTest) >>>>>>>> Time elapsed: 1.593 s <<< ERROR! >>>>>>>>>>> java.io.IOException: >>>>>>>> org.bouncycastle.operator.OperatorCreationException: Illegal key size >>>>> or >>>>>>>> default parameters >>>>>>>>>>> at >>>>>>>> >>>>>> >>>>> org.apache.zookeeper.util.PemReaderTest.testLoadCertificateFromKeyStore(PemReaderTest.java:125) >>>>>>>>>>> Caused by: org.bouncycastle.operator.OperatorCreationException: >>>>>>>> Illegal key size or default parameters >>>>>>>>>>> at >>>>>>>> >>>>>> >>>>> org.apache.zookeeper.util.PemReaderTest.testLoadCertificateFromKeyStore(PemReaderTest.java:125) >>>>>>>>>>> Caused by: java.security.InvalidKeyException: Illegal key size or >>>>>>>> default parameters >>>>>>>>>>> at >>>>>>>> >>>>>> >>>>> org.apache.zookeeper.util.PemReaderTest.testLoadCertificateFromKeyStore(PemReaderTest.java:125) >>>>>>>>>>> >>>>>>>>>>> [ERROR] >>>>>>>> >>>>>> >>>>> testLoadEncryptedPrivateKeyFromKeyStoreWithWrongPassword[1](org.apache.zookeeper.util.PemReaderTest) >>>>>>>> Time elapsed: 0.004 s <<< ERROR! >>>>>>>>>>> java.lang.Exception: Unexpected exception, >>>>>>>> expected<java.security.GeneralSecurityException> but >>>>>>>> was<java.io.IOException> >>>>>>>>>>> at >>>>>>>> >>>>>> >>>>> org.apache.zookeeper.util.PemReaderTest.testLoadEncryptedPrivateKeyFromKeyStoreWithWrongPassword(PemReaderTest.java:93) >>>>>>>>>>> Caused by: org.bouncycastle.operator.OperatorCreationException: >>>>>>>> Illegal key size or default parameters >>>>>>>>>>> at >>>>>>>> >>>>>> >>>>> org.apache.zookeeper.util.PemReaderTest.testLoadEncryptedPrivateKeyFromKeyStoreWithWrongPassword(PemReaderTest.java:93) >>>>>>>>>>> Caused by: java.security.InvalidKeyException: Illegal key size or >>>>>>>> default parameters >>>>>>>>>>> at >>>>>>>> >>>>>> >>>>> org.apache.zookeeper.util.PemReaderTest.testLoadEncryptedPrivateKeyFromKeyStoreWithWrongPassword(PemReaderTest.java:93) >>>>>>>>>>> ... >>>>>>>>>>> ================================ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Crypto policy* >>>>>>>>>>> If I uncomment this configuration option: >>>>>>>>>>> >>>>>>>>>>> # Please see the JCA documentation for additional information on >>>>>> these >>>>>>>>>>> # files and formats. >>>>>>>>>>> # crypto.policy=unlimited >>>>>>>>>>> >>>>>>>>>>> in: >>>>>>>>>>> >>>>>>>>>>> $JAVA_HOME/jre/lib/security/java.security >>>>>>>>>>> >>>>>>>>>>> then it all works and I get no error at all. This option controls >>>>>>>> cryptographic strengths according to the documentation, and is present >>>>>>>> because of crypto regulations in different countries. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> -Flavio >>>>>>>>>>> >>>>>>>>>>>> On 27 Apr 2019, at 15:52, Enrico Olivelli <[email protected]> >>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Il sab 27 apr 2019, 14:18 Flavio Junqueira <[email protected]> ha >>>>>>>> scritto: >>>>>>>>>>>> >>>>>>>>>>>>> I have a clarification question about the RC. To build the RC, I >>>>>>>> had to >>>>>>>>>>>>> enable crypto.policy unlimited in the jre (I'm using build >>>>>>>> 1.8.0_152-b16). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Flavio >>>>>>>>>>>> What do you mean with 'build' ? >>>>>>>>>>>> Make tests pass? >>>>>>>>>>>> AFAIK we are not using tweaked jdks in CI builds, so in theory >>>>> there >>>>>>>> is no >>>>>>>>>>>> need. >>>>>>>>>>>> >>>>>>>>>>>> Can you please share your error? >>>>>>>>>>>> >>>>>>>>>>>> Enrico >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I'm wondering if this is going to be an issue for some users as >>>>> this >>>>>>>> option >>>>>>>>>>>>> is related to import/export regulation. Has anyone looked into it >>>>>>>> and could >>>>>>>>>>>>> clarify it to me, please? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> -Flavio >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On 25 Apr 2019, at 15:10, Andor Molnar <[email protected]> >>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is the first stable release of 3.5 branch: 3.5.5. It >>>>> resolves >>>>>>>> 117 >>>>>>>>>>>>> issues, including Maven migration, Quorum TLS, TTL nodes and lots >>>>>>>> of other >>>>>>>>>>>>> performance and stability improvements. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The full release notes is available at: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>> >>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12343268 >>>>>>>>>>>>>> >>>>>>>>>>>>>> *** Please download, test and vote by May 3rd 2019, 23:59 UTC+0. >>>>>>>> *** >>>>>>>>>>>>>> >>>>>>>>>>>>>> Source files: >>>>>>>>>>>>>> >>>>>>>> https://dist.apache.org/repos/dist/dev/zookeeper/zookeeper-3.5.5-rc5/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> Maven staging repos: >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>> >>>>> https://repository.apache.org/content/groups/staging/org/apache/zookeeper/parent/3.5.5/ >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>> >>>>> https://repository.apache.org/content/groups/staging/org/apache/zookeeper/zookeeper-jute/3.5.5/ >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>> >>>>> https://repository.apache.org/content/groups/staging/org/apache/zookeeper/zookeeper/3.5.5/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> The release candidate tag in git to be voted upon: >>>>>>>> release-3.5.5-rc5 >>>>>>>>>>>>>> >>>>>>>>>>>>>> ZooKeeper's KEYS file containing PGP keys we use to sign the >>>>>>>> release: >>>>>>>>>>>>>> http://www.apache.org/dist/zookeeper/KEYS >>>>>>>>>>>>>> >>>>>>>>>>>>>> Should we release this candidate? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>> >>> >> >
