> On Dec. 28, 2015, 7:12 p.m., Bill Farner wrote: > > > > John Sirois wrote: > I found that by upgrading all the way to 3.4.7 the krb issues go away in > the unit tests (`./gradlew -Pq build`), although e2e krb fails (digging). > Presuming I can find a fix for that (we'll need to upgrade someday) you get a > natural state tracking variable back by using a nullable zooKeeperServer > since in 3.4.7 you cannot re-use the server after a shutdown (see failure > trace below), so you can just null it out in `shutdownNetwork` and (re)create > it in `startNetwork`. > ``` > java.lang.IllegalThreadStateException > at java.lang.Thread.start(Thread.java:705) > at > org.apache.zookeeper.server.ZooKeeperServer.startSessionTracker(ZooKeeperServer.java:455) > at > org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:413) > at > org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:119) > at > org.apache.aurora.common.zookeeper.testing.ZooKeeperTestServer.startNetwork(ZooKeeperTestServer.java:90) > at > org.apache.aurora.common.zookeeper.testing.ZooKeeperTestServer.restartNetwork(ZooKeeperTestServer.java:111) > at > org.apache.aurora.common.zookeeper.testing.BaseZooKeeperTest.restartNetwork(BaseZooKeeperTest.java:71) > at > org.apache.aurora.common.zookeeper.GroupTest.testJoinsAndWatchesSurviveDisconnect(GroupTest.java:154) > ```
Interesting that you didn't observe the same unit test failure in 3.4.7, as that's where i started. I walked backwards and noticed that it first repeatably occurs with 3.4.3. There was a related patch in 3.4.3 that i assumed was the culprit: https://github.com/apache/zookeeper/commit/51a1bf6 Here's the error i encounter with >=3.4.3 (100% repeatably): ``` com.google.inject.CreationException: Guice creation errors: 1) An exception was caught and reported. Message: No LoginModules configured for org.apache.aurora.scheduler.http.api.security.Kerberos5ShiroRealmModule at org.apache.aurora.scheduler.http.api.security.Kerberos5ShiroRealmModule.configure(Kerberos5ShiroRealmModule.java:175) 1 error at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) at com.google.inject.Guice.createInjector(Guice.java:95) at com.google.inject.Guice.createInjector(Guice.java:72) at com.google.inject.Guice.createInjector(Guice.java:62) at org.apache.aurora.scheduler.http.api.security.Kerberos5ShiroRealmModuleTest.testConfigure(Kerberos5ShiroRealmModuleTest.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54) at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: javax.security.auth.login.LoginException: No LoginModules configured for org.apache.aurora.scheduler.http.api.security.Kerberos5ShiroRealmModule at javax.security.auth.login.LoginContext.init(LoginContext.java:264) at javax.security.auth.login.LoginContext.<init>(LoginContext.java:348) at org.apache.aurora.scheduler.http.api.security.Kerberos5ShiroRealmModule.configure(Kerberos5ShiroRealmModule.java:159) at com.google.inject.AbstractModule.configure(AbstractModule.java:59) at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223) at com.google.inject.spi.Elements.getElements(Elements.java:101) at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103) ... 49 more ``` > On Dec. 28, 2015, 7:12 p.m., Bill Farner wrote: > > commons/src/main/java/org/apache/aurora/common/zookeeper/testing/ZooKeeperTestServer.java, > > line 105 > > <https://reviews.apache.org/r/41762/diff/1/?file=1177135#file1177135line105> > > > > Background - `NIOServerCnxn.Factory` extended `Thread`, > > `NIOServerCnxnFactory` is a `Runnable`, so we no longer have `isAlive()`. > > The Thread is still there, but privately managed. If we want to maintain > > this precondition and the gate below, we could do our own state tracking. > > However, i'm not sure it's worth it. > > John Sirois wrote: > Its easy enough to just null out the existing connectionFactory after > shutdown and use its null status as the check, but I'm not tied to this - I > just noticed when digging a bit on krb as noted above. Done. - Bill ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41762/#review112116 ----------------------------------------------------------- On Dec. 28, 2015, 7:07 p.m., Bill Farner wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41762/ > ----------------------------------------------------------- > > (Updated Dec. 28, 2015, 7:07 p.m.) > > > Review request for Aurora, John Sirois and Zameer Manji. > > > Repository: aurora > > > Description > ------- > > Primary motivation here is to get past 3.4.0, which changed how logging works > in ZK (primarily via slf4j). This will make it easier to have a cohesive > logging story. > > There was some fallout - I did not update to latest - 3.4.7. I encountered > issues with a kerberos-related test (see comment in `build.gradle`) with > >=3.4.3. I spent a few minutes looking for a solution, but decided to stop > short so as to return to the work i was really trying to tackle. > > > Diffs > ----- > > build.gradle d32aea5e4c888cb215af7bedd5e9c1a8f41308c8 > > commons/src/main/java/org/apache/aurora/common/zookeeper/testing/ZooKeeperTestServer.java > 03d47c37f67dee64eca2107ab1510c7efcdd0931 > > Diff: https://reviews.apache.org/r/41762/diff/ > > > Testing > ------- > > end-to-end tests > > > Thanks, > > Bill Farner > >
