[
https://issues.apache.org/jira/browse/GEODE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095144#comment-16095144
]
Kirk Lund edited comment on GEODE-3251 at 7/20/17 6:50 PM:
-----------------------------------------------------------
Looks like the problem is caused by several things:
1) The MemberStarterRule is swizzling the user.dir System property. Jinmei has
a pending change to not change the user.dir. In general, we should avoid
changing that System property at runtime and only fork JVMs for tests that need
processes to run in a specific user.dir.
2) FastClasspathScanner throws NoSuchFileException because
System.getProperty("user.dir") returns null. It's null because
MemberStarterRule set it to null.
3) Caller(s) of ManagementAdapter#handleCacheCreation(InternalCache) catches
the NoSuchFileException and logs it. The ManagementAgent then closes itself in
the finally block of #handleCacheCreation. Because the test has a log-file in a
TemporaryFolder, it's really easy to miss this stack trace.
4) When MBeanServerConnectionRule tries to connect to JMX RMI, it fails because
JMX RMI was closed in #3 above. This throws IOException "Failed to retrieve
RMIServer stub" as shown in the ticket description.
{noformat}
java.lang.RuntimeException: java.lang.RuntimeException:
java.nio.file.NoSuchFileException:
/var/folders/28/m__9dv1906n60kmz7t71wm680000gn/T/junit7797861898628691451
at
io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1115)
at
io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1143)
at
io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1166)
at
org.apache.geode.management.internal.cli.util.ClasspathScanLoadHelper.scanPackageForClassesImplementing(ClasspathScanLoadHelper.java:35)
at
org.apache.geode.management.internal.cli.CommandManager.loadGeodeCommands(CommandManager.java:223)
at
org.apache.geode.management.internal.cli.CommandManager.loadCommands(CommandManager.java:176)
at
org.apache.geode.management.internal.cli.CommandManager.<init>(CommandManager.java:83)
at
org.apache.geode.management.internal.cli.remote.CommandProcessor.<init>(CommandProcessor.java:61)
at
org.apache.geode.management.internal.cli.remote.MemberCommandService.<init>(MemberCommandService.java:36)
at
org.apache.geode.management.cli.CommandService.createLocalCommandService(CommandService.java:128)
at
org.apache.geode.management.internal.beans.MemberMBeanBridge.<init>(MemberMBeanBridge.java:348)
at
org.apache.geode.management.internal.beans.ManagementAdapter.handleCacheCreation(ManagementAdapter.java:142)
at
org.apache.geode.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:106)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2198)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:585)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1198)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:763)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:750)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:175)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:222)
at
org.apache.geode.test.dunit.rules.ServerStarterRule.startServer(ServerStarterRule.java:175)
at
org.apache.geode.test.dunit.rules.ServerStarterRule.before(ServerStarterRule.java:91)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.RuntimeException: java.nio.file.NoSuchFileException:
/var/folders/28/m__9dv1906n60kmz7t71wm680000gn/T/junit7797861898628691451
at
io.github.lukehutch.fastclasspathscanner.scanner.Scanner.call(Scanner.java:217)
at
io.github.lukehutch.fastclasspathscanner.scanner.Scanner.call(Scanner.java:61)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.NoSuchFileException:
/var/folders/28/m__9dv1906n60kmz7t71wm680000gn/T/junit7797861898628691451
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixPath.toRealPath(UnixPath.java:876)
at
io.github.lukehutch.fastclasspathscanner.scanner.Scanner.call(Scanner.java:215)
... 5 more
java.io.IOException: Failed to retrieve RMIServer stub:
javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host: 10.118.33.235; nested
exception is:
java.net.ConnectException: Connection refused]
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at
javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at
org.apache.geode.test.dunit.rules.MBeanServerConnectionRule.lambda$connect$0(MBeanServerConnectionRule.java:166)
at
org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:100)
at
org.awaitility.core.ConditionAwaiter$ConditionPoller.run(ConditionAwaiter.java:215)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host: 10.118.33.235; nested
exception is:
java.net.ConnectException: Connection refused]
at
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122)
at
com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at
javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1957)
at
javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1924)
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
... 9 more
Caused by: java.rmi.ConnectException: Connection refused to host:
10.118.33.235; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
... 14 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 19 more
{noformat}
was (Author: klund):
Looks like the problem is caused by several things:
1) The MemberStarterRule is swizzling the user.dir System property. Jinmei has
a pending change to not change the user.dir. In general, we should avoid
changing that System property at runtime and only fork JVMs for tests that need
processes to run in a specific user.dir.
2) FastClasspathScanner throws NoSuchFileException because
System.getProperty("user.dir") returns null. It's null because
MemberStarterRule set it to null.
3) The call stack for ManagementAdapter#handleCacheCreation(InternalCache)
catches the NoSuchFileException and swallows it without logging it.. The
ManagementAgent then closes itself in the finally block of #handleCacheCreation.
4) When MBeanServerConnectionRule tries to connect to JMX RMI, it fails because
JMX RMI was closed in #3 above. This throws IOException "Failed to retrieve
RMIServer stub".
{noformat}
java.lang.RuntimeException: java.lang.RuntimeException:
java.nio.file.NoSuchFileException:
/var/folders/28/m__9dv1906n60kmz7t71wm680000gn/T/junit7797861898628691451
at
io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1115)
at
io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1143)
at
io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1166)
at
org.apache.geode.management.internal.cli.util.ClasspathScanLoadHelper.scanPackageForClassesImplementing(ClasspathScanLoadHelper.java:35)
at
org.apache.geode.management.internal.cli.CommandManager.loadGeodeCommands(CommandManager.java:223)
at
org.apache.geode.management.internal.cli.CommandManager.loadCommands(CommandManager.java:176)
at
org.apache.geode.management.internal.cli.CommandManager.<init>(CommandManager.java:83)
at
org.apache.geode.management.internal.cli.remote.CommandProcessor.<init>(CommandProcessor.java:61)
at
org.apache.geode.management.internal.cli.remote.MemberCommandService.<init>(MemberCommandService.java:36)
at
org.apache.geode.management.cli.CommandService.createLocalCommandService(CommandService.java:128)
at
org.apache.geode.management.internal.beans.MemberMBeanBridge.<init>(MemberMBeanBridge.java:348)
at
org.apache.geode.management.internal.beans.ManagementAdapter.handleCacheCreation(ManagementAdapter.java:142)
at
org.apache.geode.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:106)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2198)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:585)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1198)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:763)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:750)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:175)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:222)
at
org.apache.geode.test.dunit.rules.ServerStarterRule.startServer(ServerStarterRule.java:175)
at
org.apache.geode.test.dunit.rules.ServerStarterRule.before(ServerStarterRule.java:91)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.RuntimeException: java.nio.file.NoSuchFileException:
/var/folders/28/m__9dv1906n60kmz7t71wm680000gn/T/junit7797861898628691451
at
io.github.lukehutch.fastclasspathscanner.scanner.Scanner.call(Scanner.java:217)
at
io.github.lukehutch.fastclasspathscanner.scanner.Scanner.call(Scanner.java:61)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.NoSuchFileException:
/var/folders/28/m__9dv1906n60kmz7t71wm680000gn/T/junit7797861898628691451
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixPath.toRealPath(UnixPath.java:876)
at
io.github.lukehutch.fastclasspathscanner.scanner.Scanner.call(Scanner.java:215)
... 5 more
java.io.IOException: Failed to retrieve RMIServer stub:
javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host: 10.118.33.235; nested
exception is:
java.net.ConnectException: Connection refused]
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at
javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at
org.apache.geode.test.dunit.rules.MBeanServerConnectionRule.lambda$connect$0(MBeanServerConnectionRule.java:166)
at
org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:100)
at
org.awaitility.core.ConditionAwaiter$ConditionPoller.run(ConditionAwaiter.java:215)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host: 10.118.33.235; nested
exception is:
java.net.ConnectException: Connection refused]
at
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122)
at
com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at
javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1957)
at
javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1924)
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
... 9 more
Caused by: java.rmi.ConnectException: Connection refused to host:
10.118.33.235; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
... 14 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 19 more
{noformat}
> DiskStoreMXBeanSecurityJUnitTest fails with IOException: Failed to retrieve
> RMIServer stub
> ------------------------------------------------------------------------------------------
>
> Key: GEODE-3251
> URL: https://issues.apache.org/jira/browse/GEODE-3251
> Project: Geode
> Issue Type: Bug
> Components: jmx, tests
> Reporter: Kirk Lund
> Assignee: Kirk Lund
>
> I'm not sure if the JMX RMI startup is asynchronous in some way or not, but
> at some point any of the JMX tests seem to be prone to hitting this failure:
> "java.io.IOException: Failed to retrieve RMIServer stub"
> {noformat}
> :geode-core:integrationTest
> org.apache.geode.management.internal.security.DiskStoreMXBeanSecurityJUnitTest
> > testNoAccess FAILED
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host: 10.254.0.238; nested
> exception is:
> java.net.ConnectException: Connection refused (Connection refused)]
> Caused by:
> javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host: 10.254.0.238; nested
> exception is:
> java.net.ConnectException: Connection refused (Connection refused)]
> Caused by:
> java.rmi.ConnectException: Connection refused to host:
> 10.254.0.238; nested exception is:
> java.net.ConnectException: Connection refused (Connection
> refused)
> Caused by:
> java.net.ConnectException: Connection refused (Connection
> refused)
> 3598 tests completed, 1 failed, 163 skipped
> :geode-core:integrationTest FAILED
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
