Hello all,
Sometimes we get very strange ignite cluster crush during execution of tests
for our system.
2019-07-09 11:02:59,710 [main] ERROR com.imperva.ddc.core.Driver:176 - Ldap
Connection to nodelegateddomen.local failed
<Full stack trace at the and of letter>
2019-07-09 11:02:59,715 [main] ERROR com.imperva.ddc.core.Driver:116 - Test
connection has failed. Results: Connection to host
nodelegateddomen.dev002.local has failed. Reason:
com.imperva.ddc.core.exceptions.InvalidConnectionException:
org.apache.directory.ldap.client.api.exception.InvalidConnectionException:
Cannot connect to the server: Hostname 'nodelegateddomen.local' could not be
resolved.
2019-07-09 11:03:05,482 [db-checkpoint-thread-#10414] ERROR :134 - Critical
system error detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to write
checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=2032270, len=31143],
cpTs=1562670185474, cpId=1bab48e6-f29c-4fb3-acd0-385783244ad9, type=START]]]
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=2032270,
len=31143], cpTs=1562670185474, cpId=1bab48e6-f29c-4fb3-acd0-385783244ad9,
type=START]
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:2853)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3841)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3279)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3178)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.7.0.jar:2.7.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: java.nio.file.NoSuchFileException:
/buildir/testdir/ignitewd/db/storage/node00-98797cd6-5f62-4e0f-8bbd-08163bb111ae/cp/1562670185474-1bab48e6-f29c-4fb3-acd0-385783244ad9-START.bin.tmp
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
~[?:1.8.0_212]
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
~[?:1.8.0_212]
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
~[?:1.8.0_212]
at
sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:196)
~[?:1.8.0_212]
at
java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:248)
~[?:1.8.0_212]
at
java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:301)
~[?:1.8.0_212]
at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:57)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:53)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:2836)
~[ignite-core-2.7.0.jar:2.7.0]
... 5 more
I don’t know way to reproduce =(, But we got this crash twice during last
month, on the same test. And this test don’t make any payload for cluster, this
test is to check how our authorization subsystem handle case when
ActiveDirectory is not available. No any manipulation with grid.
I have no any idea how it works, but both cluster crush was on the same place.
We use Imperva as AD client
<artifactId>ddc-core</artifactId>
<groupId>com.imperva.ddc</groupId>
<version>7.3.3.0.0.0</version>
We use embedded Ignite with enabled percistence.
Thanks for any help.
Andrey.
2019-07-09 11:02:59,710 [main] ERROR com.imperva.ddc.core.Driver:176 - Ldap
Connection to nodelegateddomen.local failed
org.apache.directory.ldap.client.api.exception.InvalidConnectionException:
Cannot connect to the server: Hostname 'nodelegateddomen.local' could not be
resolved.
at
org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:758)
~[api-all-1.0.3.jar:1.0.3]
at
org.apache.directory.ldap.client.api.LdapNetworkConnection.bindAsync(LdapNetworkConnection.java:1368)
~[api-all-1.0.3.jar:1.0.3]
at
org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1293)
~[api-all-1.0.3.jar:1.0.3]
at com.imperva.ddc.core.Driver.getLdapConnection(Driver.java:148)
[ddc-core-7.3.3.0.0.0.jar:?]
at com.imperva.ddc.core.Driver.createConnection(Driver.java:81)
[ddc-core-7.3.3.0.0.0.jar:?]
at com.imperva.ddc.core.Driver.doConnect(Driver.java:68)
[ddc-core-7.3.3.0.0.0.jar:?]
at com.imperva.ddc.core.Driver.connect(Driver.java:44)
[ddc-core-7.3.3.0.0.0.jar:?]
at
com.imperva.ddc.core.DriverRobustDecorator.retry(DriverRobustDecorator.java:50)
[ddc-core-7.3.3.0.0.0.jar:?]
at
com.imperva.ddc.core.DriverRobustDecorator.connect(DriverRobustDecorator.java:29)
[ddc-core-7.3.3.0.0.0.jar:?]
at
com.imperva.ddc.core.DriverHostResolverDecorator.connect(DriverHostResolverDecorator.java:38)
[ddc-core-7.3.3.0.0.0.jar:?]
at
com.imperva.ddc.core.QueryRequestNonPagedExecutor.execute(QueryRequestNonPagedExecutor.java:45)
[ddc-core-7.3.3.0.0.0.jar:?]
at com.imperva.ddc.core.Executor.execute(Executor.java:26)
[ddc-core-7.3.3.0.0.0.jar:?]
at com.imperva.ddc.core.Connector.execute(Connector.java:106)
[ddc-core-7.3.3.0.0.0.jar:?]
at
com.example.utils.activedirectory.ActiveDirectoryApi.resolveDistinguishedName(ActiveDirectoryApi.java:183)
[classes/:?]
at
com.example.utils.activedirectory.ActiveDirectoryApi.isEnabled(ActiveDirectoryApi.java:129)
[classes/:?]
at
com.example.utils.activedirectory.ActiveDirectoryApiTest.testGetGroups(ActiveDirectoryApiTest.java:160)
[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_212]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_212]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
[junit-4.12.jar:4.12]
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
[junit-4.12.jar:4.12]
at
org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
[junit-4.12.jar:4.12]
at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
[junit-4.12.jar:4.12]
at
org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
[junit-4.12.jar:4.12]
at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
[spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
com.example.CustomSpringJUnit4Runner.run(CustomSpringJUnit4Runner.java:27)
[test-classes/:?]
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
[surefire-junit4-2.22.1.jar:2.22.1]
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
[surefire-junit4-2.22.1.jar:2.22.1]
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
[surefire-junit4-2.22.1.jar:2.22.1]
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
[surefire-junit4-2.22.1.jar:2.22.1]
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
[surefire-booter-2.22.1.jar:2.22.1]
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
[surefire-booter-2.22.1.jar:2.22.1]
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
[surefire-booter-2.22.1.jar:2.22.1]
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[surefire-booter-2.22.1.jar:2.22.1]
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101) ~[?:1.8.0_212]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
~[?:1.8.0_212]
at
org.apache.mina.transport.socket.nio.NioSocketConnector.connect(NioSocketConnector.java:187)
~[mina-core-2.0.21.jar:?]
at
org.apache.mina.transport.socket.nio.NioSocketConnector.connect(NioSocketConnector.java:47)
~[mina-core-2.0.21.jar:?]
at
org.apache.mina.core.polling.AbstractPollingIoConnector.connect0(AbstractPollingIoConnector.java:370)
~[mina-core-2.0.21.jar:?]
at
org.apache.mina.core.service.AbstractIoConnector.connect(AbstractIoConnector.java:322)
~[mina-core-2.0.21.jar:?]
at
org.apache.mina.core.service.AbstractIoConnector.connect(AbstractIoConnector.java:206)
~[mina-core-2.0.21.jar:?]
at
org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:665)
~[api-all-1.0.3.jar:1.0.3]
... 48 more
