Hello! I can't comment on your LDAP situation, but you can see errors like this one in test if you clear Ignite work dir before node is completely stopped.
See for example https://issues.apache.org/jira/browse/IGNITE-8797 Regards, -- Ilya Kasnacheev вт, 9 июл. 2019 г. в 15:04, Andrey Davydov <[email protected]>: > > > Hello all, > > > > Sometimes we get very strange ignite cluster crush during execution of > tests for our system. > > > > 2019-07-09 11:02:59,710 [main] ERROR com.imperva.ddc.core.Driver:176 - > Ldap Connection to nodelegateddomen.local failed > > <Full stack trace at the and of letter> > > 2019-07-09 11:02:59,715 [main] ERROR com.imperva.ddc.core.Driver:116 - > Test connection has failed. Results: Connection to host > nodelegateddomen.dev002.local has failed. Reason: > com.imperva.ddc.core.exceptions.InvalidConnectionException: > org.apache.directory.ldap.client.api.exception.InvalidConnectionException: > Cannot connect to the server: Hostname 'nodelegateddomen.local' could not > be resolved. > > 2019-07-09 11:03:05,482 [db-checkpoint-thread-#10414] ERROR :134 - > Critical system error detected. Will be handled accordingly to configured > handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler > [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext > [type=CRITICAL_ERROR, err=class > o.a.i.i.processors.cache.persistence.StorageException: Failed to write > checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=2032270, len=31143], > cpTs=1562670185474, cpId=1bab48e6-f29c-4fb3-acd0-385783244ad9, type=START]]] > > org.apache.ignite.internal.processors.cache.persistence.StorageException: > Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, > fileOff=2032270, len=31143], cpTs=1562670185474, > cpId=1bab48e6-f29c-4fb3-acd0-385783244ad9, type=START] > > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:2853) > ~[ignite-core-2.7.0.jar:2.7.0] > > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3841) > ~[ignite-core-2.7.0.jar:2.7.0] > > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3279) > [ignite-core-2.7.0.jar:2.7.0] > > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3178) > [ignite-core-2.7.0.jar:2.7.0] > > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > [ignite-core-2.7.0.jar:2.7.0] > > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] > > Caused by: java.nio.file.NoSuchFileException: > /buildir/testdir/ignitewd/db/storage/node00-98797cd6-5f62-4e0f-8bbd-08163bb111ae/cp/1562670185474-1bab48e6-f29c-4fb3-acd0-385783244ad9-START.bin.tmp > > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > ~[?:1.8.0_212] > > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > ~[?:1.8.0_212] > > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > ~[?:1.8.0_212] > > at > sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:196) > ~[?:1.8.0_212] > > at > java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:248) > ~[?:1.8.0_212] > > at > java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:301) > ~[?:1.8.0_212] > > at > org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:57) > ~[ignite-core-2.7.0.jar:2.7.0] > > at > org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:53) > ~[ignite-core-2.7.0.jar:2.7.0] > > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:2836) > ~[ignite-core-2.7.0.jar:2.7.0] > > ... 5 more > > > > I don’t know way to reproduce =(, But we got this crash twice during last > month, on the same test. And this test don’t make any payload for cluster, > this test is to check how our authorization subsystem handle case when > ActiveDirectory is not available. No any manipulation with grid. > > > > I have no any idea how it works, but both cluster crush was on the same > place. > > > > We use Imperva as AD client > > > > <artifactId>ddc-core</artifactId> > > <groupId>com.imperva.ddc</groupId> > > <version>7.3.3.0.0.0</version> > > > > We use embedded Ignite with enabled percistence. > > > > Thanks for any help. > > > > Andrey. > > > > 2019-07-09 11:02:59,710 [main] ERROR com.imperva.ddc.core.Driver:176 - > Ldap Connection to nodelegateddomen.local failed > > org.apache.directory.ldap.client.api.exception.InvalidConnectionException: > Cannot connect to the server: Hostname 'nodelegateddomen.local' could not > be resolved. > > at > org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:758) > ~[api-all-1.0.3.jar:1.0.3] > > at > org.apache.directory.ldap.client.api.LdapNetworkConnection.bindAsync(LdapNetworkConnection.java:1368) > ~[api-all-1.0.3.jar:1.0.3] > > at > org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1293) > ~[api-all-1.0.3.jar:1.0.3] > > at > com.imperva.ddc.core.Driver.getLdapConnection(Driver.java:148) > [ddc-core-7.3.3.0.0.0.jar:?] > > at > com.imperva.ddc.core.Driver.createConnection(Driver.java:81) > [ddc-core-7.3.3.0.0.0.jar:?] > > at com.imperva.ddc.core.Driver.doConnect(Driver.java:68) > [ddc-core-7.3.3.0.0.0.jar:?] > > at com.imperva.ddc.core.Driver.connect(Driver.java:44) > [ddc-core-7.3.3.0.0.0.jar:?] > > at > com.imperva.ddc.core.DriverRobustDecorator.retry(DriverRobustDecorator.java:50) > [ddc-core-7.3.3.0.0.0.jar:?] > > at > com.imperva.ddc.core.DriverRobustDecorator.connect(DriverRobustDecorator.java:29) > [ddc-core-7.3.3.0.0.0.jar:?] > > at > com.imperva.ddc.core.DriverHostResolverDecorator.connect(DriverHostResolverDecorator.java:38) > [ddc-core-7.3.3.0.0.0.jar:?] > > at > com.imperva.ddc.core.QueryRequestNonPagedExecutor.execute(QueryRequestNonPagedExecutor.java:45) > [ddc-core-7.3.3.0.0.0.jar:?] > > at com.imperva.ddc.core.Executor.execute(Executor.java:26) > [ddc-core-7.3.3.0.0.0.jar:?] > > at com.imperva.ddc.core.Connector.execute(Connector.java:106) > [ddc-core-7.3.3.0.0.0.jar:?] > > at > com.example.utils.activedirectory.ActiveDirectoryApi.resolveDistinguishedName(ActiveDirectoryApi.java:183) > [classes/:?] > > at > com.example.utils.activedirectory.ActiveDirectoryApi.isEnabled(ActiveDirectoryApi.java:129) > [classes/:?] > > at > com.example.utils.activedirectory.ActiveDirectoryApiTest.testGetGroups(ActiveDirectoryApiTest.java:160) > [test-classes/:?] > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[?:1.8.0_212] > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_212] > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_212] > > at java.lang.reflect.Method.invoke(Method.java:498) > ~[?:1.8.0_212] > > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > [junit-4.12.jar:4.12] > > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > [junit-4.12.jar:4.12] > > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > [junit-4.12.jar:4.12] > > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > [junit-4.12.jar:4.12] > > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > [junit-4.12.jar:4.12] > > at > org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) > [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE] > > at > org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) > [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE] > > at > org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) > [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE] > > at > org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > [junit-4.12.jar:4.12] > > at > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252) > [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE] > > at > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94) > [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE] > > at > org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > [junit-4.12.jar:4.12] > > at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > [junit-4.12.jar:4.12] > > at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > [junit-4.12.jar:4.12] > > at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > [junit-4.12.jar:4.12] > > at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > [junit-4.12.jar:4.12] > > at > org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) > [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE] > > at > org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) > [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE] > > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > [junit-4.12.jar:4.12] > > at > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191) > [spring-test-4.3.18.RELEASE.jar:4.3.18.RELEASE] > > at > com.example.CustomSpringJUnit4Runner.run(CustomSpringJUnit4Runner.java:27) > [test-classes/:?] > > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > [surefire-junit4-2.22.1.jar:2.22.1] > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > [surefire-junit4-2.22.1.jar:2.22.1] > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > [surefire-junit4-2.22.1.jar:2.22.1] > > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > [surefire-junit4-2.22.1.jar:2.22.1] > > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) > [surefire-booter-2.22.1.jar:2.22.1] > > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) > [surefire-booter-2.22.1.jar:2.22.1] > > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) > [surefire-booter-2.22.1.jar:2.22.1] > > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) > [surefire-booter-2.22.1.jar:2.22.1] > > Caused by: java.nio.channels.UnresolvedAddressException > > at sun.nio.ch.Net.checkAddress(Net.java:101) ~[?:1.8.0_212] > > at > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) > ~[?:1.8.0_212] > > at > org.apache.mina.transport.socket.nio.NioSocketConnector.connect(NioSocketConnector.java:187) > ~[mina-core-2.0.21.jar:?] > > at > org.apache.mina.transport.socket.nio.NioSocketConnector.connect(NioSocketConnector.java:47) > ~[mina-core-2.0.21.jar:?] > > at > org.apache.mina.core.polling.AbstractPollingIoConnector.connect0(AbstractPollingIoConnector.java:370) > ~[mina-core-2.0.21.jar:?] > > at > org.apache.mina.core.service.AbstractIoConnector.connect(AbstractIoConnector.java:322) > ~[mina-core-2.0.21.jar:?] > > at > org.apache.mina.core.service.AbstractIoConnector.connect(AbstractIoConnector.java:206) > ~[mina-core-2.0.21.jar:?] > > at > org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:665) > ~[api-all-1.0.3.jar:1.0.3] > > ... 48 more >
