markjens opened a new issue #2375:
URL: https://github.com/apache/accumulo/issues/2375
**Test name(s)**
- ExternalCompaction_2_IT
- ExternalCompaction_3_IT
**Describe the failure observed**
ExternalCompaction_2_IT and ExternalCompaction_3_IT hang and timeout on
Linux ARM64.
These two tests always timeout on my test system, no matter how much time I
give them (`-Dtimeout.factor=...`)
Thread dumps of all processes could be found at
https://gist.github.com/markjens/326b681c3f8a9c1b6400f55847fe7716. I've taken
several dumps with ~5secs interval between them.
I do not see anything suspicious in the stack traces. That's why I need your
help to review them and hopefully find what the problem is!
**Testing Environment:**
- Version of this project: 2.1.0-SNAPSHOT
- First commit known to fail (or current commit): current `main` branch
- OS: Ubuntu 20.04.3
- Java: 11.0.11
- Maven: 3.8.2
- Other environment details: Linux ARM64
**What have you tried already?**
- mvn clean install -DskipTests
- mvn verify (ExternalCompaction_2_IT and ExternalCompaction_3_IT timed out)
- mvn clean verify -Dit.test=ExternalCompaction_3_IT -Dtimeout.factor=3 -o
-Pfast-build -N
<details><summary>Tests output</summary>
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.accumulo.test.compaction.ExternalCompaction_3_IT
[ERROR] Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed:
3,600.677 s <<< FAILURE! - in
org.apache.accumulo.test.compaction.ExternalCompaction_3_IT
[ERROR]
org.apache.accumulo.test.compaction.ExternalCompaction_3_IT.testMergeCancelsExternalCompaction
Time elapsed: 1800.016 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 1800
seconds
at [email protected]/java.lang.Thread.sleep(Native Method)
at
app//org.apache.accumulo.fate.util.UtilWaitThread.sleep(UtilWaitThread.java:33)
at
app//org.apache.accumulo.test.compaction.ExternalCompactionTestUtils.confirmCompactionCompleted(ExternalCompactionTestUtils.java:329)
at
app//org.apache.accumulo.test.compaction.ExternalCompaction_3_IT.testMergeCancelsExternalCompaction(ExternalCompaction_3_IT.java:113)
at
[email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
[email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
[email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
at
app//org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
app//org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
app//org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
app//org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
app//org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at
[email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at [email protected]/java.lang.Thread.run(Thread.java:833)
[ERROR]
org.apache.accumulo.test.compaction.ExternalCompaction_3_IT.testMergeCancelsExternalCompaction
Time elapsed: 1800.017 s <<< ERROR!
java.lang.Exception: Appears to be stuck in thread Time-limited
test-SendThread(localhost:35839)
at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
at
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
at
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
at
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:347)
at
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
[ERROR]
org.apache.accumulo.test.compaction.ExternalCompaction_3_IT.testCoordinatorRestartsDuringCompaction
Time elapsed: 1800.001 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 1800
seconds
at [email protected]/java.lang.Thread.sleep(Native Method)
at
app//org.apache.accumulo.fate.util.UtilWaitThread.sleep(UtilWaitThread.java:33)
at
app//org.apache.accumulo.test.compaction.ExternalCompactionTestUtils.startCoordinator(ExternalCompactionTestUtils.java:243)
at
app//org.apache.accumulo.test.compaction.ExternalCompaction_3_IT.testCoordinatorRestartsDuringCompaction(ExternalCompaction_3_IT.java:141)
at
[email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
[email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
[email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
at
app//org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
app//org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
app//org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
app//org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
app//org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at
[email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at [email protected]/java.lang.Thread.run(Thread.java:833)
[ERROR]
org.apache.accumulo.test.compaction.ExternalCompaction_3_IT.testCoordinatorRestartsDuringCompaction
Time elapsed: 1800.002 s <<< ERROR!
java.lang.Exception: Appears to be stuck in thread Time-limited
test-SendThread(localhost:41655)
at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
at
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
at
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
at
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:347)
at
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
org.apache.accumulo.test.compaction.ExternalCompaction_3_IT.testCoordinatorRestartsDuringCompaction
[ERROR] Run 1:
ExternalCompaction_3_IT.testCoordinatorRestartsDuringCompaction:141 »
TestTimedOut
[ERROR] Run 2:
ExternalCompaction_3_IT.testCoordinatorRestartsDuringCompaction » Appears to
...
[INFO]
[ERROR]
org.apache.accumulo.test.compaction.ExternalCompaction_3_IT.testMergeCancelsExternalCompaction
[ERROR] Run 1:
ExternalCompaction_3_IT.testMergeCancelsExternalCompaction:113 » TestTimedOut
...
[ERROR] Run 2: ExternalCompaction_3_IT.testMergeCancelsExternalCompaction
» Appears to be st...
[INFO]
[INFO]
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0
</details>
**Additional context**
Please let me know if you need more information!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
