ddanielr opened a new issue, #3281:
URL: https://github.com/apache/accumulo/issues/3281
**Describe the bug**
Tests are failing during testing of the 1.10.3-rc0 candidate using the
hadoop 2 profile.
The following tests failed with a server side error.
* testIsolatedScansWithInterference
* `org.apache.thrift.TApplicationException: Internal error processing
closeUpdate`
* testScansWithInterference
* `org.apache.thrift.TApplicationException: Internal error processing
closeUpdate`
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: 1.10.3-rc0
- OS: CentOS 7.9
- Java version: 1.8.0_352
**To Reproduce**
Steps to reproduce the behavior (or a link to an example repository that
reproduces the problem):
1. Download the [1.10.3.rc0 candidate
artifact](https://repository.apache.org/content/repositories/orgapacheaccumulo-1099/org/apache/accumulo/accumulo/1.10.3/accumulo-1.10.3-src.tar.gz)
2. Extract the contents and cd into that directory
`tar -xvf accumulo-1.10.3-src.tar.gz && cd accumulo-1.10.3`
3. Run the test command `mvn -Dhadoop.profile=2 -Dtimeout.factor=2 clean
verify package -Psunny`
4. See errors thrown in `Apache Accumulo Testing`
```
[INFO] --- maven-failsafe-plugin:3.0.0-M4:verify (run-integration-tests) @
accumulo-test ---
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Accumulo Project 1.10.3:
[INFO]
[INFO] Apache Accumulo Project ............................ SUCCESS [ 28.514
s]
[INFO] Apache Accumulo Fate ............................... SUCCESS [ 40.204
s]
[INFO] Apache Accumulo Start .............................. SUCCESS [01:08
min]
[INFO] Apache Accumulo Core ............................... SUCCESS [04:17
min]
[INFO] Apache Accumulo Documentation ...................... SUCCESS [ 15.547
s]
[INFO] Apache Accumulo Server Base ........................ SUCCESS [01:14
min]
[INFO] Apache Accumulo Tracer Server ...................... SUCCESS [ 15.292
s]
[INFO] Apache Accumulo Shell .............................. SUCCESS [ 27.539
s]
[INFO] Apache Accumulo Simple Examples .................... SUCCESS [ 11.084
s]
[INFO] Apache Accumulo GC Server .......................... SUCCESS [ 7.815
s]
[INFO] Apache Accumulo Master Server ...................... SUCCESS [ 19.137
s]
[INFO] Apache Accumulo Monitor Server ..................... SUCCESS [ 10.208
s]
[INFO] Apache Accumulo Tablet Server ...................... SUCCESS [ 53.683
s]
[INFO] Apache Accumulo MiniCluster ........................ SUCCESS [01:28
min]
[INFO] Apache Accumulo Native Libraries ................... SUCCESS [ 9.084
s]
[INFO] Apache Accumulo Proxy .............................. SUCCESS [ 58.097
s]
[INFO] Apache Accumulo Trace .............................. SUCCESS [ 2.679
s]
[INFO] Apache Accumulo Iterator Test Harness .............. SUCCESS [ 5.533
s]
[INFO] Apache Accumulo Testing ............................ FAILURE [ 01:16
h]
[INFO] Apache Accumulo .................................... SKIPPED
[INFO] Apache Accumulo Maven Plugin ....................... SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 01:30 h
[INFO] Finished at: 2023-04-07T02:18:22Z
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M4:verify
(run-integration-tests) on project accumulo-test: There are test failures.
```
**Stack Traces**
`
org.apache.accumulo.test.functional.ExamplesIT.testIsolatedScansWithInterference
`
```
[ERROR] Tests run: 18, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
2,802.793 s <<< FAILURE! - in org.apache.accumulo.test.functional.ExamplesIT
[ERROR]
org.apache.accumulo.test.functional.ExamplesIT.testIsolatedScansWithInterference
Time elapsed: 215.987 s <<< FAILURE!
java.lang.AssertionError:
stdout=2023-04-07 01:22:42,285 [classloader.AccumuloClassLoader] DEBUG:
ignoring classpath entry
/data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testIsolatedScansWithInterference/lib/[^.].*[.]jar
2023-04-07 01:22:42,285 [classloader.AccumuloClassLoader] DEBUG: Create 2nd
tier ClassLoader using URLs: []
2023-04-07 01:22:42,550 [cache.SoftRefFilesCache] DEBUG: putFile:
file:///data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testIsolatedScansWithInterference/lib/ext/[^.].*[.]jar
2023-04-07 01:22:42,550 [cache.SoftRefFilesCache] DEBUG: putFile: file:///
2023-04-07 01:22:42,551 [cache.SoftRefFilesCache] DEBUG: putFile:
file:///data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testIsolatedScansWithInterference/lib/ext
2023-04-07 01:22:42,556 [vfs.AccumuloReloadingVFSClassLoader] DEBUG:
monitoring
file:///data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testIsolatedScansWithInterference/lib/ext
2023-04-07 01:22:43,415 [client.ClientConfiguration] INFO : Loaded client
configuration file
/data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testIsolatedScansWithInterference/conf/client.conf
2023-04-07 01:22:43,419 [client.ClientConfiguration] INFO : Loaded client
configuration file
/data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testIsolatedScansWithInterference/conf/client.conf
2023-04-07 01:22:43,433 [zookeeper.ZooSession] DEBUG: Connecting to
localhost:44155 with timeout 30000 with auth
2023-04-07 01:26:06,315 [impl.TabletServerBatchWriter] ERROR: Server side
error on ip-10-117-153-69.evoforge.org:39195:
org.apache.thrift.TApplicationException: Internal error processing closeUpdate
2023-04-07 01:26:06,316 [impl.TabletServerBatchWriter] ERROR: Failed to send
tablet server ip-10-117-153-69.evoforge.org:39195 its batch : Error on server
ip-10-117-153-69.evoforge.org:39195
org.apache.accumulo.core.client.impl.AccumuloServerException: Error on
server ip-10-117-153-69.evoforge.org:39195
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:1037)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.access$2100(TabletServerBatchWriter.java:676)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.send(TabletServerBatchWriter.java:895)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.run(TabletServerBatchWriter.java:851)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.thrift.TApplicationException: Internal error
processing closeUpdate
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_closeUpdate(TabletClientService.java:457)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.closeUpdate(TabletClientService.java:443)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:996)
... 9 more
2023-04-07 01:26:06,325 [isolation.InterferenceTest] ERROR: Mutation was
rejected.
org.apache.accumulo.core.client.MutationsRejectedException: # constraint
violations : 0 security codes: {} # server errors 1 # exceptions 1
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:594)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:265)
at
org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:43)
at
org.apache.accumulo.examples.simple.isolation.InterferenceTest$Writer.run(InterferenceTest.java:82)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException:
Error on server ip-10-117-153-69.evoforge.org:39195
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:1037)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.access$2100(TabletServerBatchWriter.java:676)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.send(TabletServerBatchWriter.java:895)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.run(TabletServerBatchWriter.java:851)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
... 1 more
Caused by: org.apache.thrift.TApplicationException: Internal error
processing closeUpdate
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_closeUpdate(TabletClientService.java:457)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.closeUpdate(TabletClientService.java:443)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:996)
... 9 more
expected:<0> but was:<255>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at
org.apache.accumulo.test.functional.ExamplesIT.goodExec(ExamplesIT.java:726)
at
org.apache.accumulo.test.functional.ExamplesIT.testIsolatedScansWithInterference(ExamplesIT.java:608)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
```
`org.apache.accumulo.test.functional.ExamplesIT.testScansWithInterference`
```
[ERROR]
org.apache.accumulo.test.functional.ExamplesIT.testScansWithInterference Time
elapsed: 166.844 s <<< FAILURE!
java.lang.AssertionError:
stdout=2023-04-07 01:54:58,178 [classloader.AccumuloClassLoader] DEBUG:
ignoring classpath entry
/data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testScansWithInterference/lib/[^.].*[.]jar
2023-04-07 01:54:58,178 [classloader.AccumuloClassLoader] DEBUG: Create 2nd
tier ClassLoader using URLs: []
2023-04-07 01:54:58,326 [cache.SoftRefFilesCache] DEBUG: putFile:
file:///data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testScansWithInterference/lib/ext/[^.].*[.]jar
2023-04-07 01:54:58,327 [cache.SoftRefFilesCache] DEBUG: putFile: file:///
2023-04-07 01:54:58,327 [cache.SoftRefFilesCache] DEBUG: putFile:
file:///data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testScansWithInterference/lib/ext
2023-04-07 01:54:58,331 [vfs.AccumuloReloadingVFSClassLoader] DEBUG:
monitoring
file:///data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testScansWithInterference/lib/ext
2023-04-07 01:54:58,656 [client.ClientConfiguration] INFO : Loaded client
configuration file
/data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testScansWithInterference/conf/client.conf
2023-04-07 01:54:58,659 [client.ClientConfiguration] INFO : Loaded client
configuration file
/data/workspace/accumulo-1.10.3/test/target/mini-tests/org.apache.accumulo.test.functional.ExamplesIT_testScansWithInterference/conf/client.conf
2023-04-07 01:54:58,672 [zookeeper.ZooSession] DEBUG: Connecting to
localhost:44176 with timeout 30000 with auth
2023-04-07 01:57:31,202 [impl.TabletServerBatchWriter] ERROR: Server side
error on ip-10-117-153-69.evoforge.org:41194:
org.apache.thrift.TApplicationException: Internal error processing closeUpdate
2023-04-07 01:57:31,215 [impl.TabletServerBatchWriter] ERROR: Failed to send
tablet server ip-10-117-153-69.evoforge.org:41194 its batch : Error on server
ip-10-117-153-69.evoforge.org:41194
org.apache.accumulo.core.client.impl.AccumuloServerException: Error on
server ip-10-117-153-69.evoforge.org:41194
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:1037)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.access$2100(TabletServerBatchWriter.java:676)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.send(TabletServerBatchWriter.java:895)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.run(TabletServerBatchWriter.java:851)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.thrift.TApplicationException: Internal error
processing closeUpdate
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_closeUpdate(TabletClientService.java:457)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.closeUpdate(TabletClientService.java:443)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:996)
... 9 more
2023-04-07 01:57:31,219 [isolation.InterferenceTest] ERROR: Mutation was
rejected.
org.apache.accumulo.core.client.MutationsRejectedException: # constraint
violations : 0 security codes: {} # server errors 1 # exceptions 1
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:594)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:265)
at
org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:43)
at
org.apache.accumulo.examples.simple.isolation.InterferenceTest$Writer.run(InterferenceTest.java:82)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException:
Error on server ip-10-117-153-69.evoforge.org:41194
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:1037)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.access$2100(TabletServerBatchWriter.java:676)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.send(TabletServerBatchWriter.java:895)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.run(TabletServerBatchWriter.java:851)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
... 1 more
Caused by: org.apache.thrift.TApplicationException: Internal error
processing closeUpdate
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_closeUpdate(TabletClientService.java:457)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.closeUpdate(TabletClientService.java:443)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:996)
... 9 more
expected:<0> but was:<255>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at
org.apache.accumulo.test.functional.ExamplesIT.goodExec(ExamplesIT.java:726)
at
org.apache.accumulo.test.functional.ExamplesIT.testScansWithInterference(ExamplesIT.java:621)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
```
**Expected behavior**
The tests should complete without issue.
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
