keith-turner opened a new issue, #3031:
URL: https://github.com/apache/accumulo/issues/3031
**Describe the bug**
While running the random walk bulk import test I killed tablet servers
inorder to restart them with more memory. This should not have impacted the
test that was running, however I saw the test fail with the following error.
```
2022-10-19T13:16:20,224 [testing.randomwalk.Framework] ERROR: Error during
random walk
java.lang.Exception: Error running node bulk.Verify
at
org.apache.accumulo.testing.randomwalk.Module.visit(Module.java:360)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.testing.randomwalk.Framework.run(Framework.java:54)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.testing.randomwalk.Framework.main(Framework.java:98)
~[accumulo-testing-shaded.jar:?]
Caused by: java.lang.RuntimeException:
java.util.concurrent.ExecutionException:
org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server
localhost:10003
at
org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:177)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:106)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.util.PeekingIterator.next(PeekingIterator.java:70)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.client.RowIterator$SingleRowIter._next(RowIterator.java:76)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.client.RowIterator$SingleRowIter.next(RowIterator.java:69)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.client.RowIterator$SingleRowIter.next(RowIterator.java:42)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.testing.randomwalk.bulk.Verify.visit(Verify.java:86)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.testing.randomwalk.Module.lambda$visit$0(Module.java:295)
~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.util.concurrent.ExecutionException:
org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server
localhost:10003
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:172)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:106)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.util.PeekingIterator.next(PeekingIterator.java:70)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.client.RowIterator$SingleRowIter._next(RowIterator.java:76)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.client.RowIterator$SingleRowIter.next(RowIterator.java:69)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.client.RowIterator$SingleRowIter.next(RowIterator.java:42)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.testing.randomwalk.bulk.Verify.visit(Verify.java:86)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.testing.randomwalk.Module.lambda$visit$0(Module.java:295)
~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.apache.accumulo.core.clientImpl.AccumuloServerException:
Error on server localhost:10003
at
org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:363)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:152)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.trace.TraceWrappedCallable.call(TraceWrappedCallable.java:53)
~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.apache.thrift.TApplicationException: Internal error
processing continueScan
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:81)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.tabletserver.thrift.TabletScanClientService$Client.recv_continueScan(TabletScanClientService.java:162)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.tabletserver.thrift.TabletScanClientService$Client.continueScan(TabletScanClientService.java:147)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.clientImpl.ThriftScanner.scanRpc(ThriftScanner.java:646)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:575)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:354)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:152)
~[accumulo-testing-shaded.jar:?]
at
org.apache.accumulo.core.trace.TraceWrappedCallable.call(TraceWrappedCallable.java:53)
~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-testing-shaded.jar:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2022-10-19T13:16:20,233 [testing.randomwalk.Framework] INFO : Test finished
```
On one of the tservers, saw the following error.
```
2022-10-19T13:16:20,210 [thrift.ProcessFunction] ERROR: Internal error
processing continueScan
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.io.IOException: Filesystem closed
at
org.apache.accumulo.tserver.ThriftScanClientHandler.continueScan(ThriftScanClientHandler.java:281)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.ThriftScanClientHandler.continueScan(ThriftScanClientHandler.java:242)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown
Source) ~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$0(TraceUtil.java:206)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy37.continueScan(Unknown Source) ~[?:?]
at
org.apache.accumulo.core.tabletserver.thrift.TabletScanClientService$Processor$continueScan.getResult(TabletScanClientService.java:788)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.tabletserver.thrift.TabletScanClientService$Processor$continueScan.getResult(TabletScanClientService.java:764)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40)
~[libthrift-0.17.0.jar:0.17.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:147)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:54)
~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:129)
~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
~[libthrift-0.17.0.jar:0.17.0]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.io.IOException: Filesystem closed
at org.apache.accumulo.tserver.scan.ScanTask.get(ScanTask.java:158)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.ThriftScanClientHandler.continueScan(ThriftScanClientHandler.java:263)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 20 more
Caused by: java.lang.RuntimeException: java.io.IOException: Filesystem closed
at
org.apache.accumulo.core.iterators.Combiner$ValueIterator.next(Combiner.java:133)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.Combiner$ValueIterator.next(Combiner.java:96)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.TypedValueCombiner$VIterator.hasNext(TypedValueCombiner.java:90)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:33)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:29)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.TypedValueCombiner.reduce(TypedValueCombiner.java:180)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.Combiner.findTop(Combiner.java:225)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.Combiner.next(Combiner.java:181)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:95)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:100)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:60)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:178)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:149)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.tablet.TabletBase.nextBatch(TabletBase.java:288)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.Scanner.read(Scanner.java:108)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.scan.NextBatchTask.run(NextBatchTask.java:75)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer.run(ScanSession.java:62)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 5 more
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:494)
~[hadoop-client-api-3.3.3.jar:?]
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:866)
~[hadoop-client-api-3.3.3.jar:?]
at
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:957)
~[hadoop-client-api-3.3.3.jar:?]
at java.io.DataInputStream.read(DataInputStream.java:149) ~[?:?]
at
org.apache.accumulo.core.file.streams.RateLimitedInputStream.read(RateLimitedInputStream.java:52)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.io.DataInputStream.read(DataInputStream.java:149) ~[?:?]
at
org.apache.accumulo.core.file.streams.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:179)
~[hadoop-client-api-3.3.3.jar:?]
at
org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:163)
~[hadoop-client-api-3.3.3.jar:?]
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
~[hadoop-client-api-3.3.3.jar:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
~[?:?]
at java.io.BufferedInputStream.read(BufferedInputStream.java:271)
~[?:?]
at java.io.FilterInputStream.read(FilterInputStream.java:83) ~[?:?]
at java.io.DataInputStream.readByte(DataInputStream.java:270) ~[?:?]
at
org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:164)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:859)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:824)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iteratorsImpl.system.HeapIterator.next(HeapIterator.java:74)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:86)
~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iteratorsImpl.system.HeapIterator.next(HeapIterator.java:74)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iteratorsImpl.system.StatsIterator.next(StatsIterator.java:51)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iteratorsImpl.system.DeletingIterator.next(DeletingIterator.java:65)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.ServerSkippingIterator.next(ServerSkippingIterator.java:45)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.SynchronizedServerFilter.next(SynchronizedServerFilter.java:51)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.Combiner$ValueIterator.next(Combiner.java:130)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.Combiner$ValueIterator.next(Combiner.java:96)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.TypedValueCombiner$VIterator.hasNext(TypedValueCombiner.java:90)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:33)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:29)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.TypedValueCombiner.reduce(TypedValueCombiner.java:180)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.Combiner.findTop(Combiner.java:225)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.Combiner.next(Combiner.java:181)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:95)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:100)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:60)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:178)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:149)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.tablet.TabletBase.nextBatch(TabletBase.java:288)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.Scanner.read(Scanner.java:108)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.scan.NextBatchTask.run(NextBatchTask.java:75)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer.run(ScanSession.java:62)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 5 more
```
When the process gets a SIGTERM this causes the hadoop filesystem to run its
shutdown hook which causes this exception. In the [tablet server scan
code](https://github.com/apache/accumulo/blob/78142d74f3ed1a23a374f5bd5fdbd7f245a16721/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Scanner.java#L127-L132)
Accumulo tries to work around this by detecting it and wrapping the
IOException with something that will cause the client to retry. That is done
only done when IOException is seen. The bulk test had configured a combiner,
and [the combiner was wrapping the IOException with a
RuntimeException](https://github.com/apache/accumulo/blob/78142d74f3ed1a23a374f5bd5fdbd7f245a16721/core/src/main/java/org/apache/accumulo/core/iterators/Combiner.java#L132-L134)
evading the check.
**Expected behavior**
An Accumulo scanner should not fail when a tablet server is killed, it
should retry. Thinking one possible way to achive this is to do the following.
* Make the code in Combiner throw an UncheckedIOException
* Make the code tablet server scan code catch UncheckedIOException and do
the shutdown check.
* Look for other places in the code that catch IOException and throw
runtime exception and replace those with UncheckedIOException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
