At this point all but two of the Accumulo test/system/auto tests have completed
successfully. This test is failing and I'm not quite sure why:
org.apache.accumulo.server.test.functional.LargeRowTest
When I run it, this is the output I see:
./run.py -t largerowtest -d -v10
….
09:45:18 runTest (simple.largeRow.LargeRowTest) .............................
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo
org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root
-p secret -i SE-test-04-22187
org.apache.accumulo.server.test.functional.LargeRowTest getConfig
DEBUG:test.auto:{
'tserver.compaction.major.delay':'1',
}
DEBUG:test.auto:
INFO:test.auto:killing accumulo processes everywhere
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/test/system/auto/pkill.sh 9 1000
SE-test-04-22187.*org.apache.accumulo.start
DEBUG:test.auto:localhost: hadoop fs -rmr /user/mapr/accumulo-SE-test-04-22187
INFO:test.auto:Error output from command: rmr: cannot remove
/user/mapr/accumulo-SE-test-04-22187: No such file or directory.
DEBUG:test.auto:Exit code: 255
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo init
--clear-instance-name
DEBUG:test.auto:Output from command: 10 09:45:20,539 [util.Initialize] INFO :
Hadoop Filesystem is maprfs:///
10 09:45:20,541 [util.Initialize] INFO : Accumulo data dir is
/user/mapr/accumulo-SE-test-04-22187
10 09:45:20,541 [util.Initialize] INFO : Zookeeper server is
SE-test-00:5181,SE-test-01:5181,SE-test-02:5181
Instance name : SE-test-04-22187
Enter initial password for root: ******
Confirm initial password for root: ******
10 09:45:21,442 [util.NativeCodeLoader] INFO : Loaded the native-hadoop library
10 09:45:21,562 [security.ZKAuthenticator] INFO : Initialized root user with
username: root at the request of user !SYSTEM
DEBUG:test.auto:Exit code: 0
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo logger
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo tserver
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo monitor
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo
org.apache.accumulo.server.master.state.SetGoalState NORMAL
DEBUG:test.auto:Output from command: 10 09:45:22,529 [server.Accumulo] INFO :
Attempting to talk to zookeeper
10 09:45:22,750 [server.Accumulo] INFO : Zookeeper connected and initialized,
attemping to talk to HDFS
10 09:45:23,009 [server.Accumulo] INFO : Connected to HDFS
DEBUG:test.auto:Exit code: 0
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo master
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo
org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root
-p secret -i SE-test-04-22187
org.apache.accumulo.server.test.functional.LargeRowTest setup
DEBUG:test.auto:
DEBUG:test.auto:
DEBUG:test.auto:localhost: /opt/accumulo-1.4.0/bin/accumulo
org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root
-p secret -i SE-test-04-22187
org.apache.accumulo.server.test.functional.LargeRowTest run
DEBUG:test.auto:Waiting for /opt/accumulo-1.4.0/bin/accumulo
org.apache.accumulo.server.test.functional.FunctionalTest -m localhost -u root
-p secret -i SE-test-04-22187
org.apache.accumulo.server.test.functional.LargeRowTest run to stop in 240 secs
DEBUG:test.auto:err: Thread
"org.apache.accumulo.server.test.functional.FunctionalTest" died null
DEBUG:test.auto:err: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
DEBUG:test.auto:err: at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.start.Main$1.run(Main.java:89)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException:
org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server
10.250.99.204:39253
DEBUG:test.auto:err:
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
at
org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
at
org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
at
org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
at
org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
at
org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
... 6 more
DEBUG:test.auto:err: Caused by:
org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server
10.250.99.204:39253
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
at
org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
... 11 more
Caused by: org.apache.thrift.TApplicationException: Internal error processing
startScan
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
DEBUG:test.auto:err:
at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
at $Proxy1.startScan(Unknown Source)
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
... 13 more
ERROR:test.auto:This looks like a stack trace: Thread
"org.apache.accumulo.server.test.functional.FunctionalTest" died null
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.start.Main$1.run(Main.java:89)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException:
org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server
10.250.99.204:39253
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:186)
at
org.apache.accumulo.server.test.functional.LargeRowTest.verify(LargeRowTest.java:165)
at
org.apache.accumulo.server.test.functional.LargeRowTest.basicTest(LargeRowTest.java:143)
at
org.apache.accumulo.server.test.functional.LargeRowTest.test2(LargeRowTest.java:104)
at
org.apache.accumulo.server.test.functional.LargeRowTest.run(LargeRowTest.java:87)
at
org.apache.accumulo.server.test.functional.FunctionalTest.main(FunctionalTest.java:312)
... 6 more
Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error
on server 10.250.99.204:39253
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:302)
at
org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:94)
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:176)
... 11 more
Caused by: org.apache.thrift.TApplicationException: Internal error processing
startScan
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
at $Proxy1.startScan(Unknown Source)
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
... 13 more
FAIL
======================================================================
FAIL: runTest (simple.largeRow.LargeRowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
self.waitForStop(handle, self.maxRuntime)
File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in
waitForStop
self.assert_(self.processResult(out, err, handle.returncode))
AssertionError: False is not true
======================================================================
FAIL: runTest (simple.largeRow.LargeRowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/accumulo-1.4.0/test/system/auto/JavaTest.py", line 57, in runTest
self.waitForStop(handle, self.maxRuntime)
File "/opt/accumulo-1.4.0/test/system/auto/TestUtils.py", line 368, in
waitForStop
self.assert_(self.processResult(out, err, handle.returncode))
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 43.014s
FAILED (failures=1)
The only log that seems to have any relevant output is the tserver_xxxx.log
file. In it I found this error:
Note that the timestamps here do not match the previous timestamps. This is
just because I forgot to capture the data from the run that corresponds exactly
to this run.
09 06:14:22,466 [tabletserver.TabletServer] INFO : Adding 1 logs for extent
2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ...
TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR...
TRUNCATED as alias 16
09 06:14:25,018 [tabletserver.TabletServer] WARN : exception while scanning
tablet 2;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb...
TRUNCATED;\\]jx?h@XRt8nDO%{>vT-Et-P$b.<,-4b2osta{ZE\\$u9k2T-MpdF
_^<q\\M`X\\Er... TRUNCATED
java.io.IOException: invalid distance too far back
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
Method)
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
at
org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at
org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
at
org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
at
org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
at
org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
09 06:14:25,020 [tabletserver.TabletServer] INFO : Adding 1 logs for extent
!0<;~ as alias 2
09 06:14:25,022 [thrift.TabletClientService$Processor] ERROR: Internal error
processing startScan
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.io.IOException: invalid distance too far back
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
at $Proxy0.startScan(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException:
invalid distance too far back
at
org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
... 15 more
Caused by: java.io.IOException: invalid distance too far back
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
Method)
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
at
org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:81)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at
org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:116)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
at
org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
at
org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
at
org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
After guessing that the zlib error might be hiding the "real" error, I decided
to disable compression in Accumulo (compression in MapR is transparent and the
results are not affected by whether it is on or off). Normally I'd set
table.file.compress.type to none in the accumulo-site.xml file but that doesn't
work for the tests as they generate they own site files automatically. I hand
edited TestUtil.py to generate a site file with that property set.
When I rerun the test, I get the same output from run.py, but the server error
in tserver_xxx.log is very different:
10 09:45:51,650 [tabletserver.TabletServer] INFO : Adding 1 logs for extent
2;(8{]laDN>C?'1D\\;K]l:fS\\lVXKOWq[_'&8".>-wL$Y,x-k<18_#t:7CHMH:\\)Zga...
TRUNCATED;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@...
TRUNCATED as alias 8
10 09:45:51,693 [tabletserver.TabletServer] INFO : Adding 1 logs for extent
2;k9TF\\;hI"]Ij'4\\P.t&'pJm3"\\;C0qd:Q>%G3>I6!5[pVN$5R0g1LwmPUg 5-fX4jG...
TRUNCATED;h&["[>Er>fnBdhzAR_'I!Htot>R/hNK_vNG)Y1a%$DJWg#QyQHFZ RaUAF3[p!eb...
TRUNCATED as alias 22
10 09:45:51,748 [tabletserver.TabletServer] INFO : Adding 1 logs for extent
2;F]\\;J^>ioHJ*gs[4TwSIQeN_C^]1!w@7e<wL<p.xE&TR\\g!lt6+c^0a3U7%Eo'Ji ...
TRUNCATED;CJlc"pWa)g<$Gg(\\U0Kl<)ffOYm1{h@E1"nV$)z'7'8KNWt- .BISxZoDI^[?7jR...
TRUNCATED as alias 16
10 09:46:00,996 [tabletserver.TabletServer] WARN : exception while scanning
tablet 2;%9e.07Zx{t*taPSI\\;I4z*77vIG[Oa&(Dw?4_N(!OIA#Z(ZE%"v3gI9Q{ZlGNAGL@...
TRUNCATED<
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
at
org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
at
org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
at
org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
at
org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent
!0<;~ as alias 2
10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error
processing startScan
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.io.EOFException
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
at $Proxy0.startScan(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
at
org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
... 15 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
at
org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
at
org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
at
org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
at
org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
10 09:46:00,999 [tabletserver.TabletServer] INFO : Adding 1 logs for extent
!0<;~ as alias 2
10 09:46:01,000 [thrift.TabletClientService$Processor] ERROR: Internal error
processing startScan
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.io.EOFException
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1155)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startScan(TabletServer.java:1110)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
at $Proxy0.startScan(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startScan.process(TabletClientService.java:2059)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
at
org.apache.accumulo.server.tabletserver.TabletServer$ScanTask.get(TabletServer.java:662)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueScan(TabletServer.java:1146)
... 15 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
org.apache.accumulo.core.file.rfile.RelativeKey.read(RelativeKey.java:378)
at
org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:134)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:584)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:556)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:77)
at
org.apache.accumulo.core.iterators.system.HeapIterator.next(HeapIterator.java:80)
at
org.apache.accumulo.core.iterators.system.DeletingIterator.next(DeletingIterator.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at org.apache.accumulo.core.iterators.Filter.next(Filter.java:58)
at
org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:87)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:103)
at
org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:53)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:120)
at
org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
at
org.apache.accumulo.server.tabletserver.Tablet.nextBatch(Tablet.java:1766)
at
org.apache.accumulo.server.tabletserver.Tablet.access$3200(Tablet.java:143)
at
org.apache.accumulo.server.tabletserver.Tablet$Scanner.read(Tablet.java:1883)
at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:905)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
So the error would seem to be related to reading past the end of the file. What
I can't determine is what is the reason. From examining the Accumulo source,
it's clear that Accumulo has read the length of a key (I think) and is now
trying to read the key value. That second read is what is failing. The question
is why? Some ideas:
1) the file was originally written incorrectly by the writer,
2) the reader is reading too far
This could be caused by a issue in Accumulo or in MapR. It might be that MapR
more strongly enforces end of file reads than stock Hadoop.
If anyone has suggestions on how to look into this further from the Accumulo
side, I'd really appreciate it.
Thanks,
Keys
________________________________
Keys Botzum
Senior Principal Technologist
WW Systems Engineering
[email protected]
443-718-0098
MapR Technologies
http://www.mapr.com
