[
https://issues.apache.org/jira/browse/ACCUMULO-383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-383.
----------------------------------------
Resolution: Cannot Reproduce
> Shard random walk test failed w/ TableDeleted exception
> -------------------------------------------------------
>
> Key: ACCUMULO-383
> URL: https://issues.apache.org/jira/browse/ACCUMULO-383
> Project: Accumulo
> Issue Type: Bug
> Reporter: Keith Turner
> Assignee: Keith Turner
> Labels: 14_qa_bug
>
> While running the random walk test on a 10 node cluster, the shard test
> failed with the following errors.
> {noformat}
> 07 23:43:38,483 [shard.CloneIndex] DEBUG: Cloned
> ST_index_26192_1328649015908_tmp from ST_index_26192_1328649015908 flush:
> 5825ms clone: 96ms
> 07 23:43:38,503 [impl.ThriftScanner] DEBUG: Failed to locate tablet for table
> : rr row :
> 07 23:49:52,795 [impl.ThriftScanner] DEBUG: Scan failed, not serving tablet
> (ln<<,xxx.xxx.xxx.10:9997)
> 07 23:51:11,539 [shard.VerifyIndex] DEBUG: Verified 65438656 index entries
> 07 23:51:12,052 [shard.Search] DEBUG: Looking up terms [67x, 1o3g, dl4, e4m]
> expect to find 2682000000000000
> 07 23:51:12,110 [impl.TabletServerBatchReaderIterator] DEBUG: Server :
> xxx.xxx.xxx.8:9997 msg : null
> ThriftSecurityException(user:root, code:PERMISSION_DENIED)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startMultiScan_result.read(TabletClientService.java:7586)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startMultiScan(TabletClientService.java:306)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startMultiScan(TabletClientService.java:274)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:83)
> at $Proxy2.startMultiScan(Unknown Source)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:539)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:335)
> at cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 07 23:51:12,113 [impl.TabletServerBatchReaderIterator] DEBUG: Error
> PERMISSION_DENIED - User does not have permission to perform this action
> org.apache.accumulo.core.client.AccumuloSecurityException: Error
> PERMISSION_DENIED - User does not have permission to perform this action
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:576)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:335)
> at cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: ThriftSecurityException(user:root, code:PERMISSION_DENIED)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startMultiScan_result.read(TabletClientService.java:7586)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startMultiScan(TabletClientService.java:306)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startMultiScan(TabletClientService.java:274)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:83)
> at $Proxy2.startMultiScan(Unknown Source)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:539)
> ... 5 more
> 07 23:51:12,114 [randomwalk.Module] DEBUG: Properties for node: shard.Search
> 07 23:51:12,114 [randomwalk.Module] DEBUG: teardown:
> 07 23:51:12,114 [randomwalk.Module] DEBUG: maxSec:
> 07 23:51:12,114 [randomwalk.Module] DEBUG: maxHops:
> 07 23:51:12,114 [randomwalk.Module] DEBUG: Properties for node: Shard.xml
> 07 23:51:12,114 [randomwalk.Module] DEBUG: teardown:
> 07 23:51:12,114 [randomwalk.Module] DEBUG: maxSec:
> 07 23:51:12,114 [randomwalk.Module] DEBUG: maxHops:
> 07 23:51:12,114 [randomwalk.Framework] ERROR: Error during random walk
> java.lang.Exception: Error running node Shard.xml
> at
> org.apache.accumulo.server.test.randomwalk.Module.visit(Module.java:259)
> at
> org.apache.accumulo.server.test.randomwalk.Framework.run(Framework.java:61)
> at
> org.apache.accumulo.server.test.randomwalk.Framework.main(Framework.java:114)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.accumulo.start.Main$1.run(Main.java:89)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.Exception: Error running node shard.Search
> at
> org.apache.accumulo.server.test.randomwalk.Module.visit(Module.java:259)
> at
> org.apache.accumulo.server.test.randomwalk.Module.visit(Module.java:251)
> ... 8 more
> Caused by: org.apache.accumulo.core.client.TableDeletedException
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:357)
> at cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ... 1 more
> {noformat}
> After seeing this looked and saw that the table existed. So for some reason
> a tablet server and client process did not see metadata related to the table.
> This happened after a delete and rename. One possibility is that the client
> is using the old table id from before the delete and rename. However, not
> sure how this could have happened. If the log messages printed the table id
> this would be easy to verify. Another possibility is that its some sort of
> zookeeper hiccup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
