[
https://issues.apache.org/jira/browse/ACCUMULO-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15107019#comment-15107019
]
Eric Newton commented on ACCUMULO-4063:
---------------------------------------
Any chance you have a log message?
I'm trying to discern if the tserver is detecting a master with an out-of-date
or unknown lock, or if the tserver is detecting a missing, loss or failure to
get it's own lock.
These are very different scenarios, with different fixes. The title of this
JIRA ticket implies that the tserver is detecting its own missing lock, but I
have not ever seen this in practice, except during failures. But, I don't use
Ambari, so it might happen all the time.
> TServer should verify its lock is held before answering any RPC
> ---------------------------------------------------------------
>
> Key: ACCUMULO-4063
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4063
> Project: Accumulo
> Issue Type: Bug
> Components: rpc, tserver
> Affects Versions: 1.7.0
> Reporter: Josh Elser
>
> Noticing the following:
> {noformat}
> 2015-11-19 22:42:47,130 [thrift.ProcessFunction] ERROR: Internal error
> processing flush
> org.apache.thrift.TException: Lock not acquired
> at
> org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:51)
> at com.sun.proxy.$Proxy21.flush(Unknown Source)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$flush.getResult(TabletClientService.java:2671)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$flush.getResult(TabletClientService.java:2657)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
> at
> org.apache.accumulo.server.rpc.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> 2015-11-19 22:42:47,402 [tserver.TabletServer$ThriftClientHandler] ERROR:
> org.apache.accumulo.tserver.session.MultiScanSession cannot be cast to
> org.apache.accumulo.tserver.session.ScanSession
> java.lang.ClassCastException:
> org.apache.accumulo.tserver.session.MultiScanSession cannot be cast to
> org.apache.accumulo.tserver.session.ScanSession
> at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.closeScan(TabletServer.java:574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
> at
> org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:47)
> at com.sun.proxy.$Proxy21.closeScan(Unknown Source)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeScan.getResult(TabletClientService.java:2309)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeScan.getResult(TabletClientService.java:2295)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
> at
> org.apache.accumulo.server.rpc.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> a {{flush}} rpc came in and was denied, but then a {{closeScan}} came in and
> threw an unrelated error (ignore the classcast for now).
> Are there _any_ RPCs that a tabletserver should answer before it holds its
> lock?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
