[
https://issues.apache.org/jira/browse/ACCUMULO-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329665#comment-14329665
]
Josh Elser commented on ACCUMULO-3609:
--------------------------------------
Seeing lots of in ShellServerIT's output
{noformat}
2015-02-20 17:17:47,907 [impl.ThriftTransportPool] WARN : Thread "Thread-41"
stuck on IO to hostname:55675 (0) for at least 120146 ms
{noformat}
Stuck in
{noformat}
"Thread-41" prio=5 tid=0x00007fda65017000 nid=0x4517 runnable
[0x000000010bf93000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00000007c0022eb8> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.readAll(ThriftTransportPool.java:275)
at
org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:601)
at
org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:470)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.accumulo.core.master.thrift.FateService$Client.recv_waitForFateOperation(FateService.java:174)
at
org.apache.accumulo.core.master.thrift.FateService$Client.waitForFateOperation(FateService.java:159)
at
org.apache.accumulo.core.client.impl.TableOperationsImpl.waitForFateOperation(TableOperationsImpl.java:267)
at
org.apache.accumulo.core.client.impl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:309)
at
org.apache.accumulo.core.client.impl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:295)
at
org.apache.accumulo.core.client.impl.TableOperationsImpl.doTableFateOperation(TableOperationsImpl.java:1594)
at
org.apache.accumulo.core.client.impl.TableOperationsImpl.delete(TableOperationsImpl.java:681)
at
org.apache.accumulo.core.util.shell.commands.DeleteTableCommand.doTableOp(DeleteTableCommand.java:54)
at
org.apache.accumulo.core.util.shell.commands.TableOperation.execute(TableOperation.java:90)
at
org.apache.accumulo.core.util.shell.commands.DeleteTableCommand.execute(DeleteTableCommand.java:44)
at org.apache.accumulo.core.util.shell.Shell.execCommand(Shell.java:747)
at
org.apache.accumulo.test.ShellServerIT$TestShell.exec(ShellServerIT.java:150)
at
org.apache.accumulo.test.ShellServerIT$TestShell.exec(ShellServerIT.java:159)
at
org.apache.accumulo.test.ShellServerIT$TestShell.exec(ShellServerIT.java:155)
at
org.apache.accumulo.test.ShellServerIT.exporttableImporttable(ShellServerIT.java:301)
{noformat}
Master is waiting
{noformat}
"ClientPool 43" daemon prio=5 tid=0x00007f9bce000000 nid=0x9227 in
Object.wait() [0x000000011f07c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.accumulo.fate.ZooStore.waitForStatusChange(ZooStore.java:356)
- locked <0x000000010c0b9b20> (a org.apache.accumulo.fate.ZooStore)
at
org.apache.accumulo.fate.AgeOffStore.waitForStatusChange(AgeOffStore.java:210)
at org.apache.accumulo.fate.Fate.waitForCompletion(Fate.java:191)
at
org.apache.accumulo.master.FateServiceHandler.waitForFateOperation(FateServiceHandler.java:484)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.accumulo.trace.instrument.thrift.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
at
org.apache.accumulo.server.util.RpcWrapper$1.invoke(RpcWrapper.java:47)
at com.sun.proxy.$Proxy10.waitForFateOperation(Unknown Source)
at
org.apache.accumulo.core.master.thrift.FateService$Processor$waitForFateOperation.getResult(FateService.java:478)
at
org.apache.accumulo.core.master.thrift.FateService$Processor$waitForFateOperation.getResult(FateService.java:462)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:168)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
at
org.apache.accumulo.server.util.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:77)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> BalanceInPresenceOfOfflineTableIT seems to be hanging
> -----------------------------------------------------
>
> Key: ACCUMULO-3609
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3609
> Project: Accumulo
> Issue Type: Bug
> Components: test
> Reporter: Josh Elser
> Fix For: 1.7.0
>
>
> I noticed today that BalanceInPresenceOfOfflineTableIT seems to have started
> failing for me.
> I see the following repeated ad infinitum in the tserver logs
> {noformat}
> 2015-02-20 15:52:25,830 [util.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> row_0000040499 and row_00000405 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,833 [tablet.Tablet] ERROR: Failed to find midpoint Failed
> to find mid point, no entries between row_0000040499 and row_00000405 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,840 [util.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> row_0000100199 and row_00001002 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
> 2015-02-20 15:52:25,841 [tablet.Tablet] ERROR: Failed to find midpoint Failed
> to find mid point, no entries between row_0000100199 and row_00001002 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
> 2015-02-20 15:52:25,850 [util.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> row_0000141299 and row_00001413 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,855 [tablet.Tablet] ERROR: Failed to find midpoint Failed
> to find mid point, no entries between row_0000141299 and row_00001413 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,862 [util.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> row_0000179599 and row_00001796 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,865 [tablet.Tablet] ERROR: Failed to find midpoint Failed
> to find mid point, no entries between row_0000179599 and row_00001796 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,878 [util.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> row_0000040499 and row_00000405 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,881 [tablet.Tablet] ERROR: Failed to find midpoint Failed
> to find mid point, no entries between row_0000040499 and row_00000405 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,889 [util.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> row_0000100199 and row_00001002 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
> 2015-02-20 15:52:25,891 [tablet.Tablet] ERROR: Failed to find midpoint Failed
> to find mid point, no entries between row_0000100199 and row_00001002 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
> 2015-02-20 15:52:25,897 [util.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> row_0000141299 and row_00001413 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,900 [tablet.Tablet] ERROR: Failed to find midpoint Failed
> to find mid point, no entries between row_0000141299 and row_00001413 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,904 [util.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> row_0000179599 and row_00001796 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,906 [tablet.Tablet] ERROR: Failed to find midpoint Failed
> to find mid point, no entries between row_0000179599 and row_00001796 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
> 2015-02-20 15:52:25,920 [util.FileUtil] WARN : Failed to find mid point using
> indexes, falling back to data files which is slower. No entries between
> row_0000040499 and row_00000405 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> 2015-02-20 15:52:25,923 [tablet.Tablet] ERROR: Failed to find midpoint Failed
> to find mid point, no entries between row_0000040499 and row_00000405 for
> [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf,
>
> file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
> {noformat}
> It does seem to eventually finish (occasionally), so I'm not really sure
> what's happening yet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
