[
https://issues.apache.org/jira/browse/PHOENIX-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated PHOENIX-5364:
-----------------------------------
Description:
Spooling defaults to true. Yet, I see some tests failing (sometimes) with
exceptions like this:
{code}
org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.DoNotRetryIOException:
N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.:
Queue full. Consider increasing memory threshold or spooling to disk. Max
size: 1048576, Current size: 77, Number of elements:1
at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121)
at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87)
at
org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357)
at
org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182)
at
org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
Caused by: java.lang.IllegalStateException: Queue full. Consider increasing
memory threshold or spooling to disk. Max size: 1048576, Current size: 77,
Number of elements:1
at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58)
at
org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315)
at
org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245)
at
org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352)
... 11 more
{code}
This never happens with spooling enabled - it would not use the SizeBoundQueue
at all. So something must set spooling to false.
In fact I found two more tests that set
phoenix.query.server.orderBy.spooling.enabled to false:
* OrderByWithServerClientSpoolingDisabledIT, and
* OrderByWithServerMemoryLimitIT
They need to run in their own cluster.
(Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show
in the logs of the jenkins run... Perhaps because it's a direct child of
BaseTest...?))
was:
Spooling defaults to true. Yet, I see some tests failing (sometimes) with
exceptions like this:
{code}
org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.DoNotRetryIOException:
N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.:
Queue full. Consider increasing memory threshold or spooling to disk. Max
size: 1048576, Current size: 77, Number of elements:1
at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121)
at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87)
at
org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357)
at
org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182)
at
org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
Caused by: java.lang.IllegalStateException: Queue full. Consider increasing
memory threshold or spooling to disk. Max size: 1048576, Current size: 77,
Number of elements:1
at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58)
at
org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315)
at
org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245)
at
org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352)
... 11 more
{code}
This should never happen.
However, I found two more tests that set
phoenix.query.server.orderBy.spooling.enabled to false:
* OrderByWithServerClientSpoolingDisabledIT, and
* OrderByWithServerMemoryLimitIT
They need to run in their own cluster.
(Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show
in the logs of the jenkins run... Perhaps because it's a direct child of
BaseTest...?))
> Tests fail "Queue Full" only when run in parallel with other tests.
> -------------------------------------------------------------------
>
> Key: PHOENIX-5364
> URL: https://issues.apache.org/jira/browse/PHOENIX-5364
> Project: Phoenix
> Issue Type: Test
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Priority: Major
> Fix For: 4.15.0, 5.1.0
>
> Attachments: 5364.txt
>
>
> Spooling defaults to true. Yet, I see some tests failing (sometimes) with
> exceptions like this:
> {code}
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException:
> N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.:
> Queue full. Consider increasing memory threshold or spooling to disk. Max
> size: 1048576, Current size: 77, Number of elements:1
> at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121)
> at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87)
> at
> org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357)
> at
> org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182)
> at
> org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
> Caused by: java.lang.IllegalStateException: Queue full. Consider increasing
> memory threshold or spooling to disk. Max size: 1048576, Current size: 77,
> Number of elements:1
> at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58)
> at
> org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315)
> at
> org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245)
> at
> org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352)
> ... 11 more
> {code}
> This never happens with spooling enabled - it would not use the
> SizeBoundQueue at all. So something must set spooling to false.
> In fact I found two more tests that set
> phoenix.query.server.orderBy.spooling.enabled to false:
> * OrderByWithServerClientSpoolingDisabledIT, and
> * OrderByWithServerMemoryLimitIT
> They need to run in their own cluster.
> (Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not
> show in the logs of the jenkins run... Perhaps because it's a direct child of
> BaseTest...?))
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
