[
https://issues.apache.org/jira/browse/PHOENIX-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788925#comment-17788925
]
Tanuj Khurana commented on PHOENIX-7117:
----------------------------------------
I don't have a definitive fix in mind yet. I also thought about switching to
serial execution if the number of scans are greater than a configurable
threshold. The query would run slow but at-least will complete instead of
getting an error.
I am also thinking around similar lines of dynamically generating tasks per
query with an upper bound on the maximum tasks that can run in parallel for a
given query. I will think more about it.
> Improve handling of scans that span large number of table regions
> -----------------------------------------------------------------
>
> Key: PHOENIX-7117
> URL: https://issues.apache.org/jira/browse/PHOENIX-7117
> Project: Phoenix
> Issue Type: Improvement
> Affects Versions: 5.2.0, 5.1.3
> Reporter: Tanuj Khurana
> Priority: Major
>
> Phoenix determines the number of table regions a query will be sent to and
> creates that many scan objects which are executed in parallel.
>
> {code:java}
> org.apache.phoenix.exception.PhoenixIOException: Task
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask@49c71704[Not
> completed, task = org.apache.phoenix.iterate.ParallelIterators$1@597a1aae]
> rejected from org.apache.phoenix.job.JobManager$1@7c04996f[Running, pool size
> = 20, active threads = 20, queued tasks = 5000, completed tasks =
> 645934]`Failed query: DELETE FROM TEST.EVENT WHERE OID = ? and KP = ? and
> EVENT_TS < ?. Stack trace: org.apache.phoenix.exception.PhoenixIOException:
> Task org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask@49c71704[Not
> completed, task = org.apache.phoenix.iterate.ParallelIterators$1@597a1aae]
> rejected from org.apache.phoenix.job.JobManager$1@7c04996f[Running, pool size
> = 20, active threads = 20, queued tasks = 5000, completed tasks = 645934]
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1433)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1297)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:52)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:107)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:127)
> at
> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
> at
> org.apache.phoenix.compile.DeleteCompiler$ServerSelectDeleteMutationPlan.execute(DeleteCompiler.java:815)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:522)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:488)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:487)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:475)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpdate(PhoenixPreparedStatement.java:206)
>
> Caused by: java.util.concurrent.RejectedExecutionException: Task
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask@49c71704[Not
> completed, task = org.apache.phoenix.iterate.ParallelIterators$1@597a1aae]
> rejected from org.apache.phoenix.job.JobManager$1@7c04996f[Running, pool size
> = 20, active threads = 20, queued tasks = 5000, completed tasks = 645934]
> at
> org.apache.phoenix.job.JobManager$InstrumentedThreadPoolExecutor$1.rejectedExecution(JobManager.java:246)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
> at
> org.apache.phoenix.job.JobManager$InstrumentedThreadPoolExecutor.execute(JobManager.java:263)
> at
> java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
> at
> org.apache.phoenix.iterate.ParallelIterators.submitWork(ParallelIterators.java:132)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1329){code}
> The configuration parameter *phoenix.query.queueSize* which has a default
> value of 5000 controls the size of the array backing the task queue. There
> are 2 issues here:
> * This query always fails in our production systems because the range scan
> of the query spans number of table regions > than 5000.
> * Moreover, other queries running concurrently with this query also fail
> with RejectedExecutionException even though those queries don't create that
> many tasks.
> I think blindly creating as many parallel scans as the table regions doesn't
> scale for huge tables. In our production we have some tables which have more
> than 100,000 regions. Simply increasing the queue size is not a scalable
> solution. Moreover, a single query should not be able to monopolize the
> entire client JVM resources ( in this case the task queue) .
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
