[
https://issues.apache.org/jira/browse/HUDI-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui An updated HUDI-5303:
-------------------------
Description: If there are sufficient resources in the clustering job, some
clustering groups sometimes could still waits to be triggered, we use
forkJoinPool to submit these jobs, and it's also difficult for clients to
adjust this configure(--conf
spark.driver.extraJavaOptions=-Djava.util.concurrent.ForkJoinPool.common.parallelism),
and it could also affect other tasks using the forkJoinPool, so instead, we
introduce a new threadPool to control the submitting job parallelism for the
clustering.
> Allow users to control the concurrency to submit jobs in clustering
> -------------------------------------------------------------------
>
> Key: HUDI-5303
> URL: https://issues.apache.org/jira/browse/HUDI-5303
> Project: Apache Hudi
> Issue Type: Improvement
> Components: clustering, spark
> Reporter: Hui An
> Assignee: Hui An
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.13.0
>
>
> If there are sufficient resources in the clustering job, some clustering
> groups sometimes could still waits to be triggered, we use forkJoinPool to
> submit these jobs, and it's also difficult for clients to adjust this
> configure(--conf
> spark.driver.extraJavaOptions=-Djava.util.concurrent.ForkJoinPool.common.parallelism),
> and it could also affect other tasks using the forkJoinPool, so instead, we
> introduce a new threadPool to control the submitting job parallelism for the
> clustering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
