[
https://issues.apache.org/jira/browse/HUDI-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan reassigned HUDI-9067:
-----------------------------------------
Assignee: sivabalan narayanan
> Clean action sometimes falls back to spark.default.parallelism set in the env
> -----------------------------------------------------------------------------
>
> Key: HUDI-9067
> URL: https://issues.apache.org/jira/browse/HUDI-9067
> Project: Apache Hudi
> Issue Type: Improvement
> Components: cleaning
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
>
> Rarely, we notice there are 1024 (spark.default.parallelism) tasks spinning
> up for clean actions.
>
> for eg, if we try to ingest very small no of records say just 1, clean action
> executor spins up 1024 tasks even though the clean parallelism is set to
> small value.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
