[
https://issues.apache.org/jira/browse/HUDI-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-3775:
-----------------------------
Sprint: 2022/09/05, 0.13.0 Final Sprint, 0.13.0 Final Sprint 2, 0.13.0
Final Sprint 3, Sprint 2023-01-31 (was: 2022/09/05, 0.13.0 Final Sprint,
0.13.0 Final Sprint 2, 0.13.0 Final Sprint 3)
> Allow for offline compaction of MOR tables via spark streaming
> --------------------------------------------------------------
>
> Key: HUDI-3775
> URL: https://issues.apache.org/jira/browse/HUDI-3775
> Project: Apache Hudi
> Issue Type: Improvement
> Components: compaction, spark
> Reporter: Rajesh
> Assignee: Jonathan Vexler
> Priority: Critical
> Labels: easyfix, pull-request-available
> Fix For: 0.13.0
>
> Attachments: impressions.avro, run_stuff.txt, scala_commands.txt
>
>
> Currently there is no way to avoid compaction taking up a lot of resources
> when run inline or async for MOR tables via Spark Streaming. Delta Streamer
> has ways to assign resources between ingestion and async compaction but Spark
> Streaming does not have that option.
> Introducing a flag to turn off automatic compaction and allowing users to run
> compaction in a separate process will decouple both concerns.
> This will also allow the users to size the cluster just for ingestion and
> deal with compaction separate without blocking. We will need to look into
> documenting best practices for running offline compaction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
