suryaprasanna commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2784546648
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on,
would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data
privacy enforcement processes.");
+ public static final ConfigProperty<Boolean>
USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+
.key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition,
listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the
executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM
error were come we can increase driver memory alone to "
Review Comment:
Yeah, I agree. Although to begin with we can provide a config flag. Which we
can remove it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
