koochiswathiTR commented on issue #8939:
URL: https://github.com/apache/hudi/issues/8939#issuecomment-1590461829
@ad1happy2go @soumilshah1995 ,
Below are the params my hudi writer is using to disable compaction and
cleanup.
As per my understanding, MOR writes the files to log files, once compaction
is done parquet files would be created.
But even after disabling compaction, I see parquet files getting generated.
which means compation is running? but I dont see any compactions triggered
in hudi cli
DataSourceWriteOptions.TABLE_TYPE.key() ->
DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL,
DataSourceWriteOptions.RECORDKEY_FIELD.key() -> "guid",
DataSourceWriteOptions.PARTITIONPATH_FIELD.key() -> "collectionName",
DataSourceWriteOptions.PRECOMBINE_FIELD.key() -> "operationTime",
HoodieCompactionConfig.INLINE_COMPACT_TRIGGER_STRATEGY.key() ->
CompactionTriggerStrategy.TIME_ELAPSED.name,
HoodieCompactionConfig.INLINE_COMPACT_TIME_DELTA_SECONDS.key() ->
String.valueOf(10 * 10),
HoodieCompactionConfig.SCHEDULE_INLINE_COMPACT.key() -> "false",
DataSourceWriteOptions.ASYNC_COMPACT_ENABLE.key() -> "false",
HoodieCompactionConfig.AUTO_CLEAN.key() -> "false",
HoodieCompactionConfig.AUTO_ARCHIVE.key() -> "false",
HoodieCompactionConfig.INLINE_COMPACT.key() -> "false",
HoodieCompactionConfig.ASYNC_CLEAN.key() -> "false",
HoodieCompactionConfig.CLEANER_POLICY.key() ->
HoodieCleaningPolicy.KEEP_LATEST_COMMITS.name(),
HoodieCompactionConfig.CLEANER_COMMITS_RETAINED.key() -> "40",
HoodieCompactionConfig.MIN_COMMITS_TO_KEEP.key() -> "41",
HoodieCompactionConfig.MAX_COMMITS_TO_KEEP.key() -> "42",
HoodieMetricsConfig.TURN_METRICS_ON.key() -> "true",
HoodieMetricsConfig.METRICS_REPORTER_TYPE_VALUE.key() ->
MetricsReporterType.DATADOG.name(),
HoodieMetricsDatadogConfig.API_SITE_VALUE.key() -> "US",
HoodieMetricsDatadogConfig.METRIC_PREFIX_VALUE.key() ->
"tacticalnovusingest.hudi",
HoodieMetricsDatadogConfig.API_KEY_SUPPLIER.key() ->
"com.tr.indigo.tacticalnovusingest.utils.DatadogKeySupplier",
HoodieMetadataConfig.ENABLE.key() -> "false",
HoodieWriteConfig.ROLLBACK_USING_MARKERS_ENABLE.key() -> "false",
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
