eshu commented on issue #6692: URL: https://github.com/apache/hudi/issues/6692#issuecomment-1250010727
@xushiyan It worked on 0.11.1. When I reverted version to 0.11.1, it started to work again. Sorry, I do not have a simple code snippet for this case. Settings are like ``` DataSourceWriteOptions.ASYNC_COMPACT_ENABLE -> false, DataSourceWriteOptions.HIVE_STYLE_PARTITIONING -> true, DataSourceWriteOptions.PAYLOAD_CLASS_NAME -> classOf[<class inherited from org.apache.hudi.common.model.EventTimeAvroPayload>].getName, DataSourceWriteOptions.PRECOMBINE_FIELD -> DaaS.Columns.InternalTimestamp, DataSourceWriteOptions.TABLE_TYPE -> DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL, FileSystemViewStorageConfig.INCREMENTAL_TIMELINE_SYNC_ENABLE -> false, HoodieCleanConfig.CLEANER_FILE_VERSIONS_RETAINED -> 3, HoodieCleanConfig.CLEANER_INCREMENTAL_MODE_ENABLE -> true, HoodieCleanConfig.CLEANER_POLICY -> HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS, HoodieCompactionConfig.INLINE_COMPACT -> true, HoodiePayloadConfig.EVENT_TIME_FIELD -> Columns.InternalTimestamp, HoodiePayloadConfig.ORDERING_FIELD -> Columns.InternalTimestamp, // https://github.com/apache/hudi/issues/4230 HoodieWriteConfig.MARKERS_TYPE -> MarkerType.DIRECT, // TODO Remove it when https://github.com/apache/hudi/issues/6283 is fixed: HoodieWriteConfig.ROLLBACK_USING_MARKERS_ENABLE -> false HoodieSyncConfig.META_SYNC_DATABASE_NAME -> parameters(HiveDatabaseParameter), HoodieSyncConfig.META_SYNC_TABLE_NAME -> parameters(HiveTableParameter).toLowerCase, HoodieSyncConfig.META_SYNC_ENABLED -> "true", HiveSyncConfig.HIVE_SYNC_MODE -> HiveSyncMode.GLUE.name, DataSourceWriteOptions.META_SYNC_CLIENT_TOOL_CLASS_NAME -> classOf[AwsGlueCatalogSyncTool].getName HoodieWriteConfig.UPSERT_PARALLELISM_VALUE -> 200 DataSourceWriteOptions.OPERATION -> WriteOperationType.UPSERT HoodieMetadataConfig.ENABLE -> true DataSourceWriteOptions.PARTITIONPATH_FIELD -> partition, HoodieWriteConfig.TBL_NAME -> table DataSourceWriteOptions.TABLE_NAME -> table DataSourceWriteOptions.RECORDKEY_FIELD -> recordKey ``` It breaks on the piece of code: ``` df.write .mode(saveMode) // Append .format("hudi") .options(...) .save(path) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
