koochiswathiTR commented on issue #6606:
URL: https://github.com/apache/hudi/issues/6606#issuecomment-1247810081
We have solved this issue by automatic creation of Dynamodb table and table
is created with partition key (key)
We could able to setup with out aws keys.
With multi writer setup, we are running two EMR clusters which acquires lock
with one hudi table
We are facing ConcurrentModificationException
Failed after max retries with exception
java.util.ConcurrentModificationException: Cannot resolve conflicts for
overlapping writes.
Below are the Hudi Config params
DataSourceWriteOptions.TABLE_TYPE.key() ->
DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL,
DataSourceWriteOptions.RECORDKEY_FIELD.key() -> "Id",
DataSourceWriteOptions.PARTITIONPATH_FIELD.key() -> "coll",
DataSourceWriteOptions.PRECOMBINE_FIELD.key() -> "operationTime",
//TODO: Sync with glue. Dong has working.
DataSourceWriteOptions.HIVE_SYNC_ENABLED.key() -> "true",
DataSourceWriteOptions.HIVE_PARTITION_FIELDS.key() -> "coll",
DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS.key() ->
classOf[MultiPartKeysValueExtractor].getName,
DataSourceWriteOptions.HIVE_SYNC_MODE.key() -> "hms",
DataSourceWriteOptions.HIVE_USE_JDBC.key() -> "false",
HoodieCompactionConfig.INLINE_COMPACT_TRIGGER_STRATEGY.key() ->
CompactionTriggerStrategy.TIME_ELAPSED.name,
HoodieCompactionConfig.INLINE_COMPACT_TIME_DELTA_SECONDS.key() ->
String.valueOf(60 * 60),
HoodieCompactionConfig.CLEANER_COMMITS_RETAINED.key() -> "2184",
HoodieCompactionConfig.ASYNC_CLEAN.key() -> "false",
HoodieCompactionConfig.INLINE_COMPACT.key() -> "true",
HoodieCompactionConfig.MIN_COMMITS_TO_KEEP.key() -> "2185",
HoodieCompactionConfig.MAX_COMMITS_TO_KEEP.key() -> "3494",
HoodieMetricsConfig.TURN_METRICS_ON.key() -> "true",
HoodieMetricsConfig.METRICS_REPORTER_TYPE_VALUE.key() ->
MetricsReporterType.DATADOG.name(),
HoodieMetricsDatadogConfig.API_SITE_VALUE.key() -> "US",
HoodieMetricsDatadogConfig.METRIC_PREFIX_VALUE.key() ->
"tacticalnovusingest.qa.hudi",
HoodieMetricsDatadogConfig.API_KEY_SUPPLIER.key() -> "XXXX",
HoodieMetadataConfig.ENABLE.key() -> "true",
HoodieMetadataConfig.CLEANER_COMMITS_RETAINED.key() -> "2184",
HoodieMetadataConfig.MIN_COMMITS_TO_KEEP.key() -> "2185",
HoodieMetadataConfig.MAX_COMMITS_TO_KEEP.key() -> "3494",
HoodieWriteConfig.ROLLBACK_USING_MARKERS_ENABLE.key() -> "false",
HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key()->"optimistic_concurrency_control",
HoodieCompactionConfig.FAILED_WRITES_CLEANER_POLICY.key()->"LAZY",
HoodieLockConfig.LOCK_ACQUIRE_NUM_RETRIES.key()->"3000",
HoodieLockConfig.LOCK_ACQUIRE_CLIENT_NUM_RETRIES.key()->"5",
HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key()->"org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider",
DynamoDbBasedLockConfig.DYNAMODB_LOCK_TABLE_NAME.key()->"XXXX",
DynamoDbBasedLockConfig.DYNAMODB_LOCK_PARTITION_KEY.key()->"lock",
DynamoDbBasedLockConfig.DYNAMODB_LOCK_REGION.key()->"us-east-1",
DynamoDbBasedLockConfig.DYNAMODB_ENDPOINT_URL.key()->
RegionUtils.getRegion("us-east-1").getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX)
//"dynamodb.us-east-1.amazonaws.com"
Please help on this.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
