keerthiskating commented on issue #11540:
URL: https://github.com/apache/hudi/issues/11540#issuecomment-2250767531
I am facing similar issue. I am using Glue 4.0 with Hudi 0.14. The very
first run when the table does not exist in Glue catalog works fine. However,
over time, the glue job hangs and never finishes. I see errors similar to below.
`"Failure Reason": "Error checking presence of partition meta file for s3a:`
Since multiple jobs are writing to same table, I have enabled locks and
below is the hudi config I am using
```
"hudi_options" : {
'hoodie.table.cdc.enabled':'true',
'hoodie.table.cdc.supplemental.logging.mode':
'data_before_after',
'hoodie.datasource.write.recordkey.field': 'uuid',
'hoodie.datasource.write.keygenerator.class':
'org.apache.hudi.keygen.ComplexKeyGenerator',
'hoodie.table.name': "transact_table",
'hoodie.datasource.write.table.name': "transact_table",
'hoodie.datasource.hive_sync.table': "transact_table",
'hoodie.datasource.hive_sync.database': "default",
'hoodie.datasource.write.partitionpath.field': 'a,b,c',
'hoodie.datasource.hive_sync.partition_fields': 'a,b,c',
'hoodie.datasource.write.hive_style_partitioning': 'true',
'hoodie.datasource.hive_sync.enable': 'true',
'hoodie.datasource.hive_sync.partition_extractor_class':
'org.apache.hudi.hive.MultiPartKeysValueExtractor',
'hoodie.datasource.write.storage.type': 'COPY_ON_WRITE',
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.precombine.field': 'uuid',
'hoodie.datasource.hive_sync.use_jdbc': 'false',
'hoodie.datasource.hive_sync.mode': 'hms',
'hoodie.datasource.hive_sync.support_timestamp': 'true',
'hoodie.write.concurrency.mode':
'OPTIMISTIC_CONCURRENCY_CONTROL',
'hoodie.write.lock.provider':
'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider',
'hoodie.write.lock.dynamodb.table': 'hudi_locks_table',
'hoodie.write.lock.dynamodb.region': 'us-west-2',
'hoodie.write.lock.dynamodb.endpoint_url':
'dynamodb.us-west-2.amazonaws.com',
'hoodie.write.lock.dynamodb.partition_key': "transact_table",
'hoodie.cleaner.policy.failed.writes':'LAZY',
'hoodie.keep.min.commits': 10,
'hoodie.keep.max.commits': 20,
'hoodie.cleaner.commits.retained': 9
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
