sushant-searce commented on issue #11773:
URL: https://github.com/apache/hudi/issues/11773#issuecomment-2451350472
As you can see the test cases i have shared above
Pipeline was working yesterday after disabling and then enabling the
hive_sync.
Yesterday we **disabled hive_sync** in the pipeline and it ran successfully
and **enabled it again** in **next** and that run successfully
but IN TODAYs run it failed with same error
It is very concerning is there anything
we are missing here
Sharinf Hoodie Options as well
hudiOptions:
hoodie.cleaner.commits.retained: 10
hoodie.metadata.keep.max.commits: 30
hoodie.metadata.clean.async: false
hoodie.keep.max.commits: 30
hoodie.metadata.keep.min.commits: 20
hoodie.archive.async: false
hoodie.clean.automatic: true
hoodie.finalize.write.parallelism: 200
hoodie.fail.on.timeline.archiving: false
hoodie.clean.async: false
hoodie.parquet.max.file.size: 128000000
hoodie.datasource.hive_sync.support_timestamp : true
#DISABLING METADATA TO REDUCE FREQUENT CALLS TO GCS
hoodie.metadata.enable: false
hoodie.datasource.write.hive_style_partitioning : true
hoodie.parquet.small.file.limit: 100000000
hoodie.datasource.hive_sync.enable: true
hoodie.bulkinsert.shuffle.parallelism: 200
hoodie.keep.min.commits: 11
hoodie.datasource.meta.sync.enable: true
hoodie.metadata.cleaner.commits.retained: 3
hoodie.cleaner.incremental.mode: true
hoodie.commits.archival.batch: 12
hoodie.upsert.shuffle.parallelism: 200
hive_sync.support_timestamp: true
hoodie.insert.shuffle.parallelism: 200
hoodie.metadata.compact.max.delta.commits: 10
compaction.delta_commits: 5
metadata.compaction.delta_commits: 10
hoodie.compact.inline.max.delta.commits: 5
hoodie.archive.automatic: true
hoodie.cleaner.parallelism: 200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
