vedantkhandelwal opened a new issue, #8396:
URL: https://github.com/apache/hudi/issues/8396
We were using emr version **emr-6.5.0** , hudi version **0.9.0-amzn-1** ,
spark version = **3.1.2** , hive version = **3.1.2** , Hadoop = 3.2.1
Inorder to reduce s3 listing we enabled hudi metadata and we updated our
hudi version to 0.11.1.
emr version = **emr-6.8.0**, hudi version = **0.11.1-amzn-0** , spark
version = **3.3.0** , hive version = **3.1.3** , Hadoop = **3.2.1**
We are writing our data with following hudi params
"hoodie.metadata.enable = true
"hoodie.metadata.keep.max.commits" = 30
"hoodie.metadata.keep.min.commits" = 20
"hoodie.metadata.cleaner.commits.retained" = 3
hudiOptions.put("hoodie.metadata.compact.max.delta.commits" = 10
"hoodie.commits.archival.batch" = 10
"hoodie.archive.merge.enable" = true
"hoodie.archive.automatic" = true
"hoodie.archive.async" = true
"archive.min_commits" = 20
"archive.max_commits" = 30
"hoodie.compact.inline.max.delta.commits" = 5
"hoodie.keep.max.commits" = 30
"hoodie.cleaner.incremental.mode" = true
"hoodie.upsert.shuffle.parallelism" = 200
"hoodie.insert.shuffle.parallelism" = 200
"hoodie.bulkinsert.shuffle.parallelism" = 200
"hoodie.finalize.write.parallelism" = 200
We raised this issue in hudi's slack channel and they suggested some params
which are not working.
New Params added are
"hoodie.clean.automatic" = "true"
"hoodie.clean.async" = "true"
"hoodie.cleaner.policy" = 'KEEP_LATEST_FILE_VERSIONS'
"hoodie.cleaner.fileversions.retained" = 3
"hoodie-conf hoodie.cleaner.parallelism" = 200
'hoodie.cleaner.commits.retained' = 5
**.hoodie directory**
table_name/.hoodie/metadata/.hoodie/20230221014818001.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221014818001.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221015250196.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221015250196.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221015250196.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221020557154.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221020557154.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221020557154.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221021131984.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221021131984.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221021131984.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221022354408.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221022354408.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221022354408.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221022740328.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221022740328.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221022740328.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221022740328001.commit
table_name/.hoodie/metadata/.hoodie/20230221022740328001.compaction.inflight
table_name/.hoodie/metadata/.hoodie/20230221022740328001.compaction.requested
table_name/.hoodie/metadata/.hoodie/20230221030240650.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221030240650.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221030240650.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221031707194.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221031707194.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221031707194.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221034804323.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221034804323.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221034804323.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221034804323002.clean
table_name/.hoodie/metadata/.hoodie/20230221034804323002.clean.inflight
table_name/.hoodie/metadata/.hoodie/20230221034804323002.clean.requested
table_name/.hoodie/metadata/.hoodie/20230221035949496.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221035949496.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221035949496.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221042630915.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221042630915.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221042630915.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221043148964.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221043148964.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221043148964.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221064658334.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221064658334.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221064658334.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221065326918.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221065326918.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221065326918.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221072123748.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221072123748.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221072123748.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221075031944.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221075031944.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221075031944.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221075031944001.commit
table_name/.hoodie/metadata/.hoodie/20230221075031944001.compaction.inflight
table_name/.hoodie/metadata/.hoodie/20230221075031944001.compaction.requested
table_name/.hoodie/metadata/.hoodie/20230221080611224.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221080611224.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221080611224.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221082840586.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221082840586.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221082840586.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221084046851.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221084046851.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221084046851.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221084046851002.clean
table_name/.hoodie/metadata/.hoodie/20230221084046851002.clean.inflight
table_name/.hoodie/metadata/.hoodie/20230221084046851002.clean.requested
table_name/.hoodie/metadata/.hoodie/20230221085505588.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221085505588.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221085505588.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221091907897.deltacommit
table_name/.hoodie/metadata/.hoodie/20230221091907897.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230221091907897.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230221092550524.deltacommit
.
.
.
table_name/.hoodie/metadata/.hoodie/20230405032223987.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405032223987.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405032223987.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405034010492.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405034010492.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405034010492.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405034227890.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405034227890.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405034227890.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405040009502.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405040009502.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405040009502.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405040254298.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405040254298.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405040254298.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405042008694.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405042008694.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405042008694.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405042223314.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405042223314.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405042223314.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405042223314001.commit
table_name/.hoodie/metadata/.hoodie/20230405042223314001.compaction.inflight
table_name/.hoodie/metadata/.hoodie/20230405042223314001.compaction.requested
table_name/.hoodie/metadata/.hoodie/20230405044010890.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405044010890.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405044010890.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405044307030.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405044307030.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405044307030.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405050008617.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405050008617.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405050008617.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405050008617002.clean
table_name/.hoodie/metadata/.hoodie/20230405050008617002.clean.inflight
table_name/.hoodie/metadata/.hoodie/20230405050008617002.clean.requested
table_name/.hoodie/metadata/.hoodie/20230405050226782.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405050226782.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405050226782.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405052008865.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405052008865.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405052008865.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405052240790.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405052240790.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405052240790.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405054007052.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405054007052.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405054007052.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405054224377.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405054224377.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405054224377.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405060007239.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405060007239.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405060007239.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405060245083.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405060245083.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405060245083.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405060245083001.commit
table_name/.hoodie/metadata/.hoodie/20230405060245083001.compaction.inflight
table_name/.hoodie/metadata/.hoodie/20230405060245083001.compaction.requested
table_name/.hoodie/metadata/.hoodie/20230405062008329.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405062008329.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405062008329.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405062315451.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405062315451.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405062315451.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405064008288.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405064008288.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405064008288.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405064008288002.clean
table_name/.hoodie/metadata/.hoodie/20230405064008288002.clean.inflight
table_name/.hoodie/metadata/.hoodie/20230405064008288002.clean.requested
table_name/.hoodie/metadata/.hoodie/20230405064240283.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405064240283.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405064240283.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405070008852.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405070008852.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405070008852.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405070238777.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405070238777.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405070238777.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405072008382.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405072008382.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405072008382.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405072225489.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405072225489.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405072225489.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405074008194.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405074008194.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405074008194.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405074225290.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405074225290.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405074225290.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405074225290001.commit
table_name/.hoodie/metadata/.hoodie/20230405074225290001.compaction.inflight
table_name/.hoodie/metadata/.hoodie/20230405074225290001.compaction.requested
table_name/.hoodie/metadata/.hoodie/20230405080008668.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405080008668.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405080008668.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405080244404.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405080244404.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405080244404.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405082008355.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405082008355.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405082008355.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405082008355002.clean
table_name/.hoodie/metadata/.hoodie/20230405082008355002.clean.inflight
table_name/.hoodie/metadata/.hoodie/20230405082008355002.clean.requested
table_name/.hoodie/metadata/.hoodie/20230405082229381.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405082229381.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405082229381.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405084006462.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405084006462.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405084006462.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405084234497.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405084234497.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405084234497.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405090013710.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405090013710.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405090013710.deltacommit.requested
table_name/.hoodie/metadata/.hoodie/20230405090329629.deltacommit
table_name/.hoodie/metadata/.hoodie/20230405090329629.deltacommit.inflight
table_name/.hoodie/metadata/.hoodie/20230405090329629.deltacommit.requested
like this thousands of files are present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
