ad1happy2go commented on issue #11712: URL: https://github.com/apache/hudi/issues/11712#issuecomment-2461258071
@juanAmayaRamirez The issue here is it created 10 file groups in record index. Each log file is also very huge ~465 MB. So it have to merge those many big log files with the base file during compaction. For one file group it will only create one task so parallelism within file group. You can disable the metadata table once and then enable it back to recreate it and increase value of [hoodie.metadata.record.index.min.filegroup.count](https://hudi.apache.org/docs/configurations/#hoodiemetadatarecordindexminfilegroupcount) to a higher number. So it create more file groups. Although we still need to check why it is creating such large log files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
