juanAmayaRamirez commented on issue #11712: URL: https://github.com/apache/hudi/issues/11712#issuecomment-2461095627
Hi everybody! This exact thing is happening to me, in the exact same setup Glue 4.0, Hudi 0.14.1. After about 9 "fast" upserts it hangs out on that "Preparing compaction metadata" step for about 3 hours timing out my Glue job that usually takes about 6 to 9 minutes.  It is indeed a big table with lots of files (about 10MB to 130MB) As for @ad1happy2go suggestion, my .hoodie is about 11GB uncompressed and the hfiles as well as some .log in the `.hoodie/metadata/record_index/` directory are from 80B to 400MB+  > @keerthiskating Sorry for the delay on this. Do you know how many file groups are there in your table. Are you having too many partitions? If possible, can you try running this tool and share the output - https://medium.com/@simpsons/monitoring-table-stats-22684eb70ee1 > > Also, can you check the size of your .hoodie directory. if possible, can you zip it and share to community to look into this further? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
