srinikandi commented on issue #5054: URL: https://github.com/apache/hudi/issues/5054#issuecomment-1080048329
I have been experiencing a similar issue with Glue and Hudi 0.90. However in my case, I did a full load of a table that had close to a billion records and with 30 worker nodes, it took around 16 minutes using a partitioning key on a data column. When I tried to run an upsert operation on the same table with about 1.5 million records, the Glue job fails with no more containers available, which indicates that there is a memory issue/disk spill. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
