tverdokhlebd commented on issue #1491: [SUPPORT] OutOfMemoryError during upsert 53M records URL: https://github.com/apache/incubator-hudi/issues/1491#issuecomment-615141491 Hi @lamber-ken Yes, I have resolved this problem. Firstly, the problem was in SBT memory that allocated default memory size 1GB. Then, I fixed it and set 8GB and transferring 5M records successfully finished in my local environment. Next, I have tried to transfer 30M records, but upsert failed again. I tunned *.parallelism parameters (Changed from 8 to 100) and everything went fine (20MB for each parquet file). Also, I added parameter "hoodie.memory.merge.max.size" with value "2004857600000" that you had recommended. Upsert was hanging without this parameter (It is necessary to recheck again). @lamber-ken, can you explain to me, what the parameter is?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
