nsivabalan commented on issue #3933: URL: https://github.com/apache/hudi/issues/3933#issuecomment-991847568
thanks. in general, try to ensure that each executor has enough memory to accommodate the parallel tasks that it can run based on your configs. for eg, if you let 8 tasks to run concurrently per executor and if executor has 32 GB for eg, each task may get only ~4 GB. Disk spill could happen only when there isn't sufficient memory. btw, will close the issue out if you are good w/ perf now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
