xushiyan commented on issue #3697: URL: https://github.com/apache/hudi/issues/3697#issuecomment-924644754
looks like there are too many partitions (1100) for a dataset < 2GB. This resulted in many small files. Some suggestions 1) reduce number of partitions to <100 2) double check number executors were created. c5.xlarge is a very small machine also with low memory to core ratio. You set 100 executors but each requires 5g memory 3) also tune parallelism settings in Hudi for the insert and upsert -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
