ROOBALJINDAL opened a new issue, #17949: URL: https://github.com/apache/hudi/issues/17949
### Bug Description **What happened:** We are trying to ingest 50M kafka records to a hudi table using delta streamer in one-time/batch mode. **What you expected:** We are expecting it to scale number of executors when it senses more load since we have not capped/defined maxExecutors in spark. Following are the spark configs: ``` 'spark.executor.memory=6g' 'spark.executor.cores=4' 'spark.dynamicAllocation.minExecutors=1' 'spark.executor.memoryOverhead=2g' 'spark.driver.memory=4g' 'spark.driver.cores=4' 'spark.dynamicAllocation.initialExecutors=1' ``` If we increase executor memory to 15G then it works fine but its not a ideal solution since in production, having 50M records will be a rare scenario. We are expecting hudi to scale the executors in case there are huge amount of data **Questions:** 1. On what basis it partitions the data to be ingested? 2. Is there any option to define/control number of partitions it creates? ### Environment **Hudi version:** 0.15.0-amzn-7 **Query engine:** (Spark/Flink/Trino etc) : Trino **Relevant configs:** Aws EMR 7.10 ### Logs and Stack Trace _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
