nsivabalan opened a new issue, #17476: URL: https://github.com/apache/hudi/issues/17476
### Bug Description **What happened:** With streaming writes to metadata table feature, looks like we changed the write dag to support streaming writes to data table and metadata table. In 1.1.0, looks like the writes to data table is not happening in the intended stage, but the coalesced stage which has less number of tasks. For every 5000 data table tasks, we are spinning up 1 task. When release testing was done, EMR spark was used and this is not an issue in EMR spark. But other spark vendors may not have the optimization as EMR spark and hence could see the impact. **What you expected:** Writes to data table happens in the stage as intended. if we spin up 100 write handles, 100 spark tasks should be used to write the data. **Steps to reproduce:** 1. Use spark to write to hudi table with partitioned dataset using EKS spark or any other non EMR spark runtimes. 2. 3. ### Environment **Hudi version:** 1.1.- **Query engine:** (Spark/Flink/Trino etc) Spark **Relevant configs:** ### Logs and Stack Trace _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
