hudi-bot opened a new issue, #16144: URL: https://github.com/apache/hudi/issues/16144
Current S3/GCS incremental jobs do not take into account sourceLimit and this causes oom issues in the production. We need to enhance the implementation of these jobs to slice the dataset based on sourceLimit specified in bytes. ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-6629 - Type: Improvement -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
