cshuo commented on issue #20338: URL: https://github.com/apache/hudi/issues/20338#issuecomment-6186290234
Thanks @skywalker0618 for sharing the measurements. +1 to @danny0405’s suggestions. The results make a good case for batching in this workload, and I’m supportive of opening a PR. For the bounded parallelism, please consider reusing HoodieFlinkEngineContext.map(..., parallelism), which already provides local execution with a dedicated pool. Keeping the initial PR focused on filesystem-backed batching sounds reasonable. We can explore metadata-table-backed checks separately, and track the duplicate-discovery race independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
