danny0405 commented on PR #19920: URL: https://github.com/apache/hudi/pull/19920#issuecomment-5658845705
Could you share the concrete bottleneck this addresses and some before/after measurements? I'd prefer to avoid adding this complexity unless the benefit is clear, especially the logic that derives a slot group's resources from TaskManager CPU and memory settings. That introduces additional scheduling behavior and duplicates part of Flink's memory configuration logic that we would need to maintain. If the requirement is simply to place the compaction plan operator in a separate slot-sharing group, could we keep this to a configurable group name and explicitly preserve the downstream operators' intended group? Flink supports named groups without an explicit resource profile. We could consider resource sizing separately once there is evidence that the simpler approach is insufficient. Checkpoint latency, compaction delay, and resource usage before and after the change, with the cluster resource budget held constant, would help establish whether the added complexity is worthwhile. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
