irodriguezclaveria commented on PR #6426: URL: https://github.com/apache/paimon/pull/6426#issuecomment-4562074883
Hi @JingsongLi , Let me answer your points: - **Thread pool design**: The executor was already global (static final), but you're right that it was a dedicated one. Switched to the shared FileOperationThreadPool and removed the dedicated REFRESH_EXECUTOR and its static initializer. - **Race condition:** Yes, exactly that window. The ideal is to ensure the threshold is large enough that the refresh has time to complete before existing buckets saturate. I add a WARN log if the value is below the recommended 5–10% of target-row-num and in the description of the parameter let me know if you considerer that this is enough or I can do a more stricly validattion. - **621 additions**: Most of the 621 lines are tests and auto-generated docs. The actual feature logic is in paimon-core (PartitionIndex), and both engines are thin wrappers that just forward the new options. Only two production sites construct HashBucketAssigner — HashBucketAssignerOperator (Flink) and BucketProcessor (Spark) — and both pass the new parameters identically. - **Timeout (5 min)**: Sorry it was a mistake in my documentation, corrected. I considered adding an application-level timeout first, but Future.cancel(true) doesn't interrupt blocked native I/O — it would mark the future as done (allowing retries) but the underlying thread would still be held by FileIO until its timeout fires. Happy to add it as an opt-in if you'd prefer; otherwise I'll just document this in the javadoc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
