viirya commented on PR #58763: URL: https://github.com/apache/spark/pull/58763#issuecomment-5650009502
One policy question beyond the reclamation mechanics: “optional” establishes that a buffer can be discarded without breaking correctness, but it does not establish that discarding it is cheap or preferable to slowing another task. For example, A may have already paid the I/O cost for a prefetched buffer that it will consume shortly. Reclaiming it for B can increase A's latency and duplicate I/O. If A belongs to a more latency-sensitive or higher-priority workload, the tradeoff may be undesirable even though both tasks remain correct. This appears to introduce an executor-wide policy where ordinary allocations take precedence over other tasks' optional reservations, without considering workload priority or reclamation cost. Is that explicitly the intended contract? How should a consumer decide whether its buffers are suitable for such revocable reservations? It would help to evaluate this with the first consumer, measuring both the requesting task's benefit and the owner's cost, including repeated reclamation and tail latency. That would also clarify whether cross-task reclamation should apply universally or only to a more narrowly defined class of disposable work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
