yeah, the implementation is basically relying on the request/limit concept in K8S, ...
but if there is any other cluster manager coming in future, as long as it has a similar concept , it can leverage this easily as the main logic is implemented in ResourceProfile On Mon, Dec 8, 2025 at 9:34 PM Wenchen Fan <[email protected]> wrote: > This feature is only available on k8s because it allows containers to have > dynamic resources? > > On Mon, Dec 8, 2025 at 12:46 PM Yao <[email protected]> wrote: > >> Hi Folks, >> >> We are proposing a burst-aware memoryOverhead allocation algorithm for >> Spark@K8S to improve memory utilization of spark clusters. >> Please see more details in SPIP doc >> <https://docs.google.com/document/d/1v5PQel1ygVayBFS8rdtzIH8l1el6H1TDjULD3EyBeIc/edit?tab=t.0>. >> Feedbacks and discussions are welcomed. >> >> Thanks Chao for being shepard of this feature. >> Also want to thank the authors of the original paper >> <https://www.vldb.org/pvldb/vol17/p3759-shi.pdf> from ByteDance, >> specifically Rui([email protected]) and Yixin([email protected] >> ). >> >> Thank you. >> Yao Wang >> >
