Hi Folks, We are proposing a burst-aware memoryOverhead allocation algorithm for Spark@K8S to improve memory utilization of spark clusters. Please see more details in SPIP doc <https://docs.google.com/document/d/1v5PQel1ygVayBFS8rdtzIH8l1el6H1TDjULD3EyBeIc/edit?tab=t.0>. Feedbacks and discussions are welcomed.
Thanks Chao for being shepard of this feature. Also want to thank the authors of the original paper <https://www.vldb.org/pvldb/vol17/p3759-shi.pdf> from ByteDance, specifically Rui([email protected]) and Yixin([email protected]). Thank you. Yao Wang
