> In any way, I'd like to say that the root cause of the difference is those scheduler designs instead of Apache Spark itself. For example, Apache YuniKorn doesn't force us to add a new dependency at all while Volcano did.
This makes sense! > In these day, I prefer and invest more Apache YuniKorn and, for Apache Spark 4, we are going to add and focus more on K8s native features like `Scheduling Gate` additionally. This is very useful to know! Are there any tickets/SPIPs for this yet? Thanks for sharing these bits and kind regards Santosh On Wed, Aug 23, 2023 at 12:10 AM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > Of course, we can make Apache Spark distribution bigger and bigger, but > I'm a little neutral about Volcano. > > In any way, I'd like to say that the root cause of the difference is those > scheduler designs instead of Apache Spark itself. For example, Apache > YuniKorn doesn't force us to add a new dependency at all while Volcano did. > > > It would useful to support volcano in spark distro itself just like > yunikorn. > In these day, I prefer and invest more Apache YuniKorn and, for Apache > Spark 4, we are going to add and focus more on K8s native features like > `Scheduling Gate` additionally. > > Dongjoon. >
