Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Thanks William for the info. The current model of Spark on k8s has certain drawbacks with pod based scheduling as I tested it on Google Kubernetes Cluster (GKE). The constraint seems to be that you can fit one Spark executor pod per Kubernetes node and from my tests you don't seem to be able to

Re: About contribution

(There is no project chat) See https://spark.apache.org/contributing.html On Tue, Jan 4, 2022 at 11:42 PM Dennis Jung wrote: > Hello, I hope this is not a silly question. > (I couldn't find any chat room on spark project, so asking on mail) > > It has been about a year since using spark in

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi Mich, Curious what do you mean “The constraint seems to be that you can fit one Spark executor pod per Kubernetes node and from my tests you don't seem to be able to allocate more than 50% of RAM on the node to the container", Would you help to explain a bit? Asking this because there could be

Re: Spark 3.2 - ReusedExchange not present in join execution plan

Just thought I'd do a quick bump and add the dev mailing list - in case there is some insight there Feels like this should be categorized as a bug for spark 3.2.0 On Wed, Dec 29, 2021 at 5:25 PM Abdeali Kothari wrote: > Hi, > I am using pyspark for some projects. And one of the things we are

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi Bo, Thanks for the info. Let me elaborate: In theory you can set the number of executors to multiple values of Nodes. For example if you have a three node k8s cluster (in my case Google GKE), you can set the number of executors to 6 and end up with six executors queuing to start but

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Do we want to move the SPIP forward to a vote? It seems like we're mostly agreeing in principle? On Wed, Jan 5, 2022 at 11:12 AM Mich Talebzadeh wrote: > Hi Bo, > > Thanks for the info. Let me elaborate: > > In theory you can set the number of executors to multiple values of Nodes. > For

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 non-binding view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 for SPIP According our production experience, the default scheduler isn't meeting prod requirements on K8S, and such effort of integrating with batch-native schedulers makes running Spark natively on K8S much easier for users. Thanks, Bowen On Wed, Jan 5, 2022 at 11:52 AM Mich Talebzadeh

[VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

Hi all, I’d like to start a vote for SPIP: "Support Customized Kubernetes Schedulers Proposal" The SPIP is to support customized Kubernetes schedulers in Spark on Kubernetes. Please also refer to: - Previous discussion in dev mailing list: [DISCUSSION] SPIP: Support Volcano/Alternative

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 thanks Yikun for initiating this! Look forward to more progress made together by the community! On Wed, Jan 5, 2022 at 4:52 PM Weiwei Yang wrote: > +1 > > I think it is in a good shape to move forward > > On Wed, Jan 5, 2022 at 3:00 PM Bowen Li wrote: > >> +1 for SPIP >> >> According our

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

@Holden Karau Thanks for reminder, I will send the vote mail soon. and thanks for all helps on discussion and design review. Regards, Yikun Holden Karau 于2022年1月6日周四 03:16写道： > Do we want to move the SPIP forward to a vote? It seems like we're mostly > agreeing in principle? > > On Wed,

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (non-binding). Thanks Yikun for putting it all together On Wed, Jan 5, 2022 at 5:07 PM Yikun Jiang wrote: > Hi all, > > I’d like to start a vote for SPIP: "Support Customized Kubernetes > Schedulers Proposal" > > The SPIP is to support customized Kubernetes schedulers in Spark on >

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 I think it is in a good shape to move forward On Wed, Jan 5, 2022 at 3:00 PM Bowen Li wrote: > +1 for SPIP > > According our production experience, the default scheduler isn't meeting > prod requirements on K8S, and such effort of integrating with batch-native > schedulers makes running

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (non-binding) On Wed, Jan 5, 2022 at 5:09 PM Chaoran Yu wrote: > +1 (non-binding). Thanks Yikun for putting it all together > > On Wed, Jan 5, 2022 at 5:07 PM Yikun Jiang wrote: > >> Hi all, >> >> I’d like to start a vote for SPIP: "Support Customized Kubernetes >> Schedulers Proposal" >>

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (binding) On Wed, Jan 5, 2022 at 5:31 PM William Wang wrote: > +1 (non-binding) > > Yikun Jiang 于2022年1月6日周四 09:07写道： > >> Hi all, >> >> I’d like to start a vote for SPIP: "Support Customized Kubernetes >> Schedulers Proposal" >> >> The SPIP is to support customized Kubernetes schedulers in

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (non-binding) On Wed, Jan 5, 2022 at 11:01 PM Holden Karau wrote: > +1 (binding) > > On Wed, Jan 5, 2022 at 5:31 PM William Wang > wrote: > >> +1 (non-binding) >> >> Yikun Jiang 于2022年1月6日周四 09:07写道： >> >>> Hi all, >>> >>> I’d like to start a vote for SPIP: "Support Customized Kubernetes