Re: Integration testing and Scheduler Backends

2) will be ideal but given the velocity of main branch, what Mesos ended up doing was simply having a separate repo since it will take too long to merge back to main. We ended up running it pre-release (or major PR merged) and not on every PR, I will also comment on asking users to run it. We

Re: Integration testing and Scheduler Backends

I meant uncommon in Spark, i.e. in the other scheduler backends AFAIK. (2) is what we use within the Kubernetes project for example, and is preferred in general. On Mon, Jan 8, 2018 at 10:45 PM, Felix Cheung wrote: > How would (2) be uncommon elsewhere? > > On Mon, Jan

Re: Integration testing and Scheduler Backends

How would (2) be uncommon elsewhere? On Mon, Jan 8, 2018 at 10:16 PM Anirudh Ramanathan wrote: > This is with regard to the Kubernetes Scheduler Backend and scaling the > process to accept contributions. Given we're moving past upstreaming > changes from our fork, and into

Integration testing and Scheduler Backends

This is with regard to the Kubernetes Scheduler Backend and scaling the process to accept contributions. Given we're moving past upstreaming changes from our fork, and into getting *new* patches, I wanted to start this discussion sooner than later. This is more of a post-2.3 question - not

Re: Branch 2.3 is cut

Hello everyone, Just a quick update on the release. There are currently 2 correctness blockers (SPARK-22984 and SPARK-22982 ) that are targeted against 2.3.0. We'll go ahead and create an RC as

Re: Kubernetes backend and docker images

Think we can allow for different images and default to them being the same. Apologize if I missed that as being the original intention though. -Matt Cheah On 1/8/18, 1:45 PM, "Marcelo Vanzin" wrote: On Mon, Jan 8, 2018 at 1:39 PM, Matt Cheah

Re: Kubernetes backend and docker images

On Mon, Jan 8, 2018 at 1:39 PM, Matt Cheah wrote: > We would still want images to be able to be uniquely specified for the > driver vs. the executors. For example, not all of the libraries required on > the driver may be required on the executors, so the user would want to >

Re: Kubernetes backend and docker images

// Fixing Anirudh's email address From: Matt Cheah Sent: Monday, January 8, 2018 1:39:12 PM To: Anirudh Ramanathan; Felix Cheung Cc: 蒋星博; Marcelo Vanzin; dev; Timothy Chen Subject: Re: Kubernetes backend and docker images We would still want images to be able to

Re: Kubernetes backend and docker images

We would still want images to be able to be uniquely specified for the driver vs. the executors. For example, not all of the libraries required on the driver may be required on the executors, so the user would want to specify a different custom driver image from their custom executor image.

Re: Kubernetes backend and docker images

+matt +tim For reference - here's our previous thread on this dockerfile unification problem - https://github.com/apache-spark-on-k8s/spark/pull/60 I think this approach should be acceptable from both the customization and visibility perspectives. On Mon, Jan 8, 2018 at 9:40 AM, Anirudh

Re: Kubernetes backend and docker images

+1 We discussed some alternatives early on - including using a single dockerfile and different spec.container.command and spec.container.args from the Kubernetes driver/executor specification (which override entrypoint in docker). No reason that won't work also - except that it reduced the