Re: Integration testing and Scheduler Backends

2018-01-09 Thread Reynold Xin
If we can actually get our acts together and have integration tests in Jenkins (perhaps not run on every commit but can be run weekly or pre-release smoke tests), that'd be great. Then it relies less on contributors manually testing. On Tue, Jan 9, 2018 at 8:09 AM, Timothy Chen

Re: Integration testing and Scheduler Backends

2018-01-08 Thread Timothy Chen
2) will be ideal but given the velocity of main branch, what Mesos ended up doing was simply having a separate repo since it will take too long to merge back to main. We ended up running it pre-release (or major PR merged) and not on every PR, I will also comment on asking users to run it. We

Re: Integration testing and Scheduler Backends

2018-01-08 Thread Anirudh Ramanathan
I meant uncommon in Spark, i.e. in the other scheduler backends AFAIK. (2) is what we use within the Kubernetes project for example, and is preferred in general. On Mon, Jan 8, 2018 at 10:45 PM, Felix Cheung wrote: > How would (2) be uncommon elsewhere? > > On Mon, Jan

Re: Integration testing and Scheduler Backends

2018-01-08 Thread Felix Cheung
How would (2) be uncommon elsewhere? On Mon, Jan 8, 2018 at 10:16 PM Anirudh Ramanathan wrote: > This is with regard to the Kubernetes Scheduler Backend and scaling the > process to accept contributions. Given we're moving past upstreaming > changes from our fork, and into

Integration testing and Scheduler Backends

2018-01-08 Thread Anirudh Ramanathan
This is with regard to the Kubernetes Scheduler Backend and scaling the process to accept contributions. Given we're moving past upstreaming changes from our fork, and into getting *new* patches, I wanted to start this discussion sooner than later. This is more of a post-2.3 question - not