Re: code freeze and branch cut for Apache Spark 2.4

++ssuchter (who kindly set up the initial k8s builds while i hammered on the backend) while i'm pretty confident (read: 99%) that the pull request builds will work on the new ubuntu workers: 1) i'd like to do more stress testing of other spark builds (in progress) 2) i'd like to reimage more

Re: code freeze and branch cut for Apache Spark 2.4

The PR for SparkR support on the kube back-end is completed, but waiting for Shane to make some tweaks to the CI machinery for full testing support. If the code freeze is being delayed, this PR could be merged as well. On Fri, Jul 6, 2018 at 9:47 AM, Reynold Xin wrote: > FYI 6 mo is coming up

Re: code freeze and branch cut for Apache Spark 2.4

I agree that looking at it from the pov of "code paths where isBarrier tests were introduced" seems right. >From pr-21758 (the one already merged) there are 13 files touched under core/src/main/scala/org/apache/spark/scheduler/, although most of

Re: code freeze and branch cut for Apache Spark 2.4

I still would like to do more review on barrier mode changes, but from what I've seen so far I agree. I dunno if it'll really be ready for use, but it should not pose much risk for code which doesn't touch the new features. of course, every change has some risk, especially in the scheduler which

Re: code freeze and branch cut for Apache Spark 2.4

Speaking of the code from hydrogen PRs, actually we didn't remove any of the existing logic, and I tried my best to hide almost all of the newly added logic behind a `isBarrier` tag (or something similar). I have to add some new variables and new methods to the core code paths, but I think they

Re: code freeze and branch cut for Apache Spark 2.4

Sorry for late response on Hydrogen discussions! I was traveling last week. On Tue, Jul 31, 2018 at 1:20 PM Reynold Xin wrote: > I actually totally agree that we should make sure it should have no impact > on existing code if the feature is not used. > > > On Tue, Jul 31, 2018 at 1:18 PM Erik

Re: [build system] DOWNTIME jenkins unreachable overnight

the UPS has been replaced, and you can now access the wonderful entity known as jenkins via the internet superhighway! shane (who only really showed up early to work and didn't actually help replace said UPS) On Tue, Jul 31, 2018 at 5:14 PM, shane knapp wrote: > our building is finally

Re: [DISCUSS] Multiple catalog support

For the first question. This is what we already supported. A data source can implement `ReadSupportProvider` (based on my API improvement ) so that it can create `ReadSupport` by reflection. I agree

Re: Writing file

Thank you cloud0fan. That's really helpful. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org