Xiangrui and I are leading an effort to implement a highly desirable feature, Barrier Execution Mode. https://issues.apache.org/jira/browse/SPARK-24374. This introduces a new scheduling model to Apache Spark so users can properly embed distributed DL training as a Spark stage to simplify the distributed training workflow. The prototype has been demoed in the Spark Summit keynote. This new feature got a very positive feedback from the whole community. The design doc and pull requests got more comments than we initially anticipated. We want to finish this feature in the upcoming release, Spark 2.4. Would it be possible to have an extension of code freeze for a week?
Thanks, Xingbo 2018-07-07 0:47 GMT+08:00 Reynold Xin <r...@databricks.com>: > FYI 6 mo is coming up soon since the last release. We will cut the branch > and code freeze on Aug 1st in order to get 2.4 out on time. > >
