Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be uptodate with latest version of Spark. This brings us a challenge to support our consumers until they move to Spark 2.0.Based on some brainstorming, I can propose following options to keep SystemML being supported on latest Spark version quickly.
Supporting SystemML on Spark 1.x We can continue to support SystemML on Spark 1.x code base for short period of time by adding fixes and features on main branch. We will release SystemML with support to Spark 1.x next version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st 2016) Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark 2.0, we can make SystemML on Spark 2.0 immediately based on branch created on top of latest master branch code. Glen has some prototype code to transform SystemML code to be compatible with Spark 2.0, he can merge his code with new branch targeted to support SystemML on Spark 2.0 This would be "Preview" version code, and we can update it on frequent basis (on bi-monthy basis). Supporting SystemML on Spark 2.0 We will have full support of SystemML on Spark 2.0 before end of year 2016. We will formalize release date by end of Sept 2016. At the same time we will discuss if we can move support of SystemML on Spark 1.x to maintenance mode (Only required bug fixes will be merged from main branch) or we need to support both SystemML on Spark 2.0 and Spark 1.x for some additional time. SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep Learning (Library of Network layers?) - Frame - New MLContext API - Python DSL integration (Preview) - Compressed Linear Algebra (Preview) - Hydra R integration - New Algorithms (?) 0.12 (Spark 2.0) (Targeted to 4Q 2016) - GPU support (Local mode/Distributed mode?) - New Algorithms (?) Please feel free to comment on support and roadmap points. -Arvind
