Hi Marcin, This is really great work, and I agree that we should cross build against different versions of Spark. If we have a cross building infrastructure set up, we can be more flexible and adapt to different versions of external dependencies as well, such as different versions of Hadoop and Elasticsearch.
I cannot see any negative issues impacting the release cycle, besides requiring more work to generate more versions of binaries. One thing that we need to be careful of would be making sure that the core artifact, which all engine templates depend on, be always compatible with different cross build configurations of PredictionIO. I think with Spark 2 being the starting point of Scala 2.11 adoption we should be pretty safe to assume supporting only Spark < 2 for Scala 2.10 artifacts and Spark >= 2 for Scala 2.11 artifacts. How do others think? Regards, Donald On Mon, Aug 15, 2016 at 2:42 PM, Marcin Ziemiński <ziem...@gmail.com> wrote: > Hi, > > The recent version of Spark - 2.0.0 comes with many changes and > improvements. The same is for the related Spark MLlib . > -> Spark 2.0.0 release notes > <https://spark.apache.org/releases/spark-release-2-0-0.html> > -> Spark 2.0.0 overview > <https://databricks.com/blog/2016/07/26/introducing-apache-spark-2-0.html> > -> Announcement: DataFrame-based API is primary API > <http://spark.apache.org/docs/latest/ml-guide.html# > announcement-dataframe-based-api-is-primary-api> > > The changes are quite serious as they move the usage more towards DataSets > instead of RDDs and offer many performance improvements. > The current version of Spark that PredictionIO is built with is quite old > 1.4.0 and soon some people will want to make use of the latest features in > new templates. > What is more, we could come up with a new kind of workflow different from > DASE fitting better Spark ML pipelines, which would be a part of some next > future release and would set a different direction of the project. > > I am aware that simply upgrading the version of Spark would break a lot of > existing projects and cause many inconveniences to many people. Besides it > requires bumping Scala version to 2.11. Having a separate branch for the > new version is rather not maintainable, therefore I think that we could > cross-build the project against two versions of Scala - 2.10 and 2.11, > where the 2.11 version would be specifically for the new version of Spark. > Any sources not compatible with two versions at the same time, could be > split up between separate version-specific directories: > src/main/scala-2.10/, > src/main/scala-2.11/. Sbt as of 13.8 should not have problems with > that -> merged > PR <https://github.com/sbt/sbt/pull/1799> > Such setup would make it possible to work on fixes and features available > in both Scala versions, but more importantly it would let us add new > functionalities specific to the latest Spark releases without breaking the > old build. > > I have already tried to update Spark in PredictionIO in my own fork and I > have managed to get a working version without modifying too much code -> > diff > here > <https://github.com/apache/incubator-predictionio/ > compare/develop...Ziemin:upgrade#diff-fdc3abdfd754eeb24090dbd90aeec2ce> > . > Both unit-tests and integration tests were successful. > > What do you think? Would it affect the current release cycle in a negative > way? Maybe someone has a better idea on how to perform this upgrade. > Sticking to Spark 1.x version forever is probably not an option and the > sooner we upgrade the better. > > Regards, > Marcin >
