GitHub user vanzin opened a pull request:
https://github.com/apache/incubator-livy/pull/112
[LIVY-511][LIVY-512] Remove support for old Spark, Scala versions.
This change restricts Livy support to Spark 2.2+ and Scala 2.11. Both
changes
are made together because by supporting Spark 2.2+ only, it becomes
impossible
to test Scala 2.10.
As part of the change, a lot of code that used reflection to support
different
versions of Spark could be cleaned up and directly call Spark APIs.
The Scala 2.10 parts of the builds also have been removed, but the actual
support for building and running with different Scala versions (and related
tests) have been left as is. This will allow us to support 2.12 in the
future.
This change intentionally does not touch the public API (the "api/" module).
There are things that could be cleaned up now that Spark 1.x is not
supported,
but that would mean an API breakage so I chose to leave those alone for now.
The test matrix and build profiles have also been simplified a lot. There
are
now two profiles to choose from (for Spark 2.2 and 2.3); integration tests
can
be run against a different version of Spark by running just the integration
test module with the desired profile.
Tested with Spark 2.2 and 2.3, and also by building against 2.2 and running
integration tests against 2.3.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/incubator-livy LIVY-511
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-livy/pull/112.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #112
----
commit 6459626488a050935e25cceafe4e8b22e91e3012
Author: Marcelo Vanzin <vanzin@...>
Date: 2018-09-20T18:39:23Z
[LIVY-511][LIVY-512] Remove support for old Spark, Scala versions.
This change restricts Livy support to Spark 2.2+ and Scala 2.11. Both
changes
are made together because by supporting Spark 2.2+ only, it becomes
impossible
to test Scala 2.10.
As part of the change, a lot of code that used reflection to support
different
versions of Spark could be cleaned up and directly call Spark APIs.
The Scala 2.10 parts of the builds also have been removed, but the actual
support for building and running with different Scala versions (and related
tests) have been left as is. This will allow us to support 2.12 in the
future.
This change intentionally does not touch the public API (the "api/" module).
There are things that could be cleaned up now that Spark 1.x is not
supported,
but that would mean an API breakage so I chose to leave those alone for now.
The test matrix and build profiles have also been simplified a lot. There
are
now two profiles to choose from (for Spark 2.2 and 2.3); integration tests
can
be run against a different version of Spark by running just the integration
test module with the desired profile.
Tested with Spark 2.2 and 2.3, and also by building against 2.2 and running
integration tests against 2.3.
----
---
