This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch branch-0.8
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/branch-0.8 by this push:
new b004b46 [ZEPPELIN-3939] Spark 2.4 uses a more recent commons-lang3
than Zeppelin-0.8
b004b46 is described below
commit b004b4699b10470fcc646eb8500683813c361f18
Author: Maziyar Panahi <[email protected]>
AuthorDate: Fri Jan 25 12:56:53 2019 +0100
[ZEPPELIN-3939] Spark 2.4 uses a more recent commons-lang3 than Zeppelin-0.8
### What is this PR for?
`Zeppelin-0.8.1` and `Zeppelin-0.8.2` cannot fully support Spark 2.4 if
`commons-lang3` has incompaitble version from the one in Spark 2.4. The
branch-0.9 (master) has already been updated to `3.7`, however the branch-0.8
hasn't done these changes.
By updating the version from `3.4` to `3.7` we resolve the issue that I
have reported on Jira regarding reading `CSV` and `JSON` files in Spark 2.4.x.
Next release of `Zeppelin-0.8.x` can fully support Spark 2.4 without
waiting for `Zeppelin 0.9.x`.
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
https://issues.apache.org/jira/projects/ZEPPELIN/issues/ZEPPELIN-3939
### How should this be tested?
* First time? Setup Travis CI:
Done
* Outline any manual steps to test the PR here.
I have tested reading `CSV` and `JSON` in `Zeppelin-0.8.0` (released),
`Zeppelin-0.8.1` (released), and `Zeppelin-0.8.2` (not released yet) by
pointing SPARK_HOME to spark-2.4.0 and they all failed same as the Jira issue.
```
java.io.InvalidClassException:
org.apache.commons.lang3.time.FastDateParser; local class incompatible
```
After updating `common-lang3` to `3.7`, Zeppelin-0.8.2 reads the `CSV` and
`JSON` files in spark-2.4.0 without any issue.
This has happened before and the solution was to update `commons-lang3` to
appropriate version used in Apache Spark:
https://github.com/apache/zeppelin/commit/88a905caf2043d12a85462940c451b424d218dba
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Maziyar Panahi <[email protected]>
Closes #3290 from maziyarpanahi/branch-0.8 and squashes the following
commits:
0709f5b47 [Maziyar Panahi] Add spark csv and json tests
9db542668 [Maziyar Panahi] Merge branch 'branch-0.8' of
https://github.com/multivacplatform/zeppelin into branch-0.8
d59f27447 [Maziyar Panahi] Update commons-lang3 to fully support Spark 2.4
2b531755b [Maziyar Panahi] Update commons-lang3 to fully support Spark 2.4
---
zeppelin-distribution/src/bin_license/LICENSE | 2 +-
zeppelin-integration/pom.xml | 2 +-
zeppelin-interpreter/pom.xml | 2 +-
.../zeppelin/rest/ZeppelinSparkClusterTest.java | 25 +++++++++++++++++++++-
zeppelin-zengine/pom.xml | 2 +-
5 files changed, 28 insertions(+), 5 deletions(-)
diff --git a/zeppelin-distribution/src/bin_license/LICENSE
b/zeppelin-distribution/src/bin_license/LICENSE
index 8bdea23..b7c13b3 100644
--- a/zeppelin-distribution/src/bin_license/LICENSE
+++ b/zeppelin-distribution/src/bin_license/LICENSE
@@ -15,7 +15,7 @@ The following components are provided under Apache License.
(Apache 2.0) Http Components (org.apache.httpcomponents:httpclient:4.3.6 -
https://github.com/apache/httpclient)
(Apache 2.0) Http Components
(org.apache.httpcomponents:httpasyncclient:4.0.2 -
https://github.com/apache/httpclient)
(Apache 2.0) Apache Commons Lang (org.apache.commons:commons-lang:2.5 -
http://commons.apache.org/proper/commons-lang/)
- (Apache 2.0) Apache Commons Lang 3 (org.apache.commons:commons-lang3:3.4 -
http://commons.apache.org/proper/commons-lang/)
+ (Apache 2.0) Apache Commons Lang 3 (org.apache.commons:commons-lang3:3.7 -
http://commons.apache.org/proper/commons-lang/)
(Apache 2.0) Apache Commons Math 3 (org.apache.commons:commons-math3:3.6.1
- http://commons.apache.org/proper/commons-math/)
(Apache 2.0) Apache Commons Net (commons-net:commons-net:2.2 -
http://commons.apache.org/proper/commons-net/)
(Apache 2.0) Apache Commons Pool2 (commons-exec:commons-pool2:2.3 -
https://commons.apache.org/proper/commons-pool/)
diff --git a/zeppelin-integration/pom.xml b/zeppelin-integration/pom.xml
index 8c63dd3..f675df9 100644
--- a/zeppelin-integration/pom.xml
+++ b/zeppelin-integration/pom.xml
@@ -44,7 +44,7 @@
<!--test library versions-->
<selenium.java.version>3.8.1</selenium.java.version>
- <commons.lang3.version>3.4</commons.lang3.version>
+ <commons.lang3.version>3.7</commons.lang3.version>
<!--plugin library versions-->
<plugin.failsafe.version>2.16</plugin.failsafe.version>
diff --git a/zeppelin-interpreter/pom.xml b/zeppelin-interpreter/pom.xml
index 25ab399..8025dcb 100644
--- a/zeppelin-interpreter/pom.xml
+++ b/zeppelin-interpreter/pom.xml
@@ -37,7 +37,7 @@
<properties>
<!--library versions-->
- <commons.lang3.version>3.4</commons.lang3.version>
+ <commons.lang3.version>3.7</commons.lang3.version>
<commons.pool2.version>2.3</commons.pool2.version>
<commons.exec.version>1.3</commons.exec.version>
<maven.plugin.api.version>3.0</maven.plugin.api.version>
diff --git
a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
index 21b861c..e862d29 100644
---
a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
+++
b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
@@ -145,7 +145,6 @@ public class ZeppelinSparkClusterTest extends
AbstractTestRestApi {
assertTrue(p.getResult().message().get(0).getData().contains("error: "));
}
-
@Test
public void basicRDDTransformationAndActionTest() throws IOException {
Note note = ZeppelinServer.notebook.createNote(anonymous);
@@ -157,6 +156,30 @@ public class ZeppelinSparkClusterTest extends
AbstractTestRestApi {
}
@Test
+ public void sparkReadJSONTest() throws IOException {
+ Note note = ZeppelinServer.notebook.createNote(anonymous);
+ Paragraph p = note.addNewParagraph(anonymous);
+ p.setText("%spark val jsonStr = \"\"\"{ \"metadata\": { \"key\": 84896,
\"value\": 54 }}\"\"\"\n" +
+ "spark.read.json(Seq(jsonStr).toDS)");
+ note.run(p.getId(), true);
+ assertEquals(Status.FINISHED, p.getStatus());
+ assertTrue(p.getResult().message().get(0).getData().contains(
+ "org.apache.spark.sql.DataFrame = [metadata: struct<key: bigint,
value: bigint>]\n"));
+ }
+
+ @Test
+ public void sparkReadCSVTest() throws IOException {
+ Note note = ZeppelinServer.notebook.createNote(anonymous);
+ Paragraph p = note.addNewParagraph(anonymous);
+ p.setText("%spark val csvStr = \"\"\"84896,54\"\"\"\n" +
+ "spark.read.csv(Seq(csvStr).toDS)");
+ note.run(p.getId(), true);
+ assertEquals(Status.FINISHED, p.getStatus());
+ assertTrue(p.getResult().message().get(0).getData().contains(
+ "org.apache.spark.sql.DataFrame = [_c0: string, _c1: string]\n"));
+ }
+
+ @Test
public void sparkSQLTest() throws IOException {
Note note = ZeppelinServer.notebook.createNote(anonymous);
// test basic dataframe api
diff --git a/zeppelin-zengine/pom.xml b/zeppelin-zengine/pom.xml
index af38e09..191bdab 100644
--- a/zeppelin-zengine/pom.xml
+++ b/zeppelin-zengine/pom.xml
@@ -37,7 +37,7 @@
<properties>
<!--library versions-->
<hadoop.version>2.7.3</hadoop.version>
- <commons.lang3.version>3.4</commons.lang3.version>
+ <commons.lang3.version>3.7</commons.lang3.version>
<commons.vfs2.version>2.0</commons.vfs2.version>
<gcs.storage.version>1.14.0</gcs.storage.version>
<aws.sdk.s3.version>1.10.62</aws.sdk.s3.version>
