martin-g commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r755790712
##########
File path: .github/workflows/core.yml
##########
@@ -94,97 +97,25 @@ jobs:
key: ${{ runner.os }}-zeppelin-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-zeppelin-
+ - name: install environment
+ run: ./mvnw install -DskipTests -DskipRat -am -pl
.,zeppelin-interpreter,zeppelin-interpreter-shaded,${INTERPRETERS} -Pscala-2.10
-B
Review comment:
Is there a reason to still support/test Scala 2.10 ?
Most of the projects dropped support even for 2.11
##########
File path: pom.xml
##########
@@ -532,6 +550,129 @@
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>asm</groupId>
+ <artifactId>asm</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.ow2.asm</groupId>
+ <artifactId>asm</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.jboss.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey.jersey-test-framework</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey.contribs</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-compress</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
Review comment:
Should this use `${hadoop-client-runtime.artifact}` instead ?
##########
File path: pom.xml
##########
@@ -398,12 +401,27 @@
<version>${jettison.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
Review comment:
Is this dependency still needed ? It will be a duplicate of the newly
added above (line 404-409)
##########
File path: spark/spark-scala-parent/pom.xml
##########
@@ -80,6 +80,21 @@
<scope>provided</scope>
</dependency>
+ <!-- Use provided scope, hadoop dependencies are only for compilation
-->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>2.7.7</version>
Review comment:
`${hadoop.version}` ?
##########
File path:
zeppelin-interpreter-integration/src/test/java/org/apache/zeppelin/integration/ZeppelinSparkClusterTest.java
##########
@@ -154,6 +162,11 @@ private void waitForRunning(Paragraph p) {
@Test
public void scalaOutputTest() throws IOException, InterruptedException {
+ if (!isHadoopVersionMatch()) {
Review comment:
IMO this would be better by using JUnit Assumption APIs, e.g.
`assumeThat(isHadoopVersionMatch(), "Hadoop version mismatch, skipping the
test")`. No need of the `return;`
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
2.11 ?!
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
2.11 ?!
##########
File path: pom.xml
##########
@@ -997,6 +1215,195 @@
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-json</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-client</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.jackrabbit</groupId>
+ <artifactId>jackrabbit-webdav</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-httpclient</groupId>
+ <artifactId>commons-httpclient</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jgit</groupId>
+ <artifactId>org.eclipse.jgit</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.jcraft</groupId>
+ <artifactId>jsch</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-compress</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xml-apis</groupId>
+ <artifactId>xml-apis</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xerces</groupId>
+ <artifactId>xercesImpl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ </exclusion>
+ <!-- using jcl-over-slf4j instead -->
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <version>${hadoop.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-json</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-client</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.jackrabbit</groupId>
+ <artifactId>jackrabbit-webdav</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-httpclient</groupId>
+ <artifactId>commons-httpclient</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jgit</groupId>
+ <artifactId>org.eclipse.jgit</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.jcraft</groupId>
+ <artifactId>jsch</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-compress</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xml-apis</groupId>
+ <artifactId>xml-apis</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xerces</groupId>
+ <artifactId>xercesImpl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ </exclusion>
+ <!-- using jcl-over-slf4j instead -->
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
Review comment:
Maven could complain that this is a duplicate the one above where I
asked about `hadoop-client-runtime.artifact`
##########
File path: spark/interpreter/pom.xml
##########
@@ -198,10 +205,23 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.6.0</version>
+ <version>2.7.7</version>
Review comment:
`${hadoop.version}` ?
##########
File path: rlang/pom.xml
##########
@@ -113,6 +113,13 @@
<scope>compile</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>2.7.7</version>
Review comment:
Should this use `${hadoop.version}` ?
##########
File path: spark/interpreter/pom.xml
##########
@@ -113,6 +113,13 @@
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>2.7.7</version>
Review comment:
`${hadoop.version}` ?
##########
File path: .github/workflows/frontend.yml
##########
@@ -47,6 +47,10 @@ jobs:
run: ./mvnw -B install -DskipTests -DskipRat -pl ${INTERPRETERS}
-Phadoop2 -Pscala-2.11
- name: Run headless test
run: xvfb-run --auto-servernum --server-args="-screen 0 1024x768x24"
./mvnw verify -DskipRat -pl zeppelin-web -Phadoop2 -Pscala-2.11 -Pweb-e2e -B
+ - name: Print zeppelin logs
+ if: always()
Review comment:
Why is this needed ?
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
2.11 ?!
##########
File path: spark/interpreter/pom.xml
##########
@@ -198,10 +205,23 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.6.0</version>
+ <version>2.7.7</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>2.7.7</version>
Review comment:
`${hadoop.version}` ?
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +173,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
https://search.maven.org/artifact/commons-io/commons-io/2.11.0/jar is
the latest one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
