jackxu2011 commented on code in PR #4110:
URL: https://github.com/apache/linkis/pull/4110#discussion_r1091635127
##########
pom.xml:
##########
@@ -1349,6 +1359,45 @@
</build>
<profiles>
+ <!-- hadoop version: mvn validate -Phadoop-3.3 ,spark still use
hadoop2.7.2 by default, More details please check SPARK-23534 -->
+ <profile>
+ <id>hadoop-3.3</id>
+ <properties>
+ <hadoop.version>3.3.1</hadoop.version>
+ <curator.version>4.2.0</curator.version>
+
<hadoop-hdfs-client.artifact>hadoop-hdfs-client</hadoop-hdfs-client.artifact>
+ </properties>
+ </profile>
+ <!-- hadoop version: mvn validate -Phadoop-2.7 -->
+ <profile>
+ <id>hadoop-2.7</id>
+ <properties>
+ <hadoop.version>2.7.2</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ </properties>
+ </profile>
+ <!-- spark2-hadoop3 version:spark2.4 shade use hadoop2.7.2 by default mvn
validate -Pspark-2.4-hadoop-3.3 -->
+ <profile>
+ <id>spark-2.4-hadoop-3.3</id>
Review Comment:
this profile should be moved to spark's pom.xml, and this profile should
work with -Phadoop-3.3. and put the linkis-hadoop-hdfs-client-shade
dependency to this profile.
##########
pom.xml:
##########
@@ -1349,6 +1359,45 @@
</build>
<profiles>
+ <!-- hadoop version: mvn validate -Phadoop-3.3 ,spark still use
hadoop2.7.2 by default, More details please check SPARK-23534 -->
+ <profile>
+ <id>hadoop-3.3</id>
+ <properties>
+ <hadoop.version>3.3.1</hadoop.version>
+ <curator.version>4.2.0</curator.version>
+
<hadoop-hdfs-client.artifact>hadoop-hdfs-client</hadoop-hdfs-client.artifact>
+ </properties>
+ </profile>
+ <!-- hadoop version: mvn validate -Phadoop-2.7 -->
+ <profile>
+ <id>hadoop-2.7</id>
+ <properties>
+ <hadoop.version>2.7.2</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ </properties>
+ </profile>
+ <!-- spark2-hadoop3 version:spark2.4 shade use hadoop2.7.2 by default mvn
validate -Pspark-2.4-hadoop-3.3 -->
+ <profile>
+ <id>spark-2.4-hadoop-3.3</id>
+ <properties>
+ <hadoop.version>3.3.1</hadoop.version>
+ <curator.version>4.2.0</curator.version>
+
<hadoop-hdfs-client.artifact>hadoop-hdfs-client</hadoop-hdfs-client.artifact>
Review Comment:
this properties are same with profile hadoop-3.3 can be ignore
##########
linkis-hadoop-hdfs-client-shade/dependency-reduced-pom.xml:
##########
@@ -0,0 +1,281 @@
+<?xml version="1.0" encoding="UTF-8"?>
Review Comment:
this file should be ignore
##########
linkis-engineconn-plugins/spark/pom.xml:
##########
@@ -202,13 +207,103 @@
<artifactId>linkis-rpc</artifactId>
<version>${project.version}</version>
</dependency>
-
+ <dependency>
+ <groupId>org.apache.linkis</groupId>
+ <artifactId>linkis-hadoop-hdfs-client-shade</artifactId>
+ <version>${project.version}</version>
Review Comment:
move the spark-2.4-hadoop-3.3 profile to this pom, and add the dependency
to the profile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
