[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314933373 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: I made a PR. Please take a look~ - https://github.com/apache/spark/pull/25481 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314933206 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: It seems that Maven or Ivy bugs on JDK. After some investigation, the dependency seems not real. I'll make a PR very soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314932068 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: https://issues.apache.org/jira/browse/SPARK-28765 is filed. In case of `javax.annotation-api-1.2.jar`, the following is verified. The following is removed at https://github.com/apache/spark/commit/c9b49f3978666af5d59a76e1cf2c23ed76f31a2d because the dependency generation is done on JDK8. ``` $ mvn dependency:tree -Dincludes=javax.annotation:javax.annotation-api -Phadoop-3.2 -Phive-thriftserver -Pmesos -Pkubernetes -Pyarn -Phive ... [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ spark-kubernetes_2.12 --- [INFO] org.apache.spark:spark-kubernetes_2.12:jar:3.0.0-SNAPSHOT [INFO] \- io.fabric8:kubernetes-client:jar:4.1.2:compile [INFO]\- io.fabric8:kubernetes-model:jar:4.1.2:compile [INFO] \- javax.annotation:javax.annotation-api:jar:1.2:compile ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314932068 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: https://issues.apache.org/jira/browse/SPARK-28765 is filed. In case of `javax.annotation-api-1.2.jar`, the following is verified. ``` $ mvn dependency:tree -Dincludes=javax.annotation:javax.annotation-api -Phadoop-3.2 -Phive-thriftserver -Pmesos -Pkubernetes -Pyarn -Phive ... [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ spark-kubernetes_2.12 --- [INFO] org.apache.spark:spark-kubernetes_2.12:jar:3.0.0-SNAPSHOT [INFO] \- io.fabric8:kubernetes-client:jar:4.1.2:compile [INFO]\- io.fabric8:kubernetes-model:jar:4.1.2:compile [INFO] \- javax.annotation:javax.annotation-api:jar:1.2:compile ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314931221 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: @srowen . It seems that we found that our library have difference dependency on JDK8 and JDK11. On my Mac, I also noticed that this `jakarta.activation-1.2.1` occurs only on JDK11 environment. I'm not sure how this library does this, but in this case, is it possible for Apache Spark to provide a single artifact for JDK8/JDK11? It seems that `Maven` artifacts might be possible, but the pre-built Apache Spark binary should be different due to this from now. Or, we need to use always JDK11 for updating manifests. cc @dbtsai , @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314931221 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: @srowen . It seems that we found that our library have difference dependency on JDK8 and JDK11. On my Mac, I also noticed that this `jakarta.activation-1.2.1` occurs only on JDK11 environment. I'm not sure how this library does this, but in this case, is it possible for Apache Spark to provide a single artifact for JDK8/JDK11? It seems that `Maven` artifacts might be possible, but the pre-built Apache Spark binary should be different due to this from now. Or, we need to use always JDK11 for updating manifests to build the most inclusive jar list. cc @dbtsai , @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314931221 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: @srowen . It seems that we found that our library have difference dependency on JDK8 and JDK11. On my Mac, I also noticed that this `jakarta.activation-1.2.1` occurs only on JDK11 environment. I'm not sure how this library does this, but in this case, is it possible for Apache Spark to provide a single artifact for JDK8/JDK11? It seems that `Maven` artifacts might be possible, but the pre-built Apache Spark binary should be different due to this from now. cc @dbtsai , @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314931221 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: @srowen . We found that our library have difference dependency on JDK8 and JDK11. I also noticed that this `jakarta.activation-1.2.1` occurs only on JDK11 environment. I'm not sure how this library does this, but in this case, is it possible for Apache Spark to provide a single artifact for JDK8/JDK11? It seems that `Maven` artifacts might be possible, but the pre-built Apache Spark binary should be different due to this from now. cc @dbtsai , @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314172342 ## File path: pom.xml ## @@ -115,7 +115,7 @@ UTF-8 UTF-8 -11 +1.8 Review comment: Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443:
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r313686024
##
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##
@@ -59,7 +59,7 @@ private[spark] object HiveUtils extends Logging {
val isHive23: Boolean = hiveVersion.startsWith("2.3")
/** The version of hive used internally by Spark SQL. */
- val builtinHiveVersion: String = if (isHive23) hiveVersion else "1.2.1"
+ val builtinHiveVersion: String = if (isHive23) "2.3.5" else "1.2.1"
Review comment:
Oops. Yes. It's the same with @HyukjinKwon 's comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443:
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r313685898
##
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##
@@ -59,7 +59,7 @@ private[spark] object HiveUtils extends Logging {
val isHive23: Boolean = hiveVersion.startsWith("2.3")
/** The version of hive used internally by Spark SQL. */
- val builtinHiveVersion: String = if (isHive23) hiveVersion else "1.2.1"
+ val builtinHiveVersion: String = if (isHive23) "2.3.5" else "1.2.1"
Review comment:
`2.3.5` -> `2.3.6` to test `Hive 2.3.6` fully?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
