[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r316462160 ## File path: pom.xml ## @@ -253,6 +253,18 @@ false + + + staged + staged-releases + https://repository.apache.org/content/repositories/staging/ Review comment: FYI, Hive 2.3.6 is being voted at https://www.mail-archive.com/dev@hive.apache.org/msg137043.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r315439833 ## File path: pom.xml ## @@ -253,6 +253,18 @@ false + + + staged + staged-releases + https://repository.apache.org/content/repositories/staging/ Review comment: @srowen, yea almost ready per https://github.com/apache/spark/pull/25443/files#r315437547. Plus, Hive 2.3.6 vote looks going to be open now. After the release, we can switch this snapshot to the official Hive 2.3.6 and I believe we're good to go. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314932572 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: Out of curiosity, is it an option to have two releases? - Spark, Hadoop 2.7, JDK 8 - Spark, Hadoop 3.2, JDK 11 Since we currently skipped JDK 9 and 10 and JDK 8 is what we should drop later in the future anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443:
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r313780146
##
File path: dev/run-tests-jenkins
##
@@ -31,4 +31,7 @@ if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
exit -1
fi
+export JAVA_HOME=/usr/java/jdk-11.0.1
+# export PATH=${JAVA_HOME}/bin:${PATH}
Review comment:
yea, I hope rather just this was the cause.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443:
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r313701130
##
File path: dev/run-tests-jenkins
##
@@ -31,4 +31,7 @@ if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
exit -1
fi
+export JAVA_HOME=/usr/java/jdk-11.0.1
+export PATH=${JAVA_HOME}/bin:${PATH}
Review comment:
I just pushed two commits to test
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443:
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r313700184
##
File path: dev/run-tests-jenkins
##
@@ -31,4 +31,7 @@ if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
exit -1
fi
+export JAVA_HOME=/usr/java/jdk-11.0.1
+export PATH=${JAVA_HOME}/bin:${PATH}
Review comment:
cc @shaneknapp and @srowen FYI Yuming is trying to build it on the top of
https://github.com/apache/spark/pull/25423 . Seems Java 8 is not properly set
so it needs some changes at #25423
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443:
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r31364
##
File path: dev/run-tests-jenkins
##
@@ -31,4 +31,7 @@ if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
exit -1
fi
+export JAVA_HOME=/usr/java/jdk-11.0.1
+export PATH=${JAVA_HOME}/bin:${PATH}
Review comment:
@wangyum, can you try this?
Seems we should pass env at
https://github.com/apache/spark/pull/25423/files#diff-3c9f4fccf7d30ce2e8fa86db2ad1fdadR124
```diff
diff --git a/dev/run-tests-jenkins.py b/dev/run-tests-jenkins.py
index 4b91a5fa423..7a73657138f 100755
--- a/dev/run-tests-jenkins.py
+++ b/dev/run-tests-jenkins.py
@@ -121,7 +121,8 @@ def run_tests(tests_timeout):
test_result_code = subprocess.Popen(['timeout',
tests_timeout,
- os.path.join(SPARK_HOME, 'dev',
'run-tests')]).wait()
+ os.path.join(SPARK_HOME, 'dev',
'run-tests')],
+env=dict(os.environ)).wait()
```
Seems like there's something wrong when environment variables are inherited
(by default).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443:
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r313689496
##
File path: dev/run-tests-jenkins
##
@@ -31,4 +31,7 @@ if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
exit -1
fi
+export JAVA_HOME=/usr/java/jdk-11.0.1
+export PATH=${JAVA_HOME}/bin:${PATH}
Review comment:
if it doesn't work, you can even try within `build/mvn`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443:
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r313685852
##
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##
@@ -59,7 +59,7 @@ private[spark] object HiveUtils extends Logging {
val isHive23: Boolean = hiveVersion.startsWith("2.3")
/** The version of hive used internally by Spark SQL. */
- val builtinHiveVersion: String = if (isHive23) hiveVersion else "1.2.1"
+ val builtinHiveVersion: String = if (isHive23) "2.3.5" else "1.2.1"
Review comment:
https://github.com/apache/spark/pull/25443#issuecomment-521078239 - yeah,
otherwise people might keep asking why it's 2.3.5.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
