[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522208067 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109258/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522208066 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522208066 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522208067 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109258/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522208041 **[Test build #109258 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109258/testReport)** for PR 24829 at commit [`a2e9e57`](https://github.com/apache/spark/commit/a2e9e5749cadd658950f33d8c8606f3800bb70f8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522200113 **[Test build #109258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109258/testReport)** for PR 24829 at commit [`a2e9e57`](https://github.com/apache/spark/commit/a2e9e5749cadd658950f33d8c8606f3800bb70f8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon edited a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522207828 Since the test passed with JDK 11 anyway, I am going to share this official test results with Hive dev and let them release Hive 2.3.6. I assume it's roughly agreed upon given https://github.com/apache/spark/pull/25443#issuecomment-521913947 and https://github.com/apache/spark/pull/25443#issuecomment-521915753 If there are no concerns, I will ask it tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522207828 Since the test passed with JDK 11 anyway, I am going to share this official test results with Hive dev and let them release Hive 2.3.6. I assume it's roughly agreed upon given https://github.com/apache/spark/pull/25443#issuecomment-521913947 and https://github.com/apache/spark/pull/25443#issuecomment-521915753 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-522207781 The last three builds were failing for different reasons and all seams to me unrelated for my change. I trigger the build again without any change (previously rebased on top of master). jenkins retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-522207230 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-522207232 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14332/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-522207230 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-522207232 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14332/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4
AmplabJenkins commented on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4 URL: https://github.com/apache/spark/pull/25482#issuecomment-522206909 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-522206930 **[Test build #109264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109264/testReport)** for PR 24903 at commit [`9c96d61`](https://github.com/apache/spark/commit/9c96d619fdbb3c9bb11d499681d4adb7b5b084e4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4
AmplabJenkins removed a comment on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4 URL: https://github.com/apache/spark/pull/25482#issuecomment-522206785 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-522206823 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14331/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-522206822 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-522206823 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14331/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-522206822 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4
AmplabJenkins removed a comment on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4 URL: https://github.com/apache/spark/pull/25482#issuecomment-522206750 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4
AmplabJenkins commented on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4 URL: https://github.com/apache/spark/pull/25482#issuecomment-522206785 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4
AmplabJenkins commented on issue #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4 URL: https://github.com/apache/spark/pull/25482#issuecomment-522206750 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mattf-apache opened a new pull request #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4
mattf-apache opened a new pull request #25482: [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 in branch-2.4 URL: https://github.com/apache/spark/pull/25482 ### What changes were proposed in this pull request? Simple fix of https://issues.apache.org/jira/browse/SPARK-28749 ### Why are the changes needed? As discussed in the referenced Jira, currently, the PySpark tests invoked by `python/run-tests` demand the presence of kafka-0-8 libraries. If not present, a failure message will be generated regardless of whether the tests are enabled by env variables. Since Kafka-0-8 libraries are not compatible with Scala-2.12, this means we can’t successfully run pyspark tests on a Scala-2.12 build. These proposed changes fix the problem. ### Does this PR introduce any user-facing change? No. It only changes a test behavior. ### How was this patch tested? This is a fix of a test bug. The current behavior is demonstrably wrong under Scala-2.12, as stated above. The corrected behavior allows the tests to run to completion, with results that are consistent with the expected results, similar to the successful Scala-2.11 results. We performed the following: - Full mvn build under Scala-2.11, with kafka-0-8 profile - Full mvn build under Scala-2.12, without kafka-0-8 profile - Full maven Unit Test of both, with no change (as expected) - PySpark tests via `python/run-tests` with both. Both complete successfully. Former behavior before this patch was that the last step would post a Failure on the Scala-2.12 build. ### Notes for reviewers There are of course many ways to fix the problem. I chose to follow the pattern established by the Kinesis testing routines that were right “next to” the Kafka-0-8 test routines in the same file. They showed a presumably acceptable way of dealing with missing jars. I ignored the env variable, because the presence or absence of the jar seemed a sufficient and more important determinant. But if you want me to use the env variable exactly the same way as the Kinesis testing code, I’ll be happy to do so. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522206276 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109252/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522206275 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522206276 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109252/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522206275 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
SparkQA removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522183692 **[Test build #109252 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109252/testReport)** for PR 25443 at commit [`24bb028`](https://github.com/apache/spark/commit/24bb028efb2d1061c372acc1476b18200d911f64). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-522206143 **[Test build #109252 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109252/testReport)** for PR 25443 at commit [`24bb028`](https://github.com/apache/spark/commit/24bb028efb2d1061c372acc1476b18200d911f64). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config
SparkQA commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config URL: https://github.com/apache/spark/pull/22570#issuecomment-522204726 **[Test build #109263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109263/testReport)** for PR 22570 at commit [`24f95b7`](https://github.com/apache/spark/commit/24f95b72930fe2426c3f639115d7be88dec87925). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config URL: https://github.com/apache/spark/pull/22570#issuecomment-522204610 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config
AmplabJenkins removed a comment on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config URL: https://github.com/apache/spark/pull/22570#issuecomment-522204611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14330/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config URL: https://github.com/apache/spark/pull/22570#issuecomment-522204611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14330/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config
AmplabJenkins commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config URL: https://github.com/apache/spark/pull/22570#issuecomment-522204610 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-522204126 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-522204126 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
SparkQA removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-522202533 **[Test build #109262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109262/testReport)** for PR 22282 at commit [`79c3b58`](https://github.com/apache/spark/commit/79c3b5895f9e026f52ff2d8851f27dfbc89c0c2f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
AmplabJenkins commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-522204128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109262/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
AmplabJenkins removed a comment on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-522204128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109262/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-522204110 **[Test build #109262 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109262/testReport)** for PR 22282 at commit [`79c3b58`](https://github.com/apache/spark/commit/79c3b5895f9e026f52ff2d8851f27dfbc89c0c2f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1
dongjoon-hyun commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1 URL: https://github.com/apache/spark/pull/25476#issuecomment-522203749 Thanks, @srowen . Until now, I'm trying to focus on JDK11 related stuff. For 3.0.0, yes, I'll try to check the plugins and make it up-to-date. > BTW I think you're welcome to update lots of plugins in one go if you like. It'd be good to keep them pretty up to date for 3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #19373: [SPARK-22150][CORE] PeriodicCheckpointer fails in case of dependent RDDs
AmplabJenkins removed a comment on issue #19373: [SPARK-22150][CORE] PeriodicCheckpointer fails in case of dependent RDDs URL: https://github.com/apache/spark/pull/19373#issuecomment-522112610 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #19373: [SPARK-22150][CORE] PeriodicCheckpointer fails in case of dependent RDDs
AmplabJenkins commented on issue #19373: [SPARK-22150][CORE] PeriodicCheckpointer fails in case of dependent RDDs URL: https://github.com/apache/spark/pull/19373#issuecomment-522203652 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
dongjoon-hyun commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#discussion_r314934061 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/aggregates_part3.sql ## @@ -0,0 +1,273 @@ +-- +-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group +-- +-- +-- AGGREGATES [Part 3] +-- https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L352-L605 + +-- We do not support inheritance tree, skip related tests. +-- try it on an inheritance tree +-- create table minmaxtest(f1 int); +-- create table minmaxtest1() inherits (minmaxtest); +-- create table minmaxtest2() inherits (minmaxtest); +-- create table minmaxtest3() inherits (minmaxtest); +-- create index minmaxtesti on minmaxtest(f1); +-- create index minmaxtest1i on minmaxtest1(f1); +-- create index minmaxtest2i on minmaxtest2(f1 desc); +-- create index minmaxtest3i on minmaxtest3(f1) where f1 is not null; + +-- insert into minmaxtest values(11), (12); +-- insert into minmaxtest1 values(13), (14); +-- insert into minmaxtest2 values(15), (16); +-- insert into minmaxtest3 values(17), (18); + +-- explain (costs off) +-- select min(f1), max(f1) from minmaxtest; +-- select min(f1), max(f1) from minmaxtest; + +-- DISTINCT doesn't do anything useful here, but it shouldn't fail +-- explain (costs off) +-- select distinct min(f1), max(f1) from minmaxtest; +-- select distinct min(f1), max(f1) from minmaxtest; + +-- drop table minmaxtest cascade; + +-- [SPARK-9830] It is not allowed to use an aggregate function in the argument of another aggregate function +-- check for correct detection of nested-aggregate errors +-- select max(min(unique1)) from tenk1; +-- select (select max(min(unique1)) from int8_tbl) from tenk1; + +-- These tests only test the explain. Skip these tests. +-- +-- Test removal of redundant GROUP BY columns +-- + +-- create temp table t1 (a int, b int, c int, d int, primary key (a, b)); +-- create temp table t2 (x int, y int, z int, primary key (x, y)); +-- create temp table t3 (a int, b int, c int, primary key(a, b) deferrable); + +-- Non-primary-key columns can be removed from GROUP BY +-- explain (costs off) select * from t1 group by a,b,c,d; + +-- No removal can happen if the complete PK is not present in GROUP BY +-- explain (costs off) select a,c from t1 group by a,c,d; + +-- Test removal across multiple relations +-- explain (costs off) select * +-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y +-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.y,t2.z; + +-- Test case where t1 can be optimized but not t2 +-- explain (costs off) select t1.*,t2.x,t2.z +-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y +-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.z; + +-- Cannot optimize when PK is deferrable +-- explain (costs off) select * from t3 group by a,b,c; + +-- drop table t1; +-- drop table t2; +-- drop table t3; + +-- [SPARK-27974] Add built-in Aggregate Function: array_agg +-- +-- Test combinations of DISTINCT and/or ORDER BY +-- + +-- select array_agg(a order by b) +-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b); +-- select array_agg(a order by a) +-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b); +-- select array_agg(a order by a desc) +-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b); +-- select array_agg(b order by a desc) +-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b); + +-- select array_agg(distinct a) +-- from (values (1),(2),(1),(3),(null),(2)) v(a); +-- select array_agg(distinct a order by a) +-- from (values (1),(2),(1),(3),(null),(2)) v(a); +-- select array_agg(distinct a order by a desc) +-- from (values (1),(2),(1),(3),(null),(2)) v(a); +-- select array_agg(distinct a order by a desc nulls last) +-- from (values (1),(2),(1),(3),(null),(2)) v(a); + +-- Skip the test below because it requires 4 UDFs: aggf_trans, aggfns_trans, aggfstr, and aggfns +-- multi-arg aggs, strict/nonstrict, distinct/order by + +-- select aggfstr(a,b,c) +-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c); +-- select aggfns(a,b,c) +-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c); + +-- select aggfstr(distinct a,b,c) +-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c), +--generate_series(1,3) i; +-- select aggfns(distinct a,b,c) +-- from (values (1,3,'foo'),(0,null,null),(2,2,'bar'),(3,1,'baz')) v(a,b,c), +--generate_series(1,3) i; + +-- select aggfstr(distinct a,b,c order by b) Review comment: Do we have a JIRA for that? Then, please add the ID as a comment here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at:
[GitHub] [spark] AmplabJenkins commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
AmplabJenkins commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522203546 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
AmplabJenkins removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522203546 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
AmplabJenkins removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522203548 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109261/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
SparkQA commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522203523 **[Test build #109261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109261/testReport)** for PR 25480 at commit [`da84f92`](https://github.com/apache/spark/commit/da84f92f7eb011184641eefb81f669bcdf74d080). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
dongjoon-hyun commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#discussion_r314934039 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/aggregates_part3.sql ## @@ -0,0 +1,273 @@ +-- +-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group +-- +-- +-- AGGREGATES [Part 3] +-- https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L352-L605 + +-- We do not support inheritance tree, skip related tests. +-- try it on an inheritance tree +-- create table minmaxtest(f1 int); +-- create table minmaxtest1() inherits (minmaxtest); +-- create table minmaxtest2() inherits (minmaxtest); +-- create table minmaxtest3() inherits (minmaxtest); +-- create index minmaxtesti on minmaxtest(f1); +-- create index minmaxtest1i on minmaxtest1(f1); +-- create index minmaxtest2i on minmaxtest2(f1 desc); +-- create index minmaxtest3i on minmaxtest3(f1) where f1 is not null; + +-- insert into minmaxtest values(11), (12); +-- insert into minmaxtest1 values(13), (14); +-- insert into minmaxtest2 values(15), (16); +-- insert into minmaxtest3 values(17), (18); + +-- explain (costs off) +-- select min(f1), max(f1) from minmaxtest; +-- select min(f1), max(f1) from minmaxtest; + +-- DISTINCT doesn't do anything useful here, but it shouldn't fail +-- explain (costs off) +-- select distinct min(f1), max(f1) from minmaxtest; +-- select distinct min(f1), max(f1) from minmaxtest; + +-- drop table minmaxtest cascade; + +-- [SPARK-9830] It is not allowed to use an aggregate function in the argument of another aggregate function +-- check for correct detection of nested-aggregate errors +-- select max(min(unique1)) from tenk1; +-- select (select max(min(unique1)) from int8_tbl) from tenk1; + +-- These tests only test the explain. Skip these tests. +-- +-- Test removal of redundant GROUP BY columns +-- + +-- create temp table t1 (a int, b int, c int, d int, primary key (a, b)); +-- create temp table t2 (x int, y int, z int, primary key (x, y)); +-- create temp table t3 (a int, b int, c int, primary key(a, b) deferrable); + +-- Non-primary-key columns can be removed from GROUP BY +-- explain (costs off) select * from t1 group by a,b,c,d; + +-- No removal can happen if the complete PK is not present in GROUP BY +-- explain (costs off) select a,c from t1 group by a,c,d; + +-- Test removal across multiple relations +-- explain (costs off) select * +-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y +-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.y,t2.z; + +-- Test case where t1 can be optimized but not t2 +-- explain (costs off) select t1.*,t2.x,t2.z +-- from t1 inner join t2 on t1.a = t2.x and t1.b = t2.y +-- group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.z; + +-- Cannot optimize when PK is deferrable +-- explain (costs off) select * from t3 group by a,b,c; + +-- drop table t1; +-- drop table t2; +-- drop table t3; + +-- [SPARK-27974] Add built-in Aggregate Function: array_agg +-- +-- Test combinations of DISTINCT and/or ORDER BY +-- + +-- select array_agg(a order by b) +-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b); +-- select array_agg(a order by a) +-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b); +-- select array_agg(a order by a desc) +-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b); +-- select array_agg(b order by a desc) +-- from (values (1,4),(2,3),(3,1),(4,2)) v(a,b); + +-- select array_agg(distinct a) +-- from (values (1),(2),(1),(3),(null),(2)) v(a); +-- select array_agg(distinct a order by a) +-- from (values (1),(2),(1),(3),(null),(2)) v(a); +-- select array_agg(distinct a order by a desc) +-- from (values (1),(2),(1),(3),(null),(2)) v(a); +-- select array_agg(distinct a order by a desc nulls last) +-- from (values (1),(2),(1),(3),(null),(2)) v(a); + +-- Skip the test below because it requires 4 UDFs: aggf_trans, aggfns_trans, aggfstr, and aggfns Review comment: Got it. Never mind. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
SparkQA removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522202531 **[Test build #109261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109261/testReport)** for PR 25480 at commit [`da84f92`](https://github.com/apache/spark/commit/da84f92f7eb011184641eefb81f669bcdf74d080). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
AmplabJenkins commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522203548 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109261/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
dongjoon-hyun commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#discussion_r314934027 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/aggregates_part3.sql ## @@ -0,0 +1,273 @@ +-- +-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group +-- +-- +-- AGGREGATES [Part 3] +-- https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L352-L605 + +-- We do not support inheritance tree, skip related tests. +-- try it on an inheritance tree +-- create table minmaxtest(f1 int); +-- create table minmaxtest1() inherits (minmaxtest); +-- create table minmaxtest2() inherits (minmaxtest); +-- create table minmaxtest3() inherits (minmaxtest); +-- create index minmaxtesti on minmaxtest(f1); +-- create index minmaxtest1i on minmaxtest1(f1); +-- create index minmaxtest2i on minmaxtest2(f1 desc); +-- create index minmaxtest3i on minmaxtest3(f1) where f1 is not null; + +-- insert into minmaxtest values(11), (12); +-- insert into minmaxtest1 values(13), (14); +-- insert into minmaxtest2 values(15), (16); +-- insert into minmaxtest3 values(17), (18); + +-- explain (costs off) +-- select min(f1), max(f1) from minmaxtest; +-- select min(f1), max(f1) from minmaxtest; + +-- DISTINCT doesn't do anything useful here, but it shouldn't fail +-- explain (costs off) +-- select distinct min(f1), max(f1) from minmaxtest; +-- select distinct min(f1), max(f1) from minmaxtest; + +-- drop table minmaxtest cascade; + +-- [SPARK-9830] It is not allowed to use an aggregate function in the argument of another aggregate function +-- check for correct detection of nested-aggregate errors Review comment: Ya. I know. My suggestion is to check the error message like PostgreSQL. If it doesn't throw exceptions later, we can detect a regression at that time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1
AmplabJenkins commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1 URL: https://github.com/apache/spark/pull/25476#issuecomment-522203231 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109257/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1
AmplabJenkins commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1 URL: https://github.com/apache/spark/pull/25476#issuecomment-522203230 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1
AmplabJenkins removed a comment on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1 URL: https://github.com/apache/spark/pull/25476#issuecomment-522203231 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109257/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1
AmplabJenkins removed a comment on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1 URL: https://github.com/apache/spark/pull/25476#issuecomment-522203230 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1
dongjoon-hyun closed pull request #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1 URL: https://github.com/apache/spark/pull/25476 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1
SparkQA removed a comment on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1 URL: https://github.com/apache/spark/pull/25476#issuecomment-522195210 **[Test build #109257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109257/testReport)** for PR 25476 at commit [`5d6015a`](https://github.com/apache/spark/commit/5d6015abefed7c766f49eac8d7e89228a9246529). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1
SparkQA commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1 URL: https://github.com/apache/spark/pull/25476#issuecomment-522203125 **[Test build #109257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109257/testReport)** for PR 25476 at commit [`5d6015a`](https://github.com/apache/spark/commit/5d6015abefed7c766f49eac8d7e89228a9246529). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1
dongjoon-hyun commented on issue #25476: [SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.1.1 URL: https://github.com/apache/spark/pull/25476#issuecomment-522203177 Thank you, @HyukjinKwon , @srowen , @viirya . Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
wangyum commented on a change in pull request #25480: [SPARK-28763][SQL][TEST]
Flaky Tests:
SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary
type
URL: https://github.com/apache/spark/pull/25480#discussion_r314933647
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkThriftServerProtocolVersionsSuite.scala
##
@@ -224,7 +224,9 @@ class SparkThriftServerProtocolVersionsSuite extends
HiveThriftJdbcTest {
}
testExecuteStatementWithProtocolVersion(version, "SELECT cast(49960 as
binary)") { rs =>
assert(rs.next())
-assert(rs.getString(1) ===
UTF8String.fromBytes(NumberConverter.toBinary(49960)).toString)
+assertResult(Array(0, 0, 65533, 40).map(_.toChar)) {
+ rs.getString(1).toCharArray
+}
Review comment:
@dongjoon-hyun @HyukjinKwon Maybe we should uses CharArray.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
SparkQA commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522202531 **[Test build #109261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109261/testReport)** for PR 25480 at commit [`da84f92`](https://github.com/apache/spark/commit/da84f92f7eb011184641eefb81f669bcdf74d080). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
SparkQA commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-522202533 **[Test build #109262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109262/testReport)** for PR 22282 at commit [`79c3b58`](https://github.com/apache/spark/commit/79c3b5895f9e026f52ff2d8851f27dfbc89c0c2f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
AmplabJenkins commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522202442 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14329/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
AmplabJenkins removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522202442 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14329/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
AmplabJenkins removed a comment on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522202441 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
AmplabJenkins commented on issue #25480: [SPARK-28763][SQL][TEST] Flaky Tests: SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type URL: https://github.com/apache/spark/pull/25480#issuecomment-522202441 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjinleekr commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
dongjinleekr commented on issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming URL: https://github.com/apache/spark/pull/22282#issuecomment-522202301 @zsxwing @srowen Here is the fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjinleekr commented on a change in pull request #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
dongjinleekr commented on a change in pull request #22282: [SPARK-23539][SS]
Add support for Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#discussion_r314933462
##
File path:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDataConsumerSuite.scala
##
@@ -95,10 +100,12 @@ class KafkaDataConsumerSuite extends SharedSQLContext
with PrivateMethodTester {
try {
val range = consumer.getAvailableOffsetRange()
val rcvd = range.earliest until range.latest map { offset =>
- val bytes = consumer.get(offset, Long.MaxValue, 1,
failOnDataLoss = false).value()
- new String(bytes)
+ val record = consumer.get(offset, Long.MaxValue, 1,
failOnDataLoss = false)
+ val value = new String(record.value(), StandardCharsets.UTF_8)
+ val headers = record.headers().toArray.map(header => (header.key(),
header.value())).toSeq
+ (value, headers)
}
-assert(rcvd == data)
+data === rcvd
Review comment:
@srowen It was just a mistake. It is now updated with a deep comparison.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue
AmplabJenkins removed a comment on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue URL: https://github.com/apache/spark/pull/25481#issuecomment-522202062 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14328/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue
AmplabJenkins removed a comment on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue URL: https://github.com/apache/spark/pull/25481#issuecomment-522202061 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue
AmplabJenkins commented on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue URL: https://github.com/apache/spark/pull/25481#issuecomment-522202061 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue
AmplabJenkins commented on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue URL: https://github.com/apache/spark/pull/25481#issuecomment-522202062 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14328/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue
dongjoon-hyun commented on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue URL: https://github.com/apache/spark/pull/25481#issuecomment-522202004 cc @srowen , @HyukjinKwon , @wangyum . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314933373 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: I made a PR. Please take a look~ - https://github.com/apache/spark/pull/25481 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue
SparkQA commented on issue #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency issue URL: https://github.com/apache/spark/pull/25481#issuecomment-522201791 **[Test build #109260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109260/testReport)** for PR 25481 at commit [`c037543`](https://github.com/apache/spark/commit/c037543624ee35dcc8091d8e57abb95858ff4cf7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency…
dongjoon-hyun opened a new pull request #25481: [SPARK-28765][BUILD] Add explict exclusions to avoid JDK11 dependency… URL: https://github.com/apache/spark/pull/25481 … generation issues ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314933206 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: It seems that Maven or Ivy bugs on JDK. After some investigation, the dependency seems not real. I'll make a PR very soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
AmplabJenkins commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-522200906 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14327/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
AmplabJenkins removed a comment on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-522200906 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14327/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
AmplabJenkins commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-522200904 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
AmplabJenkins removed a comment on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-522200904 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
SparkQA commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-522200643 **[Test build #109259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109259/testReport)** for PR 24759 at commit [`66a3775`](https://github.com/apache/spark/commit/66a377575b300eb1cadafa21150b5d97064b3ff3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522200113 **[Test build #109258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109258/testReport)** for PR 24829 at commit [`a2e9e57`](https://github.com/apache/spark/commit/a2e9e5749cadd658950f33d8c8606f3800bb70f8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522200010 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14326/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-52229 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-522200010 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14326/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-52229 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
HyukjinKwon commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314932572 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: Out of curiosity, is it an option to have two releases? - Spark, Hadoop 2.7, JDK 8 - Spark, Hadoop 3.2, JDK 11 Since we currently skipped JDK 9 and 10 and JDK 8 is what we should drop later in the future anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
wangyum commented on a change in pull request #24829: [SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#discussion_r314932491 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/aggregates_part3.sql ## @@ -0,0 +1,273 @@ +-- +-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group +-- +-- +-- AGGREGATES [Part 3] +-- https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/aggregates.sql#L352-L605 + +-- We do not support inheritance tree, skip related tests. +-- try it on an inheritance tree +-- create table minmaxtest(f1 int); +-- create table minmaxtest1() inherits (minmaxtest); +-- create table minmaxtest2() inherits (minmaxtest); +-- create table minmaxtest3() inherits (minmaxtest); +-- create index minmaxtesti on minmaxtest(f1); +-- create index minmaxtest1i on minmaxtest1(f1); +-- create index minmaxtest2i on minmaxtest2(f1 desc); +-- create index minmaxtest3i on minmaxtest3(f1) where f1 is not null; + +-- insert into minmaxtest values(11), (12); +-- insert into minmaxtest1 values(13), (14); +-- insert into minmaxtest2 values(15), (16); +-- insert into minmaxtest3 values(17), (18); + +-- explain (costs off) +-- select min(f1), max(f1) from minmaxtest; +-- select min(f1), max(f1) from minmaxtest; + +-- DISTINCT doesn't do anything useful here, but it shouldn't fail +-- explain (costs off) +-- select distinct min(f1), max(f1) from minmaxtest; +-- select distinct min(f1), max(f1) from minmaxtest; + +-- drop table minmaxtest cascade; + +-- [SPARK-9830] It is not allowed to use an aggregate function in the argument of another aggregate function +-- check for correct detection of nested-aggregate errors Review comment: It's a correct behaviour, `aggregate function calls cannot be nested` https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/expected/aggregates.out#L1078-L1085 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API.
AmplabJenkins commented on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API. URL: https://github.com/apache/spark/pull/25304#issuecomment-522198639 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109255/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API.
AmplabJenkins commented on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API. URL: https://github.com/apache/spark/pull/25304#issuecomment-522198636 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API.
AmplabJenkins removed a comment on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API. URL: https://github.com/apache/spark/pull/25304#issuecomment-522198639 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109255/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API.
AmplabJenkins removed a comment on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API. URL: https://github.com/apache/spark/pull/25304#issuecomment-522198636 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API.
SparkQA removed a comment on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API. URL: https://github.com/apache/spark/pull/25304#issuecomment-522186595 **[Test build #109255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109255/testReport)** for PR 25304 at commit [`cf046df`](https://github.com/apache/spark/commit/cf046dfa7608696feeda2a645e6fed55180f8cd7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API.
SparkQA commented on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API. URL: https://github.com/apache/spark/pull/25304#issuecomment-522198498 **[Test build #109255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109255/testReport)** for PR 25304 at commit [`cf046df`](https://github.com/apache/spark/commit/cf046dfa7608696feeda2a645e6fed55180f8cd7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314932068 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: https://issues.apache.org/jira/browse/SPARK-28765 is filed. In case of `javax.annotation-api-1.2.jar`, the following is verified. The following is removed at https://github.com/apache/spark/commit/c9b49f3978666af5d59a76e1cf2c23ed76f31a2d because the dependency generation is done on JDK8. ``` $ mvn dependency:tree -Dincludes=javax.annotation:javax.annotation-api -Phadoop-3.2 -Phive-thriftserver -Pmesos -Pkubernetes -Pyarn -Phive ... [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ spark-kubernetes_2.12 --- [INFO] org.apache.spark:spark-kubernetes_2.12:jar:3.0.0-SNAPSHOT [INFO] \- io.fabric8:kubernetes-client:jar:4.1.2:compile [INFO]\- io.fabric8:kubernetes-model:jar:4.1.2:compile [INFO] \- javax.annotation:javax.annotation-api:jar:1.2:compile ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins
dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#discussion_r314932068 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -95,12 +95,15 @@ jackson-module-jaxb-annotations-2.9.9.jar jackson-module-paranamer-2.9.9.jar jackson-module-scala_2.12-2.9.9.jar jackson-xc-1.9.13.jar +jakarta.activation-1.2.1.jar +jakarta.activation-api-1.2.1.jar Review comment: https://issues.apache.org/jira/browse/SPARK-28765 is filed. In case of `javax.annotation-api-1.2.jar`, the following is verified. ``` $ mvn dependency:tree -Dincludes=javax.annotation:javax.annotation-api -Phadoop-3.2 -Phive-thriftserver -Pmesos -Pkubernetes -Pyarn -Phive ... [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ spark-kubernetes_2.12 --- [INFO] org.apache.spark:spark-kubernetes_2.12:jar:3.0.0-SNAPSHOT [INFO] \- io.fabric8:kubernetes-client:jar:4.1.2:compile [INFO]\- io.fabric8:kubernetes-model:jar:4.1.2:compile [INFO] \- javax.annotation:javax.annotation-api:jar:1.2:compile ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gczsjdy commented on a change in pull request #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API.
gczsjdy commented on a change in pull request #25304:
[SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API.
URL: https://github.com/apache/spark/pull/25304#discussion_r314931858
##
File path:
core/src/main/java/org/apache/spark/shuffle/api/ShuffleExecutorComponents.java
##
@@ -39,17 +40,39 @@
/**
* Called once per map task to create a writer that will be responsible for
persisting all the
* partitioned bytes written by that map task.
- * @param shuffleId Unique identifier for the shuffle the map task is a
part of
+ *
+ * @param shuffleId Unique identifier for the shuffle the map task is a part
of
* @param mapId Within the shuffle, the identifier of the map task
* @param mapTaskAttemptId Identifier of the task attempt. Multiple attempts
of the same map task
- * with the same (shuffleId, mapId) pair can be
distinguished by the
- * different values of mapTaskAttemptId.
+ * with the same (shuffleId, mapId) pair can be
distinguished by the
+ * different values of mapTaskAttemptId.
* @param numPartitions The number of partitions that will be written by the
map task. Some of
-* these partitions may be empty.
+ * these partitions may be empty.
*/
ShuffleMapOutputWriter createMapOutputWriter(
int shuffleId,
int mapId,
long mapTaskAttemptId,
int numPartitions) throws IOException;
+
+ /**
+ * An optional extension for creating a map output writer that can optimize
the transfer of a
+ * single partition file, as the entire result of a map task, to the backing
store.
+ *
+ * Most implementations should return the default {@link Optional#empty()}
to indicate that
+ * they do not support this optimization. This primarily is for
backwards-compatibility in
Review comment:
Probably it's better to indicate what kinds of implementations may support
this optimization? Otherwise it's confusing. I think the storage that has API
like `move` supports this optimization, is this right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
