[GitHub] [spark] turboFei edited a comment on issue #24447: [SPARK-27562][Shuffle]Complete the verification mechanism for shuffle transmitted data
turboFei edited a comment on issue #24447: [SPARK-27562][Shuffle]Complete the verification mechanism for shuffle transmitted data URL: https://github.com/apache/spark/pull/24447#issuecomment-487256025 @cloud-fan Could you help to review this? I think this pr can guarantee the accuracy of shuffle data transmission efficiently. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on issue #24447: [SPARK-27562][Shuffle]Complete the verification mechanism for shuffle transmitted data
turboFei commented on issue #24447: [SPARK-27562][Shuffle]Complete the verification mechanism for shuffle transmitted data URL: https://github.com/apache/spark/pull/24447#issuecomment-487256025 @cloud-fan Could you help to review this? I think this pr can guarantee the accuracy of shuffle data transmission. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jashgala commented on issue #23748: [SPARK-23619][DOCS] Add output description for some generator expressions / functions
jashgala commented on issue #23748: [SPARK-23619][DOCS] Add output description for some generator expressions / functions URL: https://github.com/apache/spark/pull/23748#issuecomment-487255993 Thanks I will fix the other documentation when I have some free time! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins
AmplabJenkins removed a comment on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins URL: https://github.com/apache/spark/pull/24044#issuecomment-487254936 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10256/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins
AmplabJenkins removed a comment on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins URL: https://github.com/apache/spark/pull/24044#issuecomment-487254935 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins
AmplabJenkins commented on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins URL: https://github.com/apache/spark/pull/24044#issuecomment-487254936 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10256/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins
AmplabJenkins commented on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins URL: https://github.com/apache/spark/pull/24044#issuecomment-487254935 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #23674: [SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record count
dongjoon-hyun commented on issue #23674: [SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record count URL: https://github.com/apache/spark/pull/23674#issuecomment-487254867 Hi, @sumitsu and @HyukjinKwon . I backported this to `branch-2.4` since Spark 2.4.x will be the last version of `2.x` lines. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak
JoshRosen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak URL: https://github.com/apache/spark/pull/24461#issuecomment-487254617 Does this component have exclusive ownership of this Hadoop `FileSystem` instance? Could we run into problems in case this ends up closing a shared instance? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins
SparkQA commented on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins URL: https://github.com/apache/spark/pull/24044#issuecomment-487254579 **[Test build #104955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104955/testReport)** for PR 24044 at commit [`bdc1d2c`](https://github.com/apache/spark/commit/bdc1d2c666deb815bfb267cd34fb45f430fc6519). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins
wangyum commented on issue #24044: [WIP][test-hadoop3.2][test-maven] Test Hadoop 3.2 on jenkins URL: https://github.com/apache/spark/pull/24044#issuecomment-487254535 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
AmplabJenkins removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487254452 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
AmplabJenkins removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487254453 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104953/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
AmplabJenkins commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487254453 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104953/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
AmplabJenkins commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487254452 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
SparkQA removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487247316 **[Test build #104953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104953/testReport)** for PR 24431 at commit [`2bc7086`](https://github.com/apache/spark/commit/2bc7086584bb271fd32cc980fb8168f06ab526bd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
SparkQA commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487254360 **[Test build #104953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104953/testReport)** for PR 24431 at commit [`2bc7086`](https://github.com/apache/spark/commit/2bc7086584bb271fd32cc980fb8168f06ab526bd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
dongjoon-hyun closed pull request #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
AmplabJenkins removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487252515 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104951/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
AmplabJenkins commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487252515 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104951/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
AmplabJenkins removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487252514 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
AmplabJenkins commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487252514 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
SparkQA removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487228208 **[Test build #104951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104951/testReport)** for PR 24476 at commit [`dfd6d6c`](https://github.com/apache/spark/commit/dfd6d6c7d0068ad1dc228eaa26075bd6098cd0c4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
SparkQA commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487252422 **[Test build #104951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104951/testReport)** for PR 24476 at commit [`dfd6d6c`](https://github.com/apache/spark/commit/dfd6d6c7d0068ad1dc228eaa26075bd6098cd0c4). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` trait BaseErrorHandler extends Closeable ` * ` class ErrorHandlingReadableChannel(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
viirya commented on a change in pull request #24431:
[SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of
scala.language.existentials
URL: https://github.com/apache/spark/pull/24431#discussion_r279141550
##
File path: core/src/test/scala/org/apache/spark/FileSuite.scala
##
@@ -206,8 +206,9 @@ class FileSuite extends SparkFunSuite with
LocalSparkContext {
Utils.withContextClassLoader(loader) {
sc = new SparkContext("local", "test")
- val objs = sc.makeRDD(1 to 3).map { x =>
-Utils.classForName(className, noSparkClassLoader =
true).getConstructor().newInstance()
+val objs = sc.makeRDD(1 to 3).map { _ =>
+Utils.classForName[AnyRef](className, noSparkClassLoader = true).
Review comment:
The indent looks incorrect here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT
SparkQA commented on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487249036 **[Test build #104954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104954/testReport)** for PR 24468 at commit [`01c91a7`](https://github.com/apache/spark/commit/01c91a71b4a817cd319c1d551601abfba7b6bfab). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248928 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10255/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248925 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248928 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10255/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248925 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT
wangyum commented on issue #24468: [WIP][TEST][test-hadoop3.2][test-maven] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248591 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248345 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104952/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
SparkQA removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487247320 **[Test build #104952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104952/testReport)** for PR 24468 at commit [`01c91a7`](https://github.com/apache/spark/commit/01c91a71b4a817cd319c1d551601abfba7b6bfab). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248343 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248345 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104952/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
SparkQA commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248338 **[Test build #104952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104952/testReport)** for PR 24468 at commit [`01c91a7`](https://github.com/apache/spark/commit/01c91a71b4a817cd319c1d551601abfba7b6bfab). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487248343 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
AmplabJenkins commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487247617 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
AmplabJenkins commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487247618 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10254/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
AmplabJenkins removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487247617 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
AmplabJenkins removed a comment on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487247618 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10254/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
SparkQA commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487247320 **[Test build #104952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104952/testReport)** for PR 24468 at commit [`01c91a7`](https://github.com/apache/spark/commit/01c91a71b4a817cd319c1d551601abfba7b6bfab). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
SparkQA commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487247316 **[Test build #104953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104953/testReport)** for PR 24431 at commit [`2bc7086`](https://github.com/apache/spark/commit/2bc7086584bb271fd32cc980fb8168f06ab526bd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487247220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10253/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487247219 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials
dongjoon-hyun commented on issue #24431: [SPARK-27536][CORE][ML][SQL][STREAMING] Remove most use of scala.language.existentials URL: https://github.com/apache/spark/pull/24431#issuecomment-487247230 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins removed a comment on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487247220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10253/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT
AmplabJenkins commented on issue #24468: [WIP][TEST][test-hadoop3.2] Verify Hive 2.3.5-SNAPSHOT URL: https://github.com/apache/spark/pull/24468#issuecomment-487247219 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #23748: [SPARK-23619][DOCS] Add output description for some generator expressions / functions
HyukjinKwon closed pull request #23748: [SPARK-23619][DOCS] Add output description for some generator expressions / functions URL: https://github.com/apache/spark/pull/23748 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #24466: [MINOR][TEST][DOC] Execute action miss name message
cloud-fan closed pull request #24466: [MINOR][TEST][DOC] Execute action miss name message URL: https://github.com/apache/spark/pull/24466 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #23748: [SPARK-23619][DOCS] Add output description for some generator expressions / functions
HyukjinKwon commented on issue #23748: [SPARK-23619][DOCS] Add output description for some generator expressions / functions URL: https://github.com/apache/spark/pull/23748#issuecomment-487243731 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #24466: [MINOR][TEST][DOC] Execute action miss name message
cloud-fan commented on issue #24466: [MINOR][TEST][DOC] Execute action miss name message URL: https://github.com/apache/spark/pull/24466#issuecomment-487243676 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #24372: [SPARK-27462][SQL] Enhance insert into hive table that could choose some columns in target table flexibly.
beliefer commented on issue #24372: [SPARK-27462][SQL] Enhance insert into hive table that could choose some columns in target table flexibly. URL: https://github.com/apache/spark/pull/24372#issuecomment-487243349 cc @vanzin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected
AmplabJenkins removed a comment on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected URL: https://github.com/apache/spark/pull/24473#issuecomment-487242228 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104947/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected
AmplabJenkins removed a comment on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected URL: https://github.com/apache/spark/pull/24473#issuecomment-487242227 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected
AmplabJenkins commented on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected URL: https://github.com/apache/spark/pull/24473#issuecomment-487242227 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected
AmplabJenkins commented on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected URL: https://github.com/apache/spark/pull/24473#issuecomment-487242228 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104947/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected
SparkQA removed a comment on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected URL: https://github.com/apache/spark/pull/24473#issuecomment-487218579 **[Test build #104947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104947/testReport)** for PR 24473 at commit [`e26053d`](https://github.com/apache/spark/commit/e26053d4cc222f0291c3a30262d9133e2523a37a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected
SparkQA commented on issue #24473: [SPARK-27534][SQL] Do not load `content` column in binary data source if it is not selected URL: https://github.com/apache/spark/pull/24473#issuecomment-487242077 **[Test build #104947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104947/testReport)** for PR 24473 at commit [`e26053d`](https://github.com/apache/spark/commit/e26053d4cc222f0291c3a30262d9133e2523a37a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL]
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r279136736
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala
##
@@ -172,15 +172,17 @@ class PlannerSuite extends SharedSQLContext {
}
test("SPARK-11390 explain should print PushedFilters of PhysicalRDD") {
-withTempPath { file =>
- val path = file.getCanonicalPath
- testData.write.parquet(path)
- val df = spark.read.parquet(path)
- df.createOrReplaceTempView("testPushed")
-
- withTempView("testPushed") {
-val exp = sql("select * from testPushed where key =
15").queryExecution.sparkPlan
-assert(exp.toString.contains("PushedFilters: [IsNotNull(key),
EqualTo(key,15)]"))
+withSQLConf(SQLConf.USE_V1_SOURCE_READER_LIST.key -> "parquet") {
+ withTempPath { file =>
+val path = file.getCanonicalPath
+testData.write.parquet(path)
+val df = spark.read.parquet(path)
+df.createOrReplaceTempView("testPushed")
+
+withTempView("testPushed") {
+ val exp = sql("select * from testPushed where key =
15").queryExecution.sparkPlan
+ assert(exp.toString.contains("PushedFilters: [IsNotNull(key),
EqualTo(key,15)]"))
Review comment:
Why not to `collectFirst` to find the expected `FileSourceScanExec` operator
and request for the `PushedFilters` instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL]
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r279136736
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala
##
@@ -172,15 +172,17 @@ class PlannerSuite extends SharedSQLContext {
}
test("SPARK-11390 explain should print PushedFilters of PhysicalRDD") {
-withTempPath { file =>
- val path = file.getCanonicalPath
- testData.write.parquet(path)
- val df = spark.read.parquet(path)
- df.createOrReplaceTempView("testPushed")
-
- withTempView("testPushed") {
-val exp = sql("select * from testPushed where key =
15").queryExecution.sparkPlan
-assert(exp.toString.contains("PushedFilters: [IsNotNull(key),
EqualTo(key,15)]"))
+withSQLConf(SQLConf.USE_V1_SOURCE_READER_LIST.key -> "parquet") {
+ withTempPath { file =>
+val path = file.getCanonicalPath
+testData.write.parquet(path)
+val df = spark.read.parquet(path)
+df.createOrReplaceTempView("testPushed")
+
+withTempView("testPushed") {
+ val exp = sql("select * from testPushed where key =
15").queryExecution.sparkPlan
+ assert(exp.toString.contains("PushedFilters: [IsNotNull(key),
EqualTo(key,15)]"))
Review comment:
Why not to `collectFirst` to find the expected scan physical operator and
request for the `PushedFilters` instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL]
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r279136632
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/DataSourceScanExecRedactionSuite.scala
##
@@ -28,8 +28,10 @@ import org.apache.spark.sql.test.SharedSQLContext
*/
class DataSourceScanExecRedactionSuite extends QueryTest with SharedSQLContext
{
+ // TODO: create test suite for Data source V2 as well.
override protected def sparkConf: SparkConf = super.sparkConf
.set("spark.redaction.string.regex", "file:/[\\w_]+")
+.set(SQLConf.USE_V1_SOURCE_READER_LIST.key, "parquet")
Review comment:
That could benefit from the object with `val name = parquet`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL]
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r279136642
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala
##
@@ -172,15 +172,17 @@ class PlannerSuite extends SharedSQLContext {
}
test("SPARK-11390 explain should print PushedFilters of PhysicalRDD") {
-withTempPath { file =>
- val path = file.getCanonicalPath
- testData.write.parquet(path)
- val df = spark.read.parquet(path)
- df.createOrReplaceTempView("testPushed")
-
- withTempView("testPushed") {
-val exp = sql("select * from testPushed where key =
15").queryExecution.sparkPlan
-assert(exp.toString.contains("PushedFilters: [IsNotNull(key),
EqualTo(key,15)]"))
+withSQLConf(SQLConf.USE_V1_SOURCE_READER_LIST.key -> "parquet") {
Review comment:
That could benefit from the object with `val name = parquet`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL]
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r279136626
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -2979,37 +2979,40 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
test("SPARK-26709: OptimizeMetadataOnlyQuery does not handle empty records
correctly") {
-Seq(true, false).foreach { enableOptimizeMetadataOnlyQuery =>
- withSQLConf(SQLConf.OPTIMIZER_METADATA_ONLY.key ->
enableOptimizeMetadataOnlyQuery.toString) {
-withTable("t") {
- sql("CREATE TABLE t (col1 INT, p1 INT) USING PARQUET PARTITIONED BY
(p1)")
- sql("INSERT INTO TABLE t PARTITION (p1 = 5) SELECT ID FROM range(1,
1)")
- if (enableOptimizeMetadataOnlyQuery) {
-// The result is wrong if we enable the configuration.
-checkAnswer(sql("SELECT MAX(p1) FROM t"), Row(5))
- } else {
-checkAnswer(sql("SELECT MAX(p1) FROM t"), Row(null))
+withSQLConf(SQLConf.USE_V1_SOURCE_READER_LIST.key -> "parquet") {
Review comment:
That could benefit from the object with `val name = parquet`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL]
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r279136390
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
##
@@ -377,7 +377,7 @@ class FileBasedDataSourceSuite extends QueryTest with
SharedSQLContext with Befo
// TODO: test file source V2 after write path is fixed.
Seq(true).foreach { useV1 =>
val useV1List = if (useV1) {
-"csv,orc"
+"csv,orc,parquet"
Review comment:
nit: Use `true` directly (and cut the remaining lines)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
AmplabJenkins removed a comment on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2 URL: https://github.com/apache/spark/pull/24327#issuecomment-487239345 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104948/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
SparkQA removed a comment on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2 URL: https://github.com/apache/spark/pull/24327#issuecomment-487220039 **[Test build #104948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104948/testReport)** for PR 24327 at commit [`30d88cb`](https://github.com/apache/spark/commit/30d88cb8fa219b76bdafba32d74194179409e62a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
AmplabJenkins removed a comment on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2 URL: https://github.com/apache/spark/pull/24327#issuecomment-487239343 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
AmplabJenkins commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2 URL: https://github.com/apache/spark/pull/24327#issuecomment-487239345 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104948/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
AmplabJenkins commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2 URL: https://github.com/apache/spark/pull/24327#issuecomment-487239343 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
SparkQA commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2 URL: https://github.com/apache/spark/pull/24327#issuecomment-487239266 **[Test build #104948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104948/testReport)** for PR 24327 at commit [`30d88cb`](https://github.com/apache/spark/commit/30d88cb8fa219b76bdafba32d74194179409e62a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL]
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r279136012
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetDataSourceV2.scala
##
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.v2.parquet
+
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat
+import org.apache.spark.sql.execution.datasources.v2._
+import org.apache.spark.sql.sources.v2.Table
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+
+class ParquetDataSourceV2 extends FileDataSourceV2 {
+
+ override def fallbackFileFormat: Class[_ <: FileFormat] =
classOf[ParquetFileFormat]
+
+ override def shortName(): String = "parquet"
Review comment:
nit: Use object with a val (for code reuse)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
jaceklaskowski commented on a change in pull request #24327: [SPARK-27418][SQL]
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r279135671
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1494,7 +1494,7 @@ object SQLConf {
" register class names for which data source V2 write paths are
disabled. Writes from these" +
" sources will fall back to the V1 sources.")
.stringConf
-.createWithDefault("csv,json,orc,text")
+.createWithDefault("csv,json,orc,text,parquet")
Review comment:
Just an idea: what about using aliases (name) of the respective providers?
That would make the list more type-safe and easily searchable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results
AmplabJenkins commented on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results URL: https://github.com/apache/spark/pull/24475#issuecomment-487232257 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104945/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results
AmplabJenkins removed a comment on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results URL: https://github.com/apache/spark/pull/24475#issuecomment-487232257 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104945/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results
AmplabJenkins removed a comment on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results URL: https://github.com/apache/spark/pull/24475#issuecomment-487232256 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results
AmplabJenkins commented on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results URL: https://github.com/apache/spark/pull/24475#issuecomment-487232256 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results
SparkQA removed a comment on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results URL: https://github.com/apache/spark/pull/24475#issuecomment-487196203 **[Test build #104945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104945/testReport)** for PR 24475 at commit [`0130f07`](https://github.com/apache/spark/commit/0130f07d80f2fa2f018ce86942bd64f07144013c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results
SparkQA commented on issue #24475: [SPARK-27580][SQL] Implement `doCanonicalize` in BatchScanExec for comparing query plan results URL: https://github.com/apache/spark/pull/24475#issuecomment-487232018 **[Test build #104945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104945/testReport)** for PR 24475 at commit [`0130f07`](https://github.com/apache/spark/commit/0130f07d80f2fa2f018ce86942bd64f07144013c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #24442: [SPARK-27547][SQL] fix DataFrame self-join problems
dilipbiswal commented on issue #24442: [SPARK-27547][SQL] fix DataFrame self-join problems URL: https://github.com/apache/spark/pull/24442#issuecomment-487229820 @cloud-fan The changes look good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
AmplabJenkins removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487227907 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10252/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
AmplabJenkins removed a comment on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487227901 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
SparkQA commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487228208 **[Test build #104951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104951/testReport)** for PR 24476 at commit [`dfd6d6c`](https://github.com/apache/spark/commit/dfd6d6c7d0068ad1dc228eaa26075bd6098cd0c4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on a change in pull request #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value
vanzin commented on a change in pull request #24465: [SPARK-27575][CORE][YARN]
Yarn file-related confs should merge new value with existing value
URL: https://github.com/apache/spark/pull/24465#discussion_r279127545
##
File path: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
##
@@ -897,6 +897,74 @@ class SparkSubmitSuite
}
}
+ test("SPARK-27575: yarn confs should merge new value with existing value") {
+val tmpJarDir = Utils.createTempDir()
+val jar1 = TestUtils.createJarWithFiles(Map("test.resource" -> "1"),
tmpJarDir)
+val jar2 = TestUtils.createJarWithFiles(Map("test.resource" -> "USER"),
tmpJarDir)
+
+val tmpJarDirYarnOpt = Utils.createTempDir()
+val jar1YarnOpt = TestUtils.createJarWithFiles(Map("test.resource" ->
"2"), tmpJarDirYarnOpt)
+val jar2YarnOpt = TestUtils.createJarWithFiles(Map("test.resource" ->
"USER2"),
+ tmpJarDirYarnOpt)
+
+val tmpFileDir = Utils.createTempDir()
+val file1 = File.createTempFile("tmpFile1", "", tmpFileDir)
+val file2 = File.createTempFile("tmpFile2", "", tmpFileDir)
+
+val tmpFileDirYarnOpt = Utils.createTempDir()
+val file1YarnOpt = File.createTempFile("tmpPy1YarnOpt", ".py",
tmpFileDirYarnOpt)
+val file2YarnOpt = File.createTempFile("tmpPy2YarnOpt", ".egg",
tmpFileDirYarnOpt)
+
+val tmpPyFileDir = Utils.createTempDir()
+val pyFile1 = File.createTempFile("tmpPy1", ".py", tmpPyFileDir)
+val pyFile2 = File.createTempFile("tmpPy2", ".egg", tmpPyFileDir)
+
+val tmpPyFileDirYarnOpt = Utils.createTempDir()
+val pyFile1YarnOpt = File.createTempFile("tmpPy1YarnOpt", ".py",
tmpPyFileDirYarnOpt)
+val pyFile2YarnOpt = File.createTempFile("tmpPy2YarnOpt", ".egg",
tmpPyFileDirYarnOpt)
+
+val tmpArchiveDir = Utils.createTempDir()
+val archive1 = File.createTempFile("archive1", ".zip", tmpArchiveDir)
+val archive2 = File.createTempFile("archive2", ".zip", tmpArchiveDir)
+
+val tmpArchiveDirYarnOpt = Utils.createTempDir()
+val archive1YarnOpt = File.createTempFile("archive1YarnOpt", ".zip",
tmpArchiveDirYarnOpt)
+val archive2YarnOpt = File.createTempFile("archive2YarnOpt", ".zip",
tmpArchiveDirYarnOpt)
+
+val tempPyFile = File.createTempFile("tmpApp", ".py")
+tempPyFile.deleteOnExit()
+
+val args = Seq(
+ "--class", UserClasspathFirstTest.getClass.getName.stripPrefix("$"),
+ "--name", "testApp",
+ "--master", "yarn",
+ "--deploy-mode", "client",
+ "--jars", s"${tmpJarDir.getAbsolutePath}/*.jar",
+ "--files", s"${tmpFileDir.getAbsolutePath}/tmpFile*",
+ "--py-files", s"${tmpPyFileDir.getAbsolutePath}/tmpPy*",
+ "--archives", s"${tmpArchiveDir.getAbsolutePath}/*.zip",
+ "--conf", "spark.yarn.dist.files=" +
+s"${Seq(file1YarnOpt,
file2YarnOpt).map(_.getAbsolutePath).mkString(",")}",
+ "--conf", "spark.yarn.dist.pyFiles=" +
+s"${Seq(pyFile1YarnOpt,
pyFile2YarnOpt).map(_.getAbsolutePath).mkString(",")}",
+ "--conf", "spark.yarn.dist.jars=" +
+s"${Seq(jar1YarnOpt,
jar2YarnOpt).map(_.toURI.toString).mkString(",")}",
+ "--conf", "spark.yarn.dist.archives=" +
+s"${Seq(archive1YarnOpt,
archive2YarnOpt).map(_.toURI.toString).mkString(",")}",
+ tempPyFile.toURI().toString())
+
+val appArgs = new SparkSubmitArguments(args)
+val (_, _, conf, _) = submit.prepareSubmitEnvironment(appArgs)
+conf.get("spark.yarn.dist.jars").split(",").toSet should be
+(Set(Seq(jar1, jar2, jar1YarnOpt,
jar2YarnOpt).map(_.toURI.toString).toList))
Review comment:
nit: indent these continuation lines
(I'd actually rather use `assert(blah === blah)` so that the parentheses
wrap the whole thing.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
AmplabJenkins commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487227907 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10252/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
AmplabJenkins commented on issue #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476#issuecomment-487227901 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #22557: [SPARK-25535][core] Work around bad error handling in commons-crypto.
dongjoon-hyun commented on issue #22557: [SPARK-25535][core] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/22557#issuecomment-487227701 Thank you so much, @vanzin and @dbtsai ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on issue #22557: [SPARK-25535][core] Work around bad error handling in commons-crypto.
vanzin commented on issue #22557: [SPARK-25535][core] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/22557#issuecomment-487227641 #24476 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin opened a new pull request #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto.
vanzin opened a new pull request #24476: [SPARK-25535][CORE][branch-2.4] Work around bad error handling in commons-crypto. URL: https://github.com/apache/spark/pull/24476 The commons-crypto library does some questionable error handling internally, which can lead to JVM crashes if some call into native code fails and cleans up state it should not. While the library is not fixed, this change adds some workarounds in Spark code so that when an error is detected in the commons-crypto side, Spark avoids calling into the library further. Tested with existing and added unit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows
AmplabJenkins removed a comment on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows URL: https://github.com/apache/spark/pull/24448#issuecomment-487226974 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows
AmplabJenkins removed a comment on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows URL: https://github.com/apache/spark/pull/24448#issuecomment-487226976 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104949/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows
AmplabJenkins commented on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows URL: https://github.com/apache/spark/pull/24448#issuecomment-487226974 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows
AmplabJenkins commented on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows URL: https://github.com/apache/spark/pull/24448#issuecomment-487226976 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104949/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows
SparkQA removed a comment on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows URL: https://github.com/apache/spark/pull/24448#issuecomment-487221392 **[Test build #104949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104949/testReport)** for PR 24448 at commit [`ae011c0`](https://github.com/apache/spark/commit/ae011c0217e07702405d8e9a5ca6a53fd3b50626). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows
SparkQA commented on issue #24448: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows URL: https://github.com/apache/spark/pull/24448#issuecomment-487226778 **[Test build #104949 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104949/testReport)** for PR 24448 at commit [`ae011c0`](https://github.com/apache/spark/commit/ae011c0217e07702405d8e9a5ca6a53fd3b50626). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value
AmplabJenkins commented on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value URL: https://github.com/apache/spark/pull/24465#issuecomment-487226559 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value
AmplabJenkins removed a comment on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value URL: https://github.com/apache/spark/pull/24465#issuecomment-487226561 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104946/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value
AmplabJenkins removed a comment on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value URL: https://github.com/apache/spark/pull/24465#issuecomment-487226559 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value
AmplabJenkins commented on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value URL: https://github.com/apache/spark/pull/24465#issuecomment-487226561 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104946/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value
SparkQA removed a comment on issue #24465: [SPARK-27575][CORE][YARN] Yarn file-related confs should merge new value with existing value URL: https://github.com/apache/spark/pull/24465#issuecomment-487198148 **[Test build #104946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104946/testReport)** for PR 24465 at commit [`50191a0`](https://github.com/apache/spark/commit/50191a03c57cf0d460e2ca936eb0bf25c0510914). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
