[GitHub] [spark] maropu commented on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name
maropu commented on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name URL: https://github.com/apache/spark/pull/26008#issuecomment-537800095 nit: we have no consistent name rule for that? Since `Json` is more popular, `JSONBenchmark` -> `JsonBenchmark` instead of `JsonBenchmark.scala` -> `JSONBenchmark.scala`? ``` $ find . -type f | grep -e "\.scala$" | grep Json ./core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ./core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ./core/src/test/scala/org/apache/spark/deploy/JsonProtocolSuite.scala ./core/src/test/scala/org/apache/spark/JsonTestUtils.scala ./core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala ./external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala ./external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/JsonUtilsSuite.scala ./mllib/src/main/scala/org/apache/spark/ml/linalg/JsonMatrixConverter.scala ./mllib/src/main/scala/org/apache/spark/ml/linalg/JsonVectorConverter.scala ./mllib/src/test/scala/org/apache/spark/ml/linalg/JsonMatrixConverterSuite.scala ./mllib/src/test/scala/org/apache/spark/ml/linalg/JsonVectorConverterSuite.scala ./sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala ./sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala ./sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JsonInferSchemaSuite.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonOutputWriter.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonUtils.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonDataSourceV2.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonPartitionReaderFactory.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScan.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonTable.scala ./sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonWriteBuilder.scala ./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmark.scala ./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonParsingOptionsSuite.scala ./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/TestJsonData.scala ./sql/core/src/test/scala/org/apache/spark/sql/execution/SQLJsonProtocolSuite.scala ./sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ./sql/hive/src/test/scala/org/apache/spark/sql/sources/JsonHadoopFsRelationSuite.scala $ find . -type f | grep -e "\.scala$" | grep JSON ./sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537798069 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111709/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537798062 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537798062 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537798069 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111709/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
SparkQA commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537797631 **[Test build #111709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111709/testReport)** for PR 25993 at commit [`1e2440c`](https://github.com/apache/spark/commit/1e2440cda26529af493a6f2336f4617a39676748). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
SparkQA removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537764496 **[Test build #111709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111709/testReport)** for PR 25993 at commit [`1e2440c`](https://github.com/apache/spark/commit/1e2440cda26529af493a6f2336f4617a39676748). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
AmplabJenkins removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537795945 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
AmplabJenkins removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537795949 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111708/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
AmplabJenkins commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537795945 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
AmplabJenkins commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537795949 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111708/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
SparkQA removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537755364 **[Test build #111708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111708/testReport)** for PR 26001 at commit [`79f94f5`](https://github.com/apache/spark/commit/79f94f59c1298c021f71b65acfdb850161075545). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
SparkQA commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537795578 **[Test build #111708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111708/testReport)** for PR 26001 at commit [`79f94f5`](https://github.com/apache/spark/commit/79f94f59c1298c021f71b65acfdb850161075545). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] karuppayya commented on issue #25995: SPARK-29324: Fix overwrite behaviour for saveAsTable
karuppayya commented on issue #25995: SPARK-29324: Fix overwrite behaviour for saveAsTable URL: https://github.com/apache/spark/pull/25995#issuecomment-537790057 Can someone help review this change This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #26004: [SPARK-29142][PYTHON][ML][FOLLOWUP][DOC] Replace incorrect :py:attr: applications
huaxingao commented on issue #26004: [SPARK-29142][PYTHON][ML][FOLLOWUP][DOC] Replace incorrect :py:attr: applications URL: https://github.com/apache/spark/pull/26004#issuecomment-537787996 LGTM. Thanks for catching the problem. @zero323 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data
viirya commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data URL: https://github.com/apache/spark/pull/25979#issuecomment-537784884 I am not sure if this is designed behavior of Hive. You can see a similar issue at Hive https://jira.apache.org/jira/browse/HIVE-18702. INSERT OVERWRITE should clean table directory. The last comment in that ticket, someone left a comment describing the same case as this. He said he will create a ticket, but I do not see that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537784438 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16707/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537784433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537784438 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16707/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537784433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
SparkQA commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537784158 **[Test build #111714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111714/testReport)** for PR 25993 at commit [`0759bcc`](https://github.com/apache/spark/commit/0759bcc4775edb2896ddd9218884de202d77e8ff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object
AmplabJenkins removed a comment on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object URL: https://github.com/apache/spark/pull/26007#issuecomment-537779887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object
AmplabJenkins removed a comment on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object URL: https://github.com/apache/spark/pull/26007#issuecomment-537779896 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111707/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object
AmplabJenkins commented on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object URL: https://github.com/apache/spark/pull/26007#issuecomment-537779887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object
AmplabJenkins commented on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object URL: https://github.com/apache/spark/pull/26007#issuecomment-537779896 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111707/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object
SparkQA removed a comment on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object URL: https://github.com/apache/spark/pull/26007#issuecomment-537741121 **[Test build #111707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111707/testReport)** for PR 26007 at commit [`215df8e`](https://github.com/apache/spark/commit/215df8e8956f237c7caa5afb010bd516ba2e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object
SparkQA commented on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object URL: https://github.com/apache/spark/pull/26007#issuecomment-537779368 **[Test build #111707 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111707/testReport)** for PR 26007 at commit [`215df8e`](https://github.com/apache/spark/commit/215df8e8956f237c7caa5afb010bd516ba2e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #20640: [SPARK-19755][Mesos] Blacklist is always active for MesosCoarseGrainedSchedulerBackend
dongjoon-hyun commented on issue #20640: [SPARK-19755][Mesos] Blacklist is always active for MesosCoarseGrainedSchedulerBackend URL: https://github.com/apache/spark/pull/20640#issuecomment-537778659 Got it, @squito . I'll follow your advice. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state.
AmplabJenkins removed a comment on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state. URL: https://github.com/apache/spark/pull/25951#issuecomment-537776749 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16706/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state.
AmplabJenkins removed a comment on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state. URL: https://github.com/apache/spark/pull/25951#issuecomment-537776743 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state.
AmplabJenkins commented on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state. URL: https://github.com/apache/spark/pull/25951#issuecomment-537776749 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16706/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state.
AmplabJenkins commented on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state. URL: https://github.com/apache/spark/pull/25951#issuecomment-537776743 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11
AmplabJenkins removed a comment on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11 URL: https://github.com/apache/spark/pull/26003#issuecomment-537776381 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11
AmplabJenkins removed a comment on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11 URL: https://github.com/apache/spark/pull/26003#issuecomment-537776386 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111706/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state.
SparkQA commented on issue #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state. URL: https://github.com/apache/spark/pull/25951#issuecomment-537776459 **[Test build #111713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111713/testReport)** for PR 25951 at commit [`f0a662f`](https://github.com/apache/spark/commit/f0a662f48fc3a3c17ea468f44f16613f23b039b3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11
AmplabJenkins commented on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11 URL: https://github.com/apache/spark/pull/26003#issuecomment-537776381 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11
AmplabJenkins commented on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11 URL: https://github.com/apache/spark/pull/26003#issuecomment-537776386 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111706/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11
SparkQA commented on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11 URL: https://github.com/apache/spark/pull/26003#issuecomment-537776033 **[Test build #111706 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111706/testReport)** for PR 26003 at commit [`9eb1e83`](https://github.com/apache/spark/commit/9eb1e83c88d9fedceacfea59b455e44617398573). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11
SparkQA removed a comment on issue #26003: [WIP][SPARK-29320][TESTS] Compare `sql/core` module in JDK8/11 URL: https://github.com/apache/spark/pull/26003#issuecomment-537738004 **[Test build #111706 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111706/testReport)** for PR 26003 at commit [`9eb1e83`](https://github.com/apache/spark/commit/9eb1e83c88d9fedceacfea59b455e44617398573). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25816: [SPARK-29107][SQL][TESTS] Port window.sql (Part 1)
dongjoon-hyun commented on issue #25816: [SPARK-29107][SQL][TESTS] Port window.sql (Part 1) URL: https://github.com/apache/spark/pull/25816#issuecomment-537774869 Hmm. Still `Result did not match for query ` for the following. ``` SELECT four, ten/4 as two, sum(ten/4) over (partition by four order by ten/4 range between unbounded preceding and current row), last(ten/4) over (partition by four order by ten/4 range between unbounded preceding and current row) FROM (select distinct ten, four from tenk1) ss ``` @DylanGuedes . Could you post the query result in both `spark-shell` and `spark-sql`? Previously, there was a bug which the result is different in both shells and `spark-sql` is similar with Spark Thrift Server. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on a change in pull request #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
felixcheung commented on a change in pull request #25993: [SPARK-29339][R]
Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
URL: https://github.com/apache/spark/pull/25993#discussion_r330855381
##
File path: appveyor.yml
##
@@ -46,9 +46,9 @@ install:
# Here, we use the fixed version of testthat. For more details, please see
SPARK-22817.
# As of devtools 2.1.0, it requires testthat higher then 2.1.1 as a
dependency. SparkR test requires testthat 1.0.2.
# Therefore, we don't use devtools but installs it directly from the archive
including its dependencies.
- - cmd: R -e "install.packages(c('crayon', 'praise', 'R6'),
repos='https://cloud.r-project.org/')"
+ - cmd: R -e "install.packages(c('crayon', 'praise', 'R6', 'arrow'),
repos='https://cloud.r-project.org/')"
- cmd: R -e
"install.packages('https://cloud.r-project.org/src/contrib/Archive/testthat/testthat_1.0.2.tar.gz',
repos=NULL, type='source')"
- - cmd: R -e "packageVersion('knitr'); packageVersion('rmarkdown');
packageVersion('testthat'); packageVersion('e1071'); packageVersion('survival')"
+ - cmd: R -e "packageVersion('knitr'); packageVersion('rmarkdown');
packageVersion('testthat'); packageVersion('e1071');
packageVersion('survival'); packageVersion('arrow')"
Review comment:
SG
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data
felixcheung commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data URL: https://github.com/apache/spark/pull/25979#issuecomment-537773757 or we do > exception thrown (Hive 2.3.2) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data
felixcheung commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data URL: https://github.com/apache/spark/pull/25979#issuecomment-537773541 yeah, this is a hard one, obviously the behavior is buggy, hard to detect etc. but that's how Hive is designed. I think we should log a warn in Spark at least so interested folks (like us) can detect this after the job is run > On Hive 2.1.0, two "INSERT OVERWRITE" produces data file with same name like 00_0. The second "INSERT OVERWRITE" moves the file into and overwrite old file. > On Hive 2.3.2, the second "INSERT OVERWRITE" causes following failure when moving file with same name we can't really rely on the name being the same to overwrite. it depends on a number of things. for instance, if the original partition has 10B row and 1M file, overwritten with new partition having 1B and 100k file, then a lot of files are not going to be overwritten (like 900k) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
HyukjinKwon commented on a change in pull request #25993: [SPARK-29339][R]
Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
URL: https://github.com/apache/spark/pull/25993#discussion_r330854715
##
File path: appveyor.yml
##
@@ -46,9 +46,9 @@ install:
# Here, we use the fixed version of testthat. For more details, please see
SPARK-22817.
# As of devtools 2.1.0, it requires testthat higher then 2.1.1 as a
dependency. SparkR test requires testthat 1.0.2.
# Therefore, we don't use devtools but installs it directly from the archive
including its dependencies.
- - cmd: R -e "install.packages(c('crayon', 'praise', 'R6'),
repos='https://cloud.r-project.org/')"
+ - cmd: R -e "install.packages(c('crayon', 'praise', 'R6', 'arrow'),
repos='https://cloud.r-project.org/')"
- cmd: R -e
"install.packages('https://cloud.r-project.org/src/contrib/Archive/testthat/testthat_1.0.2.tar.gz',
repos=NULL, type='source')"
- - cmd: R -e "packageVersion('knitr'); packageVersion('rmarkdown');
packageVersion('testthat'); packageVersion('e1071'); packageVersion('survival')"
+ - cmd: R -e "packageVersion('knitr'); packageVersion('rmarkdown');
packageVersion('testthat'); packageVersion('e1071');
packageVersion('survival'); packageVersion('arrow')"
Review comment:
Maybe .. it will needs to install `devtools` which upgrades `testthat` ...
so it's a bit tricky to install a specific version for now.
I was thinking about testing the latest in AppVeyor and old version at
Jenkins anyway. should be fine.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on a change in pull request #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
felixcheung commented on a change in pull request #25993: [SPARK-29339][R]
Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
URL: https://github.com/apache/spark/pull/25993#discussion_r330854350
##
File path: appveyor.yml
##
@@ -46,9 +46,9 @@ install:
# Here, we use the fixed version of testthat. For more details, please see
SPARK-22817.
# As of devtools 2.1.0, it requires testthat higher then 2.1.1 as a
dependency. SparkR test requires testthat 1.0.2.
# Therefore, we don't use devtools but installs it directly from the archive
including its dependencies.
- - cmd: R -e "install.packages(c('crayon', 'praise', 'R6'),
repos='https://cloud.r-project.org/')"
+ - cmd: R -e "install.packages(c('crayon', 'praise', 'R6', 'arrow'),
repos='https://cloud.r-project.org/')"
- cmd: R -e
"install.packages('https://cloud.r-project.org/src/contrib/Archive/testthat/testthat_1.0.2.tar.gz',
repos=NULL, type='source')"
- - cmd: R -e "packageVersion('knitr'); packageVersion('rmarkdown');
packageVersion('testthat'); packageVersion('e1071'); packageVersion('survival')"
+ - cmd: R -e "packageVersion('knitr'); packageVersion('rmarkdown');
packageVersion('testthat'); packageVersion('e1071');
packageVersion('survival'); packageVersion('arrow')"
Review comment:
should we specify the arrow version to install? just wonder about
compatibility issue
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25996: [SPARK-29322][CORE] Enable closeFrameOnFlush on ZstdOutputStream for event log file
dongjoon-hyun closed pull request #25996: [SPARK-29322][CORE] Enable closeFrameOnFlush on ZstdOutputStream for event log file URL: https://github.com/apache/spark/pull/25996 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya edited a comment on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data
viirya edited a comment on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data URL: https://github.com/apache/spark/pull/25979#issuecomment-537771098 Currently behavior of INSERT OVERWRITE into a Hive external table partition is consistent with Hive. If the partition was dropped by DROP PARTITION before, INSERT OVERWRITE will not overwrite existing partition directory. If there are files in the directory, conflicting files will be overwritten (Hive 2.1) or exception thrown (Hive 2.3.2). Unique files will be read as duplicated data. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data
viirya commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data URL: https://github.com/apache/spark/pull/25979#issuecomment-537771098 Currently behavior of INSERT OVERWRITE into a Hive external table partition is consistent with Hive. If the partition was dropped by DROP PARTITION before, INSERT OVERWRITE will not overwrite existing partition directory. If there are files in the directory, conflicting files will be overwritten or exception thrown. Unique files will be read as duplicated data. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
AmplabJenkins removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537770683 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16705/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
AmplabJenkins removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537770677 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
AmplabJenkins commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537770677 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
AmplabJenkins commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537770683 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16705/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
SparkQA commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537770434 **[Test build #111712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111712/testReport)** for PR 26006 at commit [`67114a7`](https://github.com/apache/spark/commit/67114a70b327a74028b7aa47f9998392fce3b03a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data
cloud-fan commented on issue #25979: [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data URL: https://github.com/apache/spark/pull/25979#issuecomment-537769089 Do you mean the current behavior is consistent with some Hive versions? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #25600: [SPARK-11150][SQL] Dynamic Partition Pruning
maryannxue commented on a change in pull request #25600: [SPARK-11150][SQL] Dynamic Partition Pruning URL: https://github.com/apache/spark/pull/25600#discussion_r330851430 ## File path: sql/core/src/main/scala/org/apache/spark/sql/dynamicpruning/PlanDynamicPruningFilters.scala ## @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.dynamicpruning Review comment: I'll submit a follow-up to move the package. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25024: [SPARK-27296][SQL] User Defined Aggregators that do not ser/de on each input row
cloud-fan commented on issue #25024: [SPARK-27296][SQL] User Defined
Aggregators that do not ser/de on each input row
URL: https://github.com/apache/spark/pull/25024#issuecomment-537768473
What I want to see is a single UDAF API that works for all the use cases, if
it's possible. It's really confusing to end-users if there are a lof of UDAF
APIs in Spark and they don't know which one to use.
Since ser/de is inevitable (`InternalRow` <-> `Row`), it's always preferred
to do the ser/de only on partition boundaries. AFAIK other requirements are:
1. can operate on multiple columns
2. can be deterministic or not
3. can cast the input columns to desired types
4. can be registered as SQL function
5. can operate on Dataset
6. ... (please add more if I missed something)
I think `Aggregator[IN, BUF, OUT]` is good enough to specify what the
aggregating logic is. What we need to add is:
1. an API to specify the input columns
2. an API to specify the determinism
2. a mechanism to cast input columns
3. an API to register Aggregator as SQL function
My proposal is:
1. add a new method in `UDFRegistration`
```
def register[IN: TypeTag, BUF: TypeTag, OUT: TypeTag](
name: String, func: Aggregator[IN, BUF, OUT]): UserDefinedFunction
```
2. add a new implementation of `UserDefinedFunction`
```
class AggregatorAsFunction[IN: TypeTag, BUF: TypeTag, OUT: TypeTag](
aggregator: Aggregator[IN, BUF, OUT]) extends UserDefinedFunction {
def apply(exprs: Column*): Column = {
// create a special expression which is smilar to `ScalaUDAF`. It
projects the input row
// according to the given `exprs`, and convert the projected internal
row to `IN` type object
// via the encoder, and feed the `IN` type object to the `Aggregator`.
The encoder can cast
// the input columns to desired types, please see `ScalaReflection`.
}
// this simply sets the nondeterministic flag of `AggregatorAsFunction`,
which will be passed to
// the new expression metioned in `apply`
def asNondeterministic ...
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537768175 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537768178 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16704/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537768175 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537768178 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16704/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
SparkQA commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537767982 **[Test build #111711 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111711/testReport)** for PR 25993 at commit [`1586fee`](https://github.com/apache/spark/commit/1586fee7e818d6101b85f5d96cf0e75b72fadf97). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name
AmplabJenkins removed a comment on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name URL: https://github.com/apache/spark/pull/26008#issuecomment-537767050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16703/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name
AmplabJenkins removed a comment on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name URL: https://github.com/apache/spark/pull/26008#issuecomment-537767044 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name
AmplabJenkins commented on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name URL: https://github.com/apache/spark/pull/26008#issuecomment-537767044 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name
AmplabJenkins commented on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name URL: https://github.com/apache/spark/pull/26008#issuecomment-537767050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16703/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name
SparkQA commented on issue #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name URL: https://github.com/apache/spark/pull/26008#issuecomment-537766801 **[Test build #111710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111710/testReport)** for PR 26008 at commit [`3be1ce7`](https://github.com/apache/spark/commit/3be1ce7b83644575b84044857d958abbebb2677a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name
dongjoon-hyun opened a new pull request #26008: [MINOR][TESTS] Rename JsonBenchmark.scala to JSONBenchmark.scala to match class name URL: https://github.com/apache/spark/pull/26008 ### What changes were proposed in this pull request? This PR renames `JsonBenchmark.scala` to `JSONBenchmark.scala`. ### Why are the changes needed? Currently, the file name doesn't match with `object JSONBenchmark`. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? This is just renaming file. The existing object doesn't change at all. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
HyukjinKwon commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537764993 cc @felixcheung and @viirya This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537764735 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins removed a comment on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537764739 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16702/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537764739 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16702/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
AmplabJenkins commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537764735 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
SparkQA commented on issue #25993: [SPARK-29339][R] Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build) URL: https://github.com/apache/spark/pull/25993#issuecomment-537764496 **[Test build #111709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111709/testReport)** for PR 25993 at commit [`1e2440c`](https://github.com/apache/spark/commit/1e2440cda26529af493a6f2336f4617a39676748). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25670: [SPARK-28869][CORE] Roll over event log files
HeartSaVioR commented on a change in pull request #25670: [SPARK-28869][CORE] Roll over event log files URL: https://github.com/apache/spark/pull/25670#discussion_r330847692 ## File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ## @@ -1165,12 +1161,15 @@ private[history] case class LogInfo( logType: LogType.Value, appId: Option[String], attemptId: Option[String], -fileSize: Long) +fileSize: Long, +lastIndex: Option[Long], Review comment: I missed to check this. Will check and fix if it causes problem via adding UT which loads LevelDB from Spark 2.4.4 and read more applications. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25993: [DO-NOT-MERGE][R] Install Arrow and test Arrow optimization in AppVeyor build
HyukjinKwon commented on a change in pull request #25993: [DO-NOT-MERGE][R]
Install Arrow and test Arrow optimization in AppVeyor build
URL: https://github.com/apache/spark/pull/25993#discussion_r330847032
##
File path: R/pkg/inst/worker/worker.R
##
@@ -50,7 +50,7 @@ compute <- function(mode, partition, serializer,
deserializer, key,
} else {
# Check to see if inputData is a valid data.frame
stopifnot(deserializer == "byte" || deserializer == "arrow")
- stopifnot(class(inputData) == "data.frame")
+ stopifnot(is.data.frame(inputData))
Review comment:
In Arrow 0.14, seems `as.data.frame(arrow_batch)` returns non-naive R
DataFrame ("tbl_df tbl data.frame"). So, direct class comparison doesn't look
working here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] nvander1 commented on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object
nvander1 commented on issue #26007: [SPARK-28962][SPARK-27297][SQL] Add overload for filter with index to functions object URL: https://github.com/apache/spark/pull/26007#issuecomment-537762211 Sure thing! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
AmplabJenkins removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537759168 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111705/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
AmplabJenkins commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537759168 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111705/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
AmplabJenkins commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537759165 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
AmplabJenkins removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537759165 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
SparkQA removed a comment on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537727823 **[Test build #111705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111705/testReport)** for PR 26006 at commit [`db8968a`](https://github.com/apache/spark/commit/db8968a2a3a9b88dc2d22fbf13fba074c6372681). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
SparkQA commented on issue #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path URL: https://github.com/apache/spark/pull/26006#issuecomment-537759048 **[Test build #111705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111705/testReport)** for PR 26006 at commit [`db8968a`](https://github.com/apache/spark/commit/db8968a2a3a9b88dc2d22fbf13fba074c6372681). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
AmplabJenkins commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537755617 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16701/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
AmplabJenkins removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537755617 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16701/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
AmplabJenkins removed a comment on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537755615 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
AmplabJenkins commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537755615 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan
SparkQA commented on issue #26001: [SPARK-29331][SQL] create DS v2 Write at physical plan URL: https://github.com/apache/spark/pull/26001#issuecomment-537755364 **[Test build #111708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111708/testReport)** for PR 26001 at commit [`79f94f5`](https://github.com/apache/spark/commit/79f94f59c1298c021f71b65acfdb850161075545). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
HyukjinKwon commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-537754335 +1 ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26004: [SPARK-29142][PYTHON][ML][FOLLOWUP][DOC] Replace incorrect :py:attr: applications
maropu commented on issue #26004: [SPARK-29142][PYTHON][ML][FOLLOWUP][DOC] Replace incorrect :py:attr: applications URL: https://github.com/apache/spark/pull/26004#issuecomment-537748479 cc: @HyukjinKwon @ueshin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2
AmplabJenkins removed a comment on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2 URL: https://github.com/apache/spark/pull/26005#issuecomment-537747396 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2
AmplabJenkins removed a comment on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2 URL: https://github.com/apache/spark/pull/26005#issuecomment-537747405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111704/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2
AmplabJenkins commented on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2 URL: https://github.com/apache/spark/pull/26005#issuecomment-537747405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111704/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2
AmplabJenkins commented on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2 URL: https://github.com/apache/spark/pull/26005#issuecomment-537747396 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2
SparkQA removed a comment on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2 URL: https://github.com/apache/spark/pull/26005#issuecomment-537697694 **[Test build #111704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111704/testReport)** for PR 26005 at commit [`a04be5f`](https://github.com/apache/spark/commit/a04be5f46fe160a5dc3ce9db4c9492cc4ec222df). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2
SparkQA commented on issue #26005: [SPARK-29163] Simplify Hadoop Configuration access in DataSourcev2 URL: https://github.com/apache/spark/pull/26005#issuecomment-537746974 **[Test build #111704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111704/testReport)** for PR 26005 at commit [`a04be5f`](https://github.com/apache/spark/commit/a04be5f46fe160a5dc3ce9db4c9492cc4ec222df). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution
maropu commented on a change in pull request #25960: [SPARK-29283][SQL] Error
message is hidden when query from JDBC, especially enabled adaptive execution
URL: https://github.com/apache/spark/pull/25960#discussion_r330831160
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetCatalogsOperation.scala
##
@@ -68,11 +69,18 @@ private[hive] class SparkGetCatalogsOperation(
}
setState(OperationState.FINISHED)
} catch {
- case e: HiveSQLException =>
+ case e: Throwable =>
Review comment:
If so, we should list up `InterruptedExpression` here, too? IIUC the reason
why we mainly use `NonFatal` in this case is not to catch
`NonLocalReturnControl`. But, yea, this is not my area, so I think @wangyum
could suggest more about this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25988: [SPARK-29313][SQL] Fix failure on writing to `noop` in benchmarks
cloud-fan commented on issue #25988: [SPARK-29313][SQL] Fix failure on writing to `noop` in benchmarks URL: https://github.com/apache/spark/pull/25988#issuecomment-537742759 we only disable file source v2, but noop souce is not file source. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] YongjinZhou commented on issue #25976: [SPARK-25468][WEBUI] Current page index keep style with dataTable in the spark UI
YongjinZhou commented on issue #25976: [SPARK-25468][WEBUI] Current page index keep style with dataTable in the spark UI URL: https://github.com/apache/spark/pull/25976#issuecomment-537742435 @srowen For match whth jquery.dataTables.1.10.18.min.css 。Selected page should be '.current',but result is '.active'.The reason is the order of dataTables.bootstrap.min.js and jquery.dataTables.1.10.18.min.css. IF we exchange the order, well be other css probrom. So, keep stype in this way. thanks.  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #26006: [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path
imback82 commented on a change in pull request #26006: [SPARK-29279][SQL] Merge
SHOW NAMESPACES and SHOW DATABASES code path
URL: https://github.com/apache/spark/pull/26006#discussion_r330829071
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala
##
@@ -170,22 +170,11 @@ case class DataSourceResolution(
DeleteFromTable(aliased, delete.condition)
case ShowNamespacesStatement(None, pattern) =>
- defaultCatalog match {
-case Some(catalog) =>
- ShowNamespaces(catalog.asNamespaceCatalog, None, pattern)
-case None =>
- throw new AnalysisException("No default v2 catalog is set.")
- }
+ ShowNamespaces(currentCatalog.asNamespaceCatalog, None, pattern)
Review comment:
@cloud-fan I can wait for https://github.com/apache/spark/pull/25747 if you
plan to do the current catalog changes in that PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26007: Add overload for filter with index to functions object
AmplabJenkins removed a comment on issue #26007: Add overload for filter with index to functions object URL: https://github.com/apache/spark/pull/26007#issuecomment-537741381 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
