[GitHub] [spark] AmplabJenkins removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
AmplabJenkins removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509089279 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107324/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
AmplabJenkins removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509089275 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
SparkQA removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509068276 **[Test build #107324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107324/testReport)** for PR 25069 at commit [`da08011`](https://github.com/apache/spark/commit/da080111d55ecd9fe8b14fdd8ffa0ae78e7ff4e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
AmplabJenkins commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509089279 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107324/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
SparkQA commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509089142 **[Test build #107324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107324/testReport)** for PR 25069 at commit [`da08011`](https://github.com/apache/spark/commit/da080111d55ecd9fe8b14fdd8ffa0ae78e7ff4e9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
AmplabJenkins commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509089275 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509087022 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509087027 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107321/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509087027 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107321/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509087022 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
SparkQA removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509059124 **[Test build #107321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107321/testReport)** for PR 24637 at commit [`86e795f`](https://github.com/apache/spark/commit/86e795fb3eb5844aae8e881fb8446bee8239bdbe). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509086695 **[Test build #107321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107321/testReport)** for PR 24637 at commit [`86e795f`](https://github.com/apache/spark/commit/86e795fb3eb5844aae8e881fb8446bee8239bdbe). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085905 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085909 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107320/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085909 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107320/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085905 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
SparkQA removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509059126 **[Test build #107320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107320/testReport)** for PR 24993 at commit [`3520138`](https://github.com/apache/spark/commit/352013884694e9ba7d4fbf2c0c1ba259c2fb3174). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085490 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
SparkQA commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085564 **[Test build #107320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107320/testReport)** for PR 24993 at commit [`3520138`](https://github.com/apache/spark/commit/352013884694e9ba7d4fbf2c0c1ba259c2fb3174). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085493 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107319/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085490 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085493 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107319/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
SparkQA removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509057956 **[Test build #107319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107319/testReport)** for PR 24993 at commit [`1fbe526`](https://github.com/apache/spark/commit/1fbe526d0e99a9eca9d83cb5c1baa2403a432440). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
SparkQA commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509085133 **[Test build #107319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107319/testReport)** for PR 24993 at commit [`1fbe526`](https://github.com/apache/spark/commit/1fbe526d0e99a9eca9d83cb5c1baa2403a432440). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base
AmplabJenkins removed a comment on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base URL: https://github.com/apache/spark/pull/25070#issuecomment-509078941 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12467/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base
AmplabJenkins removed a comment on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base URL: https://github.com/apache/spark/pull/25070#issuecomment-509078937 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base
SparkQA commented on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base URL: https://github.com/apache/spark/pull/25070#issuecomment-509079228 **[Test build #107325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107325/testReport)** for PR 25070 at commit [`cd70381`](https://github.com/apache/spark/commit/cd7038156ad48a34063182acb1cc499eae9d8399). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base
AmplabJenkins commented on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base URL: https://github.com/apache/spark/pull/25070#issuecomment-509078937 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base
AmplabJenkins commented on issue #25070: [SPARK-28273][SQL][PYTHON] Convert and port 'pgSQL/case.sql' into UDF test base URL: https://github.com/apache/spark/pull/25070#issuecomment-509078941 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12467/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #25070: Convert and port 'pgSQL/case.sql' into UDF test base
HyukjinKwon opened a new pull request #25070: Convert and port 'pgSQL/case.sql'
into UDF test base
URL: https://github.com/apache/spark/pull/25070
## What changes were proposed in this pull request?
This PR adds some tests converted from `pgSQL/case.sql'` to test UDFs.
Please see contribution guide of this umbrella ticket -
[SPARK-27921](https://issues.apache.org/jira/browse/SPARK-27921).
This PR also contains two minor fixes:
1. Change name of Scala UDF from `UDF:name(...)` to `name(...)` to be
consistent with Python'
2. Fix Scala UDF at `IntegratedUDFTestUtils.scala ` to handle `null` in
strings.
Diff comparing to 'pgSQL/case.sql'
```diff
diff --git
a/sql/core/src/test/resources/sql-tests/results/pgSQL/case.sql.out
b/sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-case.sql.out
index fa078d16d6d..55bef64338f 100644
--- a/sql/core/src/test/resources/sql-tests/results/pgSQL/case.sql.out
+++
b/sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-case.sql.out
@@ -115,7 +115,7 @@ struct<>
-- !query 13
SELECT '3' AS `One`,
CASE
-WHEN 1 < 2 THEN 3
+WHEN CAST(udf(1 < 2) AS boolean) THEN 3
END AS `Simple WHEN`
-- !query 13 schema
struct
@@ -126,10 +126,10 @@ struct
-- !query 14
SELECT '' AS `One`,
CASE
-WHEN 1 > 2 THEN 3
+WHEN 1 > 2 THEN udf(3)
END AS `Simple default`
-- !query 14 schema
-struct
+struct
-- !query 14 output
NULL
@@ -137,17 +137,17 @@ struct
-- !query 15
SELECT '3' AS `One`,
CASE
-WHEN 1 < 2 THEN 3
-ELSE 4
+WHEN udf(1) < 2 THEN udf(3)
+ELSE udf(4)
END AS `Simple ELSE`
-- !query 15 schema
-struct
+struct
-- !query 15 output
3 3
-- !query 16
-SELECT '4' AS `One`,
+SELECT udf('4') AS `One`,
CASE
WHEN 1 > 2 THEN 3
ELSE 4
@@ -159,10 +159,10 @@ struct
-- !query 17
-SELECT '6' AS `One`,
+SELECT udf('6') AS `One`,
CASE
-WHEN 1 > 2 THEN 3
-WHEN 4 < 5 THEN 6
+WHEN CAST(udf(1 > 2) AS boolean) THEN 3
+WHEN udf(4) < 5 THEN 6
ELSE 7
END AS `Two WHEN with default`
-- !query 17 schema
@@ -173,7 +173,7 @@ struct
-- !query 18
SELECT '7' AS `None`,
- CASE WHEN rand() < 0 THEN 1
+ CASE WHEN rand() < udf(0) THEN 1
END AS `NULL on no matches`
-- !query 18 schema
struct
@@ -182,36 +182,36 @@ struct
-- !query 19
-SELECT CASE WHEN 1=0 THEN 1/0 WHEN 1=1 THEN 1 ELSE 2/0 END
+SELECT CASE WHEN CAST(udf(1=0) AS boolean) THEN 1/0 WHEN 1=1 THEN 1 ELSE
2/0 END
-- !query 19 schema
-struct
+struct
-- !query 19 output
1.0
-- !query 20
-SELECT CASE 1 WHEN 0 THEN 1/0 WHEN 1 THEN 1 ELSE 2/0 END
+SELECT CASE 1 WHEN 0 THEN 1/udf(0) WHEN 1 THEN 1 ELSE 2/0 END
-- !query 20 schema
-struct
+struct
-- !query 20 output
1.0
-- !query 21
-SELECT CASE WHEN i > 100 THEN 1/0 ELSE 0 END FROM case_tbl
+SELECT CASE WHEN i > 100 THEN udf(1/0) ELSE udf(0) END FROM case_tbl
-- !query 21 schema
-struct 100) THEN (CAST(1 AS DOUBLE) / CAST(0 AS DOUBLE))
ELSE CAST(0 AS DOUBLE) END:double>
+struct 100) THEN udf((cast(1 as double) / cast(0 as
double))) ELSE udf(0) END:string>
-- !query 21 output
-0.0
-0.0
-0.0
-0.0
+0
+0
+0
+0
-- !query 22
-SELECT CASE 'a' WHEN 'a' THEN 1 ELSE 2 END
+SELECT CASE 'a' WHEN 'a' THEN udf(1) ELSE udf(2) END
-- !query 22 schema
-struct
+struct
-- !query 22 output
1
@@ -283,7 +283,7 @@ big
-- !query 27
-SELECT * FROM CASE_TBL WHERE COALESCE(f,i) = 4
+SELECT * FROM CASE_TBL WHERE udf(COALESCE(f,i)) = 4
-- !query 27 schema
struct
-- !query 27 output
@@ -291,7 +291,7 @@ struct
-- !query 28
-SELECT * FROM CASE_TBL WHERE NULLIF(f,i) = 2
+SELECT * FROM CASE_TBL WHERE udf(NULLIF(f,i)) = 2
-- !query 28 schema
struct
-- !query 28 output
@@ -299,10 +299,10 @@ struct
-- !query 29
-SELECT COALESCE(a.f, b.i, b.j)
+SELECT udf(COALESCE(a.f, b.i, b.j))
FROM CASE_TBL a, CASE2_TBL b
-- !query 29 schema
-struct
+struct
-- !query 29 output
-30.3
-30.3
@@ -332,8 +332,8 @@ struct
-- !query 30
SELECT *
- FROM CASE_TBL a, CASE2_TBL b
- WHERE COALESCE(a.f, b.i, b.j) = 2
+ FROM CASE_TBL a, CASE2_TBL b
+ WHERE udf(COALESCE(a.f, b.i, b.j)) = 2
-- !query 30 schema
struct
-- !query 30 output
@@ -342,7 +342,7 @@ struct
-- !query 31
-SELECT '' AS Five, NULLIF(a.i,b.i) AS `NULLIF(a.i,b.i)`,
+SELECT udf('') AS Five, NULLIF(a.i,b.i) AS `NULLIF(a.i,b.i)`,
NULLIF(b.i, 4) AS `NULLIF(b.i,4)`
FROM CASE_TBL a, CASE2_TBL b
-- !query 31 schema
@@ -377,7 +377,7 @@ struct
-- !query 32
SELECT
[GitHub] [spark] henrydavidge commented on issue #24953: [SPARK-28140][MLLIB][PYTHON] Accept DataFrames in RowMatrix and IndexedRowMatrix constructors
henrydavidge commented on issue #24953: [SPARK-28140][MLLIB][PYTHON] Accept DataFrames in RowMatrix and IndexedRowMatrix constructors URL: https://github.com/apache/spark/pull/24953#issuecomment-509074840 Thanks for the initial look @srowen. I fixed the accidental import changes. @jkbradley Looks like the incantation to enable tests didn't work This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] henrydavidge commented on a change in pull request #24953: [SPARK-28140][MLLIB][PYTHON] Accept DataFrames in RowMatrix and IndexedRowMatrix constructors
henrydavidge commented on a change in pull request #24953:
[SPARK-28140][MLLIB][PYTHON] Accept DataFrames in RowMatrix and
IndexedRowMatrix constructors
URL: https://github.com/apache/spark/pull/24953#discussion_r300918271
##
File path:
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
##
@@ -21,16 +21,15 @@ import java.util.Arrays
import scala.collection.mutable.ListBuffer
-import breeze.linalg.{axpy => brzAxpy, inv, svd => brzSvd, DenseMatrix => BDM,
DenseVector => BDV,
- MatrixSingularException, SparseVector => BSV}
+import breeze.linalg.{MatrixSingularException, inv, DenseMatrix => BDM,
DenseVector => BDV, SparseVector => BSV, axpy => brzAxpy, svd => brzSvd}
import breeze.numerics.{sqrt => brzSqrt}
-
import org.apache.spark.annotation.Since
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config.MAX_RESULT_SIZE
import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.stat.{MultivariateOnlineSummarizer,
MultivariateStatisticalSummary}
import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{Dataset, Row}
Review comment:
臘♂ oops
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory
AmplabJenkins commented on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory URL: https://github.com/apache/spark/pull/25067#issuecomment-509072607 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory
AmplabJenkins removed a comment on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory URL: https://github.com/apache/spark/pull/25067#issuecomment-509072607 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory
AmplabJenkins removed a comment on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory URL: https://github.com/apache/spark/pull/25067#issuecomment-509072612 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107316/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory
AmplabJenkins commented on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory URL: https://github.com/apache/spark/pull/25067#issuecomment-509072612 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107316/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory
SparkQA removed a comment on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory URL: https://github.com/apache/spark/pull/25067#issuecomment-509053334 **[Test build #107316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107316/testReport)** for PR 25067 at commit [`895af34`](https://github.com/apache/spark/commit/895af34271485c173a81768cb33edd26909709d6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory
SparkQA commented on issue #25067: [SPARK-28290][CORE][SQL][test-hadoop3.2][test-maven] Use SslContextFactory.Server instead of SslContextFactory URL: https://github.com/apache/spark/pull/25067#issuecomment-509072298 **[Test build #107316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107316/testReport)** for PR 25067 at commit [`895af34`](https://github.com/apache/spark/commit/895af34271485c173a81768cb33edd26909709d6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Deegue closed pull request #25057: [SPARK-28239][SHUFFLE] Allow TCP connections created by shuffle service auto close on YARN NodeManagers
Deegue closed pull request #25057: [SPARK-28239][SHUFFLE] Allow TCP connections created by shuffle service auto close on YARN NodeManagers URL: https://github.com/apache/spark/pull/25057 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Deegue commented on issue #25057: [SPARK-28239][SHUFFLE] Allow TCP connections created by shuffle service auto close on YARN NodeManagers
Deegue commented on issue #25057: [SPARK-28239][SHUFFLE] Allow TCP connections created by shuffle service auto close on YARN NodeManagers URL: https://github.com/apache/spark/pull/25057#issuecomment-509070042 I omitted some other changes to this problem. I will check the code and open a new one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
AmplabJenkins removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509069017 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
AmplabJenkins removed a comment on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509069019 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12466/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
AmplabJenkins commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509069017 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
AmplabJenkins commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509069019 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12466/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
SparkQA commented on issue #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069#issuecomment-509068276 **[Test build #107324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107324/testReport)** for PR 25069 at commit [`da08011`](https://github.com/apache/spark/commit/da080111d55ecd9fe8b14fdd8ffa0ae78e7ff4e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base
HyukjinKwon opened a new pull request #25069: [SPARK-28270][SQL][PYTHON] Convert and port 'pgSQL/aggregates_part1.sql' into UDF test base URL: https://github.com/apache/spark/pull/25069 ## What changes were proposed in this pull request? This PR adds some tests in general. Please see contribution guide of this umbrella ticket - [SPARK-27921](https://issues.apache.org/jira/browse/SPARK-27921). This PR contains two minor fixes: 1. Change name of Scala UDF from `UDF:name(...)` to `name(...)` to be consistent with Python' 2. Fix Scala UDF at `IntegratedUDFTestUtils.scala ` to handle `null` in strings. Diff comparing to 'pgSQL/aggregates_part1.sql' ```diff diff --git a/sql/core/src/test/resources/sql-tests/results/pgSQL/aggregates_part1.sql.out b/sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-aggregates_part1.sql.out index 51ca1d55869..124fdd6416e 100644 --- a/sql/core/src/test/resources/sql-tests/results/pgSQL/aggregates_part1.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-aggregates_part1.sql.out @@ -3,7 +3,7 @@ -- !query 0 -SELECT avg(four) AS avg_1 FROM onek +SELECT avg(udf(four)) AS avg_1 FROM onek -- !query 0 schema struct -- !query 0 output @@ -11,15 +11,15 @@ struct -- !query 1 -SELECT avg(a) AS avg_32 FROM aggtest WHERE a < 100 +SELECT udf(avg(a)) AS avg_32 FROM aggtest WHERE a < 100 -- !query 1 schema -struct +struct -- !query 1 output 32.664 -- !query 2 -select CAST(avg(b) AS Decimal(10,3)) AS avg_107_943 FROM aggtest +select CAST(avg(udf(b)) AS Decimal(10,3)) AS avg_107_943 FROM aggtest -- !query 2 schema struct -- !query 2 output @@ -27,285 +27,286 @@ struct -- !query 3 -SELECT sum(four) AS sum_1500 FROM onek +SELECT sum(udf(four)) AS sum_1500 FROM onek -- !query 3 schema -struct +struct -- !query 3 output -1500 +1500.0 -- !query 4 -SELECT sum(a) AS sum_198 FROM aggtest +SELECT udf(sum(a)) AS sum_198 FROM aggtest -- !query 4 schema -struct +struct -- !query 4 output 198 -- !query 5 -SELECT sum(b) AS avg_431_773 FROM aggtest +SELECT udf(udf(sum(b))) AS avg_431_773 FROM aggtest -- !query 5 schema -struct +struct -- !query 5 output 431.77260909229517 -- !query 6 -SELECT max(four) AS max_3 FROM onek +SELECT udf(max(four)) AS max_3 FROM onek -- !query 6 schema -struct +struct -- !query 6 output 3 -- !query 7 -SELECT max(a) AS max_100 FROM aggtest +SELECT max(udf(a)) AS max_100 FROM aggtest -- !query 7 schema -struct +struct -- !query 7 output -100 +56 -- !query 8 -SELECT max(aggtest.b) AS max_324_78 FROM aggtest +SELECT CAST(udf(udf(max(aggtest.b))) AS int) AS max_324_78 FROM aggtest -- !query 8 schema -struct +struct -- !query 8 output -324.78 +324 -- !query 9 -SELECT stddev_pop(b) FROM aggtest +SELECT CAST(stddev_pop(udf(b)) AS int) FROM aggtest -- !query 9 schema -struct +struct -- !query 9 output -131.10703231895047 +131 -- !query 10 -SELECT stddev_samp(b) FROM aggtest +SELECT udf(stddev_samp(b)) FROM aggtest -- !query 10 schema -struct +struct -- !query 10 output 151.38936080399804 -- !query 11 -SELECT var_pop(b) FROM aggtest +SELECT CAST(var_pop(udf(b)) as int) FROM aggtest -- !query 11 schema -struct +struct -- !query 11 output -17189.053923482323 +17189 -- !query 12 -SELECT var_samp(b) FROM aggtest +SELECT udf(var_samp(b)) FROM aggtest -- !query 12 schema -struct +struct -- !query 12 output 22918.738564643096 -- !query 13 -SELECT stddev_pop(CAST(b AS Decimal(38,0))) FROM aggtest +SELECT udf(stddev_pop(CAST(b AS Decimal(38,0 FROM aggtest -- !query 13 schema -struct +struct -- !query 13 output 131.18117242958306 -- !query 14 -SELECT stddev_samp(CAST(b AS Decimal(38,0))) FROM aggtest +SELECT stddev_samp(CAST(udf(b) AS Decimal(38,0))) FROM aggtest -- !query 14 schema -struct +struct -- !query 14 output 151.47497042966097 -- !query 15 -SELECT var_pop(CAST(b AS Decimal(38,0))) FROM aggtest +SELECT udf(var_pop(CAST(b AS Decimal(38,0 FROM aggtest -- !query 15 schema -struct +struct -- !query 15 output 17208.5 -- !query 16 -SELECT var_samp(CAST(b AS Decimal(38,0))) FROM aggtest +SELECT var_samp(udf(CAST(b AS Decimal(38,0 FROM aggtest -- !query 16 schema -struct +struct -- !query 16 output 22944.6668 -- !query 17 -SELECT var_pop(1.0), var_samp(2.0) +SELECT udf(var_pop(1.0)), var_samp(udf(2.0)) -- !query 17 schema -struct
[GitHub] [spark] AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509066794 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107323/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509066788 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509066788 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
SparkQA commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509066722 **[Test build #107323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107323/testReport)** for PR 25063 at commit [`8bdb147`](https://github.com/apache/spark/commit/8bdb1476fa06ee467467a9f79c80e3805fb2bff2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509066794 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107323/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
SparkQA removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509064870 **[Test build #107323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107323/testReport)** for PR 25063 at commit [`8bdb147`](https://github.com/apache/spark/commit/8bdb1476fa06ee467467a9f79c80e3805fb2bff2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509065746 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12465/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509065743 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509065746 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12465/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2)
AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(support build with hadoop-3.2) URL: https://github.com/apache/spark/pull/25063#issuecomment-509065743 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gczsjdy commented on a change in pull request #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
gczsjdy commented on a change in pull request #25007:
[SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
URL: https://github.com/apache/spark/pull/25007#discussion_r300910745
##
File path:
core/src/main/java/org/apache/spark/api/shuffle/ShuffleWriteSupport.java
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.api.shuffle;
+
+import java.io.IOException;
+
+import org.apache.spark.annotation.Experimental;
+
+/**
+ * :: Experimental ::
+ * An interface for deploying a shuffle map output writer
+ *
+ * @since 3.0.0
+ */
+@Experimental
+public interface ShuffleWriteSupport {
Review comment:
Found another reason to remove this `Write/ReadSupport` layer, in
`ReadSupport`, although not proposed yet : P, it straightway contains partition
level functions:
https://github.com/palantir/spark/blob/62c2664f1f298889357c6ebeb9b6f08962c94ceb/core/src/main/java/org/apache/spark/api/shuffle/ShuffleReadSupport.java#L31-L38.
However for `WriteSupport` path this is what 1-level-lower layer would
do(partition-level writer).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3)
SparkQA commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3) URL: https://github.com/apache/spark/pull/25063#issuecomment-509064870 **[Test build #107323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107323/testReport)** for PR 25063 at commit [`8bdb147`](https://github.com/apache/spark/commit/8bdb1476fa06ee467467a9f79c80e3805fb2bff2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24996: [SPARK-28199][SS] Replace deprecated ProcessingTime with ProcessingTimeTrigger and make invisible to end users
srowen commented on a change in pull request #24996: [SPARK-28199][SS] Replace
deprecated ProcessingTime with ProcessingTimeTrigger and make invisible to end
users
URL: https://github.com/apache/spark/pull/24996#discussion_r300909196
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Triggers.scala
##
@@ -27,3 +32,34 @@ import org.apache.spark.sql.streaming.Trigger
@Experimental
@Evolving
case object OneTimeTrigger extends Trigger
+
+/**
+ * A [[Trigger]] that runs a query periodically based on the processing time.
If `interval` is 0,
+ * the query will run as fast as possible.
+ */
+@Evolving
+private[sql] case class ProcessingTimeTrigger(intervalMs: Long) extends
Trigger {
Review comment:
@dongjoon-hyun I don't particularly care where the implementations live, I'm
just looking for consistency. It's not terribly consistent to begin with. I
think putting the private implementations in `Triggers.scala` is a step in the
right direction, given that this file exists at all. I think this change is OK
as-is. I don't think we have to move other implementations, though, anything
that improves consistency is OK.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
AmplabJenkins removed a comment on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-509063297 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
AmplabJenkins commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-509063407 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
AmplabJenkins commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-509063297 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gczsjdy commented on a change in pull request #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
gczsjdy commented on a change in pull request #25007:
[SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
URL: https://github.com/apache/spark/pull/25007#discussion_r300908424
##
File path:
core/src/main/java/org/apache/spark/api/shuffle/ShuffleWriteSupport.java
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.api.shuffle;
+
+import java.io.IOException;
+
+import org.apache.spark.annotation.Experimental;
+
+/**
+ * :: Experimental ::
+ * An interface for deploying a shuffle map output writer
+ *
+ * @since 3.0.0
+ */
+@Experimental
+public interface ShuffleWriteSupport {
Review comment:
Since `ShuffleWriteSupport` only contains 1 function, why do we add this
layer(Also, `xxSupport` is ambiguous)?
Could we make `ShuffleExecutorComponents.writes` return a
`ShuffleMapOutputWriter` directly?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gczsjdy commented on a change in pull request #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
gczsjdy commented on a change in pull request #25007:
[SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
URL: https://github.com/apache/spark/pull/25007#discussion_r300908424
##
File path:
core/src/main/java/org/apache/spark/api/shuffle/ShuffleWriteSupport.java
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.api.shuffle;
+
+import java.io.IOException;
+
+import org.apache.spark.annotation.Experimental;
+
+/**
+ * :: Experimental ::
+ * An interface for deploying a shuffle map output writer
+ *
+ * @since 3.0.0
+ */
+@Experimental
+public interface ShuffleWriteSupport {
Review comment:
Since `ShuffleWrite(/Read)Support` only contains 1 function, why do we add
this layer(Also, `xxSupport` is ambiguous)?
Could we make `ShuffleExecutorComponents.writes` return a
`ShuffleMapOutputWriter` directly?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
SparkQA commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-509062561 **[Test build #107322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107322/testReport)** for PR 25068 at commit [`f2645aa`](https://github.com/apache/spark/commit/f2645aa28e99577dfed9645337cce17376cf53a5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bersprockets opened a new pull request #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
bersprockets opened a new pull request #25068: [SPARK-28156][SQL][BACKPORT-2.4]
Self-join should not miss cached view
URL: https://github.com/apache/spark/pull/25068
Back-port of #24960 to branch-2.4.
The issue is when self-join a cached view, only one side of join uses cached
relation. The cause is in `ResolveReferences` we do deduplicate for a view to
have new output attributes. Then in `AliasViewChild`, the rule adds extra
project under a view. So it breaks cache matching.
The fix is when dedup, we only dedup a view which has output different to
its child plan. Otherwise, we dedup on the view's child plan.
```scala
val df = Seq.tabulate(5) { x => (x, x + 1, x + 2, x + 3) }.toDF("a", "b",
"c", "d")
df.write.mode("overwrite").format("orc").saveAsTable("table1")
sql("drop view if exists table1_vw")
sql("create view table1_vw as select * from table1")
val cachedView = sql("select a, b, c, d from table1_vw")
cachedView.createOrReplaceTempView("cachedview")
cachedView.persist()
val queryDf = sql(
s"""select leftside.a, leftside.b
|from cachedview leftside
|join cachedview rightside
|on leftside.a = rightside.a
""".stripMargin)
```
Query plan before this PR:
```scala
== Physical Plan ==
*(2) Project [a#12664, b#12665]
+- *(2) BroadcastHashJoin [a#12664], [a#12660], Inner, BuildRight
:- *(2) Filter isnotnull(a#12664)
: +- *(2) InMemoryTableScan [a#12664, b#12665], [isnotnull(a#12664)]
:+- InMemoryRelation [a#12664, b#12665, c#12666, d#12667],
StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) FileScan orc
default.table1[a#12660,b#12661,c#12662,d#12663] Batched: true, DataFilters: [],
Format: ORC, Location: InMemoryF
ileIndex[file:/Users/viirya/repos/spark-1/sql/core/spark-warehouse/org.apache.spark.sql,
PartitionFilters: [], PushedFilters: [], ReadSchema: struc
t
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int,
true] as bigint)))
+- *(1) Project [a#12660]
+- *(1) Filter isnotnull(a#12660)
+- *(1) FileScan orc default.table1[a#12660] Batched: true,
DataFilters: [isnotnull(a#12660)], Format: ORC, Location: InMemoryFileIndex[fil
e:/Users/viirya/repos/spark-1/sql/core/spark-warehouse/org.apache.spark.sql,
PartitionFilters: [], PushedFilters: [IsNotNull(a)], ReadSchema: struc
t
```
Query plan after this PR:
```scala
== Physical Plan ==
*(2) Project [a#12664, b#12665]
+- *(2) BroadcastHashJoin [a#12664], [a#12692], Inner, BuildRight
:- *(2) Filter isnotnull(a#12664)
: +- *(2) InMemoryTableScan [a#12664, b#12665], [isnotnull(a#12664)]
:+- InMemoryRelation [a#12664, b#12665, c#12666, d#12667],
StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) FileScan orc
default.table1[a#12660,b#12661,c#12662,d#12663] Batched: true, DataFilters: [],
Format: ORC, Location:
InMemoryFileIndex[file:/Users/viirya/repos/spark-1/sql/core/spark-warehouse/org.apache.spark.sql,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int,
false] as bigint)))
+- *(1) Filter isnotnull(a#12692)
+- *(1) InMemoryTableScan [a#12692], [isnotnull(a#12692)]
+- InMemoryRelation [a#12692, b#12693, c#12694, d#12695],
StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) FileScan orc
default.table1[a#12660,b#12661,c#12662,d#12663] Batched: true, DataFilters: [],
Format: ORC, Location:
InMemoryFileIndex[file:/Users/viirya/repos/spark-1/sql/core/spark-warehouse/org.apache.spark.sql,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct
```
Added test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #25061: [SPARK-28262][SQL] Support `DROP GLOBAL TEMPORARY VIEW`
wangyum commented on issue #25061: [SPARK-28262][SQL] Support `DROP GLOBAL TEMPORARY VIEW` URL: https://github.com/apache/spark/pull/25061#issuecomment-509062011 Teradata support `DROP TEMPORARY TABLE`. https://docs.teradata.com/reader/_a1eqWqCcebCUBcUpkaVvA/zoq2K_VQpej_91CX51Lkbg https://github.com/apache/spark/blob/7b7f16f2a7a6a6685a8917a9b5ba403fff76/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala#L424-L428 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3)
dongjoon-hyun commented on a change in pull request #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3) URL: https://github.com/apache/spark/pull/25063#discussion_r300906838 ## File path: docs/building-spark.md ## @@ -70,6 +70,8 @@ This will build Spark distribution along with Python pip and R packages. For mor ## Specifying the Hadoop Version and Enabling YARN +You can enable the exact profile of Hadoop to compile against through `-Phadoop-2.7`(default) and `-Phadoop-3.2`. + Review comment: Here, `exact profile` might look strange because we also have `hadoop-provided` profile. What about just omitting this line 73~74? The following example (82~84) looks clear enough to me (although it's an implicit way.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3)
dongjoon-hyun commented on a change in pull request #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3) URL: https://github.com/apache/spark/pull/25063#discussion_r300906890 ## File path: docs/building-spark.md ## @@ -79,15 +81,20 @@ Example: ./build/mvn -Pyarn -Dhadoop.version=2.8.5 -DskipTests clean package +./build/mvn -Pyarn -Phadoop-3.2 -Dhadoop.version=3.1.2 -DskipTests clean package Review comment: Also, could you update the PR title because this PR includes this `hadoop` example update additionally? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3)
dongjoon-hyun commented on a change in pull request #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3) URL: https://github.com/apache/spark/pull/25063#discussion_r300906838 ## File path: docs/building-spark.md ## @@ -70,6 +70,8 @@ This will build Spark distribution along with Python pip and R packages. For mor ## Specifying the Hadoop Version and Enabling YARN +You can enable the exact profile of Hadoop to compile against through `-Phadoop-2.7`(default) and `-Phadoop-3.2`. + Review comment: Here, `exact profile` might look strange because we also have `hadoop-provided` profile. What about just omitting this line 73~74? The following example (82~84) looks clear enough to me (although it's an implicit way.) Also, could you update the PR title because this PR includes `hadoop` example update additionally? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns
AmplabJenkins removed a comment on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns URL: https://github.com/apache/spark/pull/22595#issuecomment-509059822 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns
AmplabJenkins commented on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns URL: https://github.com/apache/spark/pull/22595#issuecomment-509059826 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107315/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns
AmplabJenkins commented on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns URL: https://github.com/apache/spark/pull/22595#issuecomment-509059822 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns
AmplabJenkins removed a comment on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns URL: https://github.com/apache/spark/pull/22595#issuecomment-509059826 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107315/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns
SparkQA removed a comment on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns URL: https://github.com/apache/spark/pull/22595#issuecomment-509044591 **[Test build #107315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107315/testReport)** for PR 22595 at commit [`c758db0`](https://github.com/apache/spark/commit/c758db01da06d0bc5f4466bab10a330ef812bcc6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns
SparkQA commented on issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display the executor additional columns URL: https://github.com/apache/spark/pull/22595#issuecomment-509059523 **[Test build #107315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107315/testReport)** for PR 22595 at commit [`c758db0`](https://github.com/apache/spark/commit/c758db01da06d0bc5f4466bab10a330ef812bcc6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509059124 **[Test build #107321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107321/testReport)** for PR 24637 at commit [`86e795f`](https://github.com/apache/spark/commit/86e795fb3eb5844aae8e881fb8446bee8239bdbe). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Deegue commented on issue #23593: [SPARK-26667][DOC] Add `Scanning Input Table` to Performance Tuning Guide
Deegue commented on issue #23593: [SPARK-26667][DOC] Add `Scanning Input Table` to Performance Tuning Guide URL: https://github.com/apache/spark/pull/23593#issuecomment-509059134 > I don't disagree with the general comments here, but I'm not sure it adds enough concise and precise information to be worthwhile. It would need to be significantly rewritten. It's also fairly specific to HDFS, right? Yes, you are right. I will rewrite the docs and make it concise and precise. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
SparkQA commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509059126 **[Test build #107320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107320/testReport)** for PR 24993 at commit [`3520138`](https://github.com/apache/spark/commit/352013884694e9ba7d4fbf2c0c1ba259c2fb3174). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #24872: [SPARK-28023][SQL] Trim the string when cast string type to Boolean/Numeric types
srowen commented on issue #24872: [SPARK-28023][SQL] Trim the string when cast string type to Boolean/Numeric types URL: https://github.com/apache/spark/pull/24872#issuecomment-509059035 I personally don't see a compelling reason to do this, even though `trim()` has been optimized somewhat. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509058847 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509058851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12464/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509058795 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12463/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509058847 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509058792 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-509058851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12464/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509058795 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12463/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509058792 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode
viirya commented on a change in pull request #24637: [SPARK-27707][SQL] Prune
unnecessary nested fields from Generate in explode
URL: https://github.com/apache/spark/pull/24637#discussion_r300905402
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##
@@ -588,6 +588,16 @@ object ColumnPruning extends Rule[LogicalPlan] {
.map(_._2)
p.copy(child = g.copy(child = newChild, unrequiredChildIndex =
unrequiredIndices))
+// prune unrequired nested fields
+case p @ Project(projectList, g: Generate) if
SQLConf.get.nestedSchemaPruningEnabled =>
+ NestedColumnAliasing.getAliasSubMap(projectList).map {
+case (nestedFieldToAlias, attrToAliases) =>
+ val newChild = g.withNewChildren(g.children.map { child =>
+Project(child.output ++ attrToAliases.values.flatten, child)
Review comment:
This is good point. I was planning to deal this as separate issue. As you
mentioned and we control it with a pruning on expressions config now, I think
it is good to deal with it here. I will update.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] nooberfsh commented on a change in pull request #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
nooberfsh commented on a change in pull request #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#discussion_r300905150 ## File path: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ## @@ -520,6 +520,71 @@ class KeyValueGroupedDataset[K, V] private[sql]( col4: TypedColumn[V, U4]): Dataset[(K, U1, U2, U3, U4)] = aggUntyped(col1, col2, col3, col4).asInstanceOf[Dataset[(K, U1, U2, U3, U4)]] + /** + * Computes the given aggregations, returning a [[Dataset]] of tuples for each unique key + * and the result of computing these aggregations over all elements in the group. + * + * @since 1.6.0 Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
SparkQA commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509057956 **[Test build #107319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107319/testReport)** for PR 24993 at commit [`1fbe526`](https://github.com/apache/spark/commit/1fbe526d0e99a9eca9d83cb5c1baa2403a432440). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Deegue commented on issue #23593: [SPARK-26667][DOC] Add `Scanning Input Table` to Performance Tuning Guide
Deegue commented on issue #23593: [SPARK-26667][DOC] Add `Scanning Input Table` to Performance Tuning Guide URL: https://github.com/apache/spark/pull/23593#issuecomment-509057830 > Hi, @Deegue . Are you still working on this PR? Yes, I'm waiting for more reviews or I think maybe, we could merge this one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509057608 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12462/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509057605 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins commented on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509057608 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12462/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
AmplabJenkins removed a comment on issue #24993: [SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset URL: https://github.com/apache/spark/pull/24993#issuecomment-509057605 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3)
wangyum commented on a change in pull request #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3) URL: https://github.com/apache/spark/pull/25063#discussion_r300904457 ## File path: docs/building-spark.md ## @@ -79,15 +81,23 @@ Example: ./build/mvn -Pyarn -Dhadoop.version=2.8.5 -DskipTests clean package +./build/mvn -Pyarn -Phadoop-3.2 -Dhadoop.version=3.1.2 -DskipTests clean package + ## Building With Hive and JDBC Support To enable Hive integration for Spark SQL along with its JDBC server and CLI, add the `-Phive` and `Phive-thriftserver` profiles to your existing build options. -By default Spark will build with Hive 1.2.1 bindings. +By default, Spark will build hadoop-2.7 with Hive 1.2.1 and hadoop-3.2 with Hive 2.3.5. # With Hive 1.2.1 support ./build/mvn -Pyarn -Phive -Phive-thriftserver -DskipTests clean package +# With Hive 2.3.5 support +./build/mvn -Pyarn -Phive -Phive-thriftserver -Phadoop-3.2 -DskipTests clean package + +# Hadoop 2.7 with Hive 2.3.5 support +./build/mvn -Pyarn -Phive -Phive-thriftserver -Phadoop-3.2 -Dhadoop.version=2.7.4 -Dcurator.version=2.7.1 -Dzookeeper.version=3.4.6 -DskipTests clean package Review comment: OK. removed it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3)
AmplabJenkins commented on issue #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3) URL: https://github.com/apache/spark/pull/25063#issuecomment-509057168 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3)
SparkQA removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3) URL: https://github.com/apache/spark/pull/25063#issuecomment-509055596 **[Test build #107318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107318/testReport)** for PR 25063 at commit [`a0f2b0a`](https://github.com/apache/spark/commit/a0f2b0a87cb63c03bf8bd50142854ae2843550b8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3)
AmplabJenkins removed a comment on issue #25063: [SPARK-28267][DOC] Update building-spark.md(build with Hive 2.3) URL: https://github.com/apache/spark/pull/25063#issuecomment-509057172 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107318/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
