[GitHub] [spark] AmplabJenkins commented on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system
AmplabJenkins commented on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system URL: https://github.com/apache/spark/pull/26953#issuecomment-595080181 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system
AmplabJenkins removed a comment on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system URL: https://github.com/apache/spark/pull/26953#issuecomment-595080190 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24110/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27643: [SPARK-30886][SQL] Deprecate LTRIM, RTRIM, and two-parameter TRIM functions
AmplabJenkins commented on issue #27643: [SPARK-30886][SQL] Deprecate LTRIM, RTRIM, and two-parameter TRIM functions URL: https://github.com/apache/spark/pull/27643#issuecomment-595080199 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24109/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27643: [SPARK-30886][SQL] Deprecate LTRIM, RTRIM, and two-parameter TRIM functions
AmplabJenkins removed a comment on issue #27643: [SPARK-30886][SQL] Deprecate LTRIM, RTRIM, and two-parameter TRIM functions URL: https://github.com/apache/spark/pull/27643#issuecomment-595080199 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24109/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27643: [SPARK-30886][SQL] Deprecate LTRIM, RTRIM, and two-parameter TRIM functions
AmplabJenkins commented on issue #27643: [SPARK-30886][SQL] Deprecate LTRIM, RTRIM, and two-parameter TRIM functions URL: https://github.com/apache/spark/pull/27643#issuecomment-595080192 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system
AmplabJenkins removed a comment on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system URL: https://github.com/apache/spark/pull/26953#issuecomment-595080181 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system
AmplabJenkins commented on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system URL: https://github.com/apache/spark/pull/26953#issuecomment-595080190 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24110/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27643: [SPARK-30886][SQL] Deprecate LTRIM, RTRIM, and two-parameter TRIM functions
AmplabJenkins removed a comment on issue #27643: [SPARK-30886][SQL] Deprecate LTRIM, RTRIM, and two-parameter TRIM functions URL: https://github.com/apache/spark/pull/27643#issuecomment-595080192 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system
SparkQA commented on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and throughput metrics using Spark Metrics system URL: https://github.com/apache/spark/pull/26953#issuecomment-595079598 **[Test build #119372 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119372/testReport)** for PR 26953 at commit [`989dba1`](https://github.com/apache/spark/commit/989dba136ef6dc3d12a64765f524b19fe89af0ce). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
AmplabJenkins removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595077974 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119370/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
AmplabJenkins removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595077962 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #27789: [SPARK-30541][TEST] testRetry flaky KafkaDelegationTokenSuite
HyukjinKwon edited a comment on issue #27789: [SPARK-30541][TEST] testRetry flaky KafkaDelegationTokenSuite URL: https://github.com/apache/spark/pull/27789#issuecomment-595070637 Reading the error messages, yea, seems it's failed during `setup`. Let's disable this test case and enable it back in a separate PR. Conditional test can be an option if we know this `setup` being failed isn't an issue but just being flaky for other external reasons, and it's difficult to fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
SparkQA commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595077827 **[Test build #119370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119370/testReport)** for PR 27785 at commit [`1a19da7`](https://github.com/apache/spark/commit/1a19da714c4a607454d3dd36d827893454190552). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
SparkQA removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595073049 **[Test build #119370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119370/testReport)** for PR 27785 at commit [`1a19da7`](https://github.com/apache/spark/commit/1a19da714c4a607454d3dd36d827893454190552). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
AmplabJenkins commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595077974 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119370/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
AmplabJenkins commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595077962 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27759: [SPARK-31008][SQL]Support json_array_length function
MaxGekk commented on a change in pull request #27759: [SPARK-31008][SQL]Support
json_array_length function
URL: https://github.com/apache/spark/pull/27759#discussion_r388124829
##
File path: sql/core/src/test/resources/sql-tests/inputs/json-functions.sql
##
@@ -58,5 +58,16 @@ select schema_of_json('{"c1":01, "c2":0.1}',
map('allowNumericLeadingZeros', 'tr
select schema_of_json(null);
CREATE TEMPORARY VIEW jsonTable(jsonField, a) AS SELECT * FROM VALUES ('{"a":
1, "b": 2}', 'a');
SELECT schema_of_json(jsonField) FROM jsonTable;
+
+-- json_array_length
+select json_array_length('');
+select json_array_length('[]');
+select json_array_length('[1,2,3]');
+select json_array_length('[[1,2],[5,6,7]]');
+select json_array_length('[{"a":123},{"b":"hello"}]');
+select json_array_length('[1,2,3,[33,44],{"key":[2,3,4]}]');
+select json_array_length('{"key":"not a json array"}');
Review comment:
The goal of adding of the `json_array_length` function is to make migration
from other DBMS to Spark SQL easier. I think we should focus on the specific
function `json_array_length` already provided by others.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json`
AmplabJenkins removed a comment on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json` URL: https://github.com/apache/spark/pull/27804#issuecomment-595076684 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24108/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json`
AmplabJenkins removed a comment on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json` URL: https://github.com/apache/spark/pull/27804#issuecomment-595076674 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json`
AmplabJenkins commented on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json` URL: https://github.com/apache/spark/pull/27804#issuecomment-595076674 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json`
AmplabJenkins commented on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json` URL: https://github.com/apache/spark/pull/27804#issuecomment-595076684 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24108/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
AmplabJenkins commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595076171 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119358/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
AmplabJenkins removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595076171 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119358/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json`
SparkQA commented on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json` URL: https://github.com/apache/spark/pull/27804#issuecomment-595076195 **[Test build #119371 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119371/testReport)** for PR 27804 at commit [`cd93715`](https://github.com/apache/spark/commit/cd93715861f58035e61fc240ff5eca590757b5b9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
AmplabJenkins commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595076168 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
AmplabJenkins removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595076168 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
SparkQA removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-594995897 **[Test build #119358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119358/testReport)** for PR 27776 at commit [`fa7560a`](https://github.com/apache/spark/commit/fa7560a0b742f8bfca9f9f6293c9fb14d6466d52). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
SparkQA commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595075245 **[Test build #119358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119358/testReport)** for PR 27776 at commit [`fa7560a`](https://github.com/apache/spark/commit/fa7560a0b742f8bfca9f9f6293c9fb14d6466d52). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json`
MaxGekk commented on issue #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json` URL: https://github.com/apache/spark/pull/27804#issuecomment-595074873 @HyukjinKwon @cloud-fan Please, review the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk opened a new pull request #27804: [SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args by `from_csv/json` and `schema_of_csv/json`
MaxGekk opened a new pull request #27804:
[SPARK-31020][SPARK-31023][SPARK-31025][SPARK-31044][SQL] Support foldable args
by `from_csv/json` and `schema_of_csv/json`
URL: https://github.com/apache/spark/pull/27804
### What changes were proposed in this pull request?
In the PR, I propose:
1. To replace matching by `Literal` in `ExprUtils.evalSchemaExpr()` to
checking foldable property of the `schema` expression.
2. To replace matching by `Literal` in `ExprUtils.evalTypeExpr()` to
checking foldable property of the `schema` expression.
3. To change checking of the input parameter in the `SchemaOfCsv`
expression, and allow foldable `child` expression.
4. To change checking of the input parameter in the `SchemaOfJson`
expression, and allow foldable `child` expression.
Closes #27771
Closes #27774
Closes #2
Closes #27797
### Why are the changes needed?
This should improve Spark SQL UX for `from_csv`/`from_json`. Currently,
Spark expects only literals:
```sql
spark-sql> select from_csv('1,Moscow', replace('dpt_org_id INT, dpt_org_city
STRING', 'dpt_org_', ''));
Error in query: Schema should be specified in DDL format as a string literal
or output of the schema_of_csv function instead of replace('dpt_org_id INT,
dpt_org_city STRING', 'dpt_org_', '');; line 1 pos 7
spark-sql> select from_json('{"id":1, "city":"Moscow"}', replace('dpt_org_id
INT, dpt_org_city STRING', 'dpt_org_', ''));
Error in query: Schema should be specified in DDL format as a string literal
or output of the schema_of_json function instead of replace('dpt_org_id INT,
dpt_org_city STRING', 'dpt_org_', '');; line 1 pos 7
```
and only string literals are acceptable as CSV examples by
`schema_of_csv`/`schema_of_json`:
```sql
spark-sql> select schema_of_csv(concat_ws(',', 0.1, 1));
Error in query: cannot resolve 'schema_of_csv(concat_ws(',', CAST(0.1BD AS
STRING), CAST(1 AS STRING)))' due to data type mismatch: The input csv should
be a string literal and not null; however, got concat_ws(',', CAST(0.1BD AS
STRING), CAST(1 AS STRING)).; line 1 pos 7;
'Project [unresolvedalias(schema_of_csv(concat_ws(,, cast(0.1 as string),
cast(1 as string))), None)]
+- OneRowRelation
spark-sql> select schema_of_json(regexp_replace('{"item_id": 1,
"item_price": 0.1}', 'item_', ''));
Error in query: cannot resolve 'schema_of_json(regexp_replace('{"item_id":
1, "item_price": 0.1}', 'item_', ''))' due to data type mismatch: The input
json should be a string literal and not null; however, got
regexp_replace('{"item_id": 1, "item_price": 0.1}', 'item_', '').; line 1 pos 7;
'Project [unresolvedalias(schema_of_json(regexp_replace({"item_id": 1,
"item_price": 0.1}, item_, )), None)]
+- OneRowRelation
```
### Does this PR introduce any user-facing change?
Yes, after the changes users can pass any foldable string expression as the
`schema` parameter to `from_csv()/from_json()`. For the example above:
```sql
spark-sql> select from_csv('1,Moscow', replace('dpt_org_id INT, dpt_org_city
STRING', 'dpt_org_', ''));
{"id":1,"city":"Moscow"}
spark-sql> select from_json('{"id":1, "city":"Moscow"}', replace('dpt_org_id
INT, dpt_org_city STRING', 'dpt_org_', ''));
{"id":1,"city":"Moscow"}
```
After change the `schema_of_csv`/`schema_of_json` functions accept foldable
expressions, for example:
```sql
spark-sql> select schema_of_csv(concat_ws(',', 0.1, 1));
struct<_c0:double,_c1:int>
spark-sql> select schema_of_json(regexp_replace('{"item_id": 1,
"item_price": 0.1}', 'item_', ''));
struct
```
### How was this patch tested?
Added new test to `CsvFunctionsSuite` and to `JsonFunctionsSuite`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone
AmplabJenkins removed a comment on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone URL: https://github.com/apache/spark/pull/27792#issuecomment-595074137 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119357/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone
AmplabJenkins commented on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone URL: https://github.com/apache/spark/pull/27792#issuecomment-595074126 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone
AmplabJenkins removed a comment on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone URL: https://github.com/apache/spark/pull/27792#issuecomment-595074126 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone
AmplabJenkins commented on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone URL: https://github.com/apache/spark/pull/27792#issuecomment-595074137 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119357/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-595073561 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24106/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
AmplabJenkins removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595073587 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24107/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
AmplabJenkins removed a comment on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595073581 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-595073548 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
AmplabJenkins commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595073581 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-595073561 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24106/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone
SparkQA removed a comment on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone URL: https://github.com/apache/spark/pull/27792#issuecomment-594995921 **[Test build #119357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119357/testReport)** for PR 27792 at commit [`641b74b`](https://github.com/apache/spark/commit/641b74b4a3987476961e047103a33a03d8743bca). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-595073548 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone
SparkQA commented on issue #27792: [SPARK-31038][SQL] Add checkValue for spark.sql.session.timeZone URL: https://github.com/apache/spark/pull/27792#issuecomment-595073303 **[Test build #119357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119357/testReport)** for PR 27792 at commit [`641b74b`](https://github.com/apache/spark/commit/641b74b4a3987476961e047103a33a03d8743bca). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
AmplabJenkins commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595073587 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24107/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-595073051 **[Test build #119369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119369/testReport)** for PR 27803 at commit [`a98942a`](https://github.com/apache/spark/commit/a98942ac1b42b58b2dc899e18f2062bbe46ffa47). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release
SparkQA commented on issue #27785: [SPARK-30934][ML][DOCS] Update ml-guide and ml-migration-guide for 3.0 release URL: https://github.com/apache/spark/pull/27785#issuecomment-595073049 **[Test build #119370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119370/testReport)** for PR 27785 at commit [`1a19da7`](https://github.com/apache/spark/commit/1a19da714c4a607454d3dd36d827893454190552). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27759: [SPARK-31008][SQL]Support json_array_length function
maropu commented on a change in pull request #27759: [SPARK-31008][SQL]Support
json_array_length function
URL: https://github.com/apache/spark/pull/27759#discussion_r388119969
##
File path: sql/core/src/test/resources/sql-tests/inputs/json-functions.sql
##
@@ -58,5 +58,16 @@ select schema_of_json('{"c1":01, "c2":0.1}',
map('allowNumericLeadingZeros', 'tr
select schema_of_json(null);
CREATE TEMPORARY VIEW jsonTable(jsonField, a) AS SELECT * FROM VALUES ('{"a":
1, "b": 2}', 'a');
SELECT schema_of_json(jsonField) FROM jsonTable;
+
+-- json_array_length
+select json_array_length('');
+select json_array_length('[]');
+select json_array_length('[1,2,3]');
+select json_array_length('[[1,2],[5,6,7]]');
+select json_array_length('[{"a":123},{"b":"hello"}]');
+select json_array_length('[1,2,3,[33,44],{"key":[2,3,4]}]');
+select json_array_length('{"key":"not a json array"}');
Review comment:
WDYT? @HyukjinKwon @MaxGekk
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27759: [SPARK-31008][SQL]Support json_array_length function
maropu commented on a change in pull request #27759: [SPARK-31008][SQL]Support
json_array_length function
URL: https://github.com/apache/spark/pull/27759#discussion_r388119969
##
File path: sql/core/src/test/resources/sql-tests/inputs/json-functions.sql
##
@@ -58,5 +58,16 @@ select schema_of_json('{"c1":01, "c2":0.1}',
map('allowNumericLeadingZeros', 'tr
select schema_of_json(null);
CREATE TEMPORARY VIEW jsonTable(jsonField, a) AS SELECT * FROM VALUES ('{"a":
1, "b": 2}', 'a');
SELECT schema_of_json(jsonField) FROM jsonTable;
+
+-- json_array_length
+select json_array_length('');
+select json_array_length('[]');
+select json_array_length('[1,2,3]');
+select json_array_length('[[1,2],[5,6,7]]');
+select json_array_length('[{"a":123},{"b":"hello"}]');
+select json_array_length('[1,2,3,[33,44],{"key":[2,3,4]}]');
+select json_array_length('{"key":"not a json array"}');
Review comment:
WDYT? @HyukjinKwon
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
maropu commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-595071982 Thanks, @HyukjinKwon ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu opened a new pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
maropu opened a new pull request #27803: [SPARK-31049][SQL] Support nested
adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803
### What changes were proposed in this pull request?
In the master, we currently don't support any nested generators, but I think
supporting limited nested cases is somewhat useful for users, e.g.,
explode(explode(v)). This PR intends to add some logics in `ExtractGenerator`
for supporting the nested generators as follows;
```
// before this PR
scala> sql("select explode(explode(array(array(1, 2), array(3").show()
org.apache.spark.sql.AnalysisException: Generators are not supported when
it's nested in expressions, but got: explode(explode(array(array(1, 2),
array(3;
// after this PR
scala> sql("select explode(explode(array(array(1, 2), array(3").show()
+---+
|col|
+---+
| 1|
| 2|
| 3|
+---+
```
### Why are the changes needed?
For usability.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit
AmplabJenkins removed a comment on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit URL: https://github.com/apache/spark/pull/27802#issuecomment-595063286 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119360/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
AmplabJenkins removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595065020 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119352/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit
AmplabJenkins removed a comment on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit URL: https://github.com/apache/spark/pull/27802#issuecomment-595063281 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
AmplabJenkins removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595065013 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
SparkQA removed a comment on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-594979436 **[Test build #119352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119352/testReport)** for PR 27776 at commit [`fa7560a`](https://github.com/apache/spark/commit/fa7560a0b742f8bfca9f9f6293c9fb14d6466d52). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27789: [SPARK-30541][TEST] testRetry flaky KafkaDelegationTokenSuite
HyukjinKwon commented on issue #27789: [SPARK-30541][TEST] testRetry flaky KafkaDelegationTokenSuite URL: https://github.com/apache/spark/pull/27789#issuecomment-595070637 Reading from the error messages, yea, seems it's failed during `setup`. Let's disable this test case and enable it back in a separate PR. Conditional test can be an option if we know this `setup` being failed isn't an issue but just being flaky for other external reasons, and it's difficult to fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27537: [SPARK-30668][SQL][FOLLOWUP] Raise exception instead of silent change for new DateFormatter
cloud-fan commented on issue #27537: [SPARK-30668][SQL][FOLLOWUP] Raise exception instead of silent change for new DateFormatter URL: https://github.com/apache/spark/pull/27537#issuecomment-595070303 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27537: [SPARK-30668][SQL][FOLLOWUP] Raise exception instead of silent change for new DateFormatter
cloud-fan closed pull request #27537: [SPARK-30668][SQL][FOLLOWUP] Raise exception instead of silent change for new DateFormatter URL: https://github.com/apache/spark/pull/27537 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow
cloud-fan edited a comment on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow URL: https://github.com/apache/spark/pull/27627#issuecomment-595067684 cc @viirya @maropu @dongjoon-hyun as well This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow
cloud-fan commented on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow URL: https://github.com/apache/spark/pull/27627#issuecomment-595067684 cc @viirya @maropu as well This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow
cloud-fan commented on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect
results for decimal aggregate sum by returning null on decimal overflow
URL: https://github.com/apache/spark/pull/27627#issuecomment-595067541
find a way to reproduce without join
```
scala> val decimalStr = "1" + "0" * 19
decimalStr: String = 1000
scala> val df = spark.range(0, 12, 1, 1)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]
scala> df.select(expr(s"cast('$decimalStr' as decimal (38, 18)) as
d")).agg(sum($"d")).show
// This is correct
+--+
|sum(d)|
+--+
| null|
+--+
scala> val df = spark.range(0, 1, 1, 1).union(spark.range(0, 11, 1, 1))
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]
scala> df.select(expr(s"cast('$decimalStr' as decimal (38, 18)) as
d")).agg(sum($"d")).show
// This is wrong
++
| sum(d)|
++
|1...|
++
```
I think the root cause is, `sum` in partial aggregate overflows and write
null to the unsafe row. `sum` in final aggregate reads null from the unsafe row
and mistakenly think it's caused by empty data and convert it to 0.
We should create a `DecimalSum`, which use 2 buffer attributes: `sum` and
`isEmpty`. Then in final aggregate we can check the `isEmpty` flag to konw if
the null is caused by overflow or empty data.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27789: [SPARK-30541][TEST] testRetry flaky KafkaDelegationTokenSuite
HeartSaVioR commented on issue #27789: [SPARK-30541][TEST] testRetry flaky KafkaDelegationTokenSuite URL: https://github.com/apache/spark/pull/27789#issuecomment-595067025 Have we observed the case where the setup was passed and the test "roundtrip" failed? The errors what I have been observed are all setup failure. If we don't mind about conditional test, maybe we can "catch" exception in beforeAll, and run the code "roundtrip" conditionally depending on the result of setup. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
AmplabJenkins commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595065013 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
AmplabJenkins commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595065020 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119352/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables
SparkQA commented on issue #27776: [SPARK-31024][SQL] Allow specifying session catalog name `spark_catalog` in qualified column names for v1 tables URL: https://github.com/apache/spark/pull/27776#issuecomment-595064114 **[Test build #119352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119352/testReport)** for PR 27776 at commit [`fa7560a`](https://github.com/apache/spark/commit/fa7560a0b742f8bfca9f9f6293c9fb14d6466d52). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit
AmplabJenkins commented on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit URL: https://github.com/apache/spark/pull/27802#issuecomment-595063281 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit
AmplabJenkins commented on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit URL: https://github.com/apache/spark/pull/27802#issuecomment-595063286 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119360/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on a change in pull request #27789: [SPARK-30541][TEST] testRetry flaky KafkaDelegationTokenSuite
zsxwing commented on a change in pull request #27789: [SPARK-30541][TEST]
testRetry flaky KafkaDelegationTokenSuite
URL: https://github.com/apache/spark/pull/27789#discussion_r388110374
##
File path:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDelegationTokenSuite.scala
##
@@ -62,7 +62,7 @@ class KafkaDelegationTokenSuite extends StreamTest with
SharedSparkSession with
}
}
- test("Roundtrip") {
+ testRetry("Roundtrip", 3) {
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit
SparkQA removed a comment on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit URL: https://github.com/apache/spark/pull/27802#issuecomment-595017164 **[Test build #119360 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119360/testReport)** for PR 27802 at commit [`345d24e`](https://github.com/apache/spark/commit/345d24ea96aef8ddcd9c9dd155e585439661d992). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit
SparkQA commented on issue #27802: [MINOR][CORE] Expose the alias -c flag of --conf for spark-submit URL: https://github.com/apache/spark/pull/27802#issuecomment-595062647 **[Test build #119360 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119360/testReport)** for PR 27802 at commit [`345d24e`](https://github.com/apache/spark/commit/345d24ea96aef8ddcd9c9dd155e585439661d992). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27789: [SPARK-30541][TEST] testRetry flaky KafkaDelegationTokenSuite
HyukjinKwon commented on a change in pull request #27789: [SPARK-30541][TEST]
testRetry flaky KafkaDelegationTokenSuite
URL: https://github.com/apache/spark/pull/27789#discussion_r388108959
##
File path:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDelegationTokenSuite.scala
##
@@ -62,7 +62,7 @@ class KafkaDelegationTokenSuite extends StreamTest with
SharedSparkSession with
}
}
- test("Roundtrip") {
+ testRetry("Roundtrip", 3) {
Review comment:
Sounds fine to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27793: [SPARK-31037][SQL] refine AQE config names
SparkQA commented on issue #27793: [SPARK-31037][SQL] refine AQE config names URL: https://github.com/apache/spark/pull/27793#issuecomment-595060829 **[Test build #119368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119368/testReport)** for PR 27793 at commit [`f2dea40`](https://github.com/apache/spark/commit/f2dea40824eb9fcd3b148822c774508ded919725). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #27789: [SPARK-30541][TEST] testRetry flaky KafkaDelegationTokenSuite
gatorsmile commented on a change in pull request #27789: [SPARK-30541][TEST]
testRetry flaky KafkaDelegationTokenSuite
URL: https://github.com/apache/spark/pull/27789#discussion_r388107442
##
File path:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDelegationTokenSuite.scala
##
@@ -62,7 +62,7 @@ class KafkaDelegationTokenSuite extends StreamTest with
SharedSparkSession with
}
}
- test("Roundtrip") {
+ testRetry("Roundtrip", 3) {
Review comment:
@zsxwing @gaborgsomogyi Should we disable this flaky test first and create a
blocker ticket for 3.0?
If we keep hitting the test failure, it will impact the productivity of the
open source development.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27793: [SPARK-31037][SQL] refine AQE config names
AmplabJenkins removed a comment on issue #27793: [SPARK-31037][SQL] refine AQE config names URL: https://github.com/apache/spark/pull/27793#issuecomment-595058878 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24105/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27793: [SPARK-31037][SQL] refine AQE config names
AmplabJenkins removed a comment on issue #27793: [SPARK-31037][SQL] refine AQE config names URL: https://github.com/apache/spark/pull/27793#issuecomment-595058869 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27793: [SPARK-31037][SQL] refine AQE config names
AmplabJenkins commented on issue #27793: [SPARK-31037][SQL] refine AQE config names URL: https://github.com/apache/spark/pull/27793#issuecomment-595058869 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27793: [SPARK-31037][SQL] refine AQE config names
AmplabJenkins commented on issue #27793: [SPARK-31037][SQL] refine AQE config names URL: https://github.com/apache/spark/pull/27793#issuecomment-595058878 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24105/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
AmplabJenkins removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595055792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119354/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
AmplabJenkins removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595055789 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-595056218 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-595056224 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119353/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-595056224 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119353/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-595056218 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
AmplabJenkins commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595055792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119354/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
AmplabJenkins commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595055789 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
SparkQA removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-594979488 **[Test build #119353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119353/testReport)** for PR 27728 at commit [`ea2c495`](https://github.com/apache/spark/commit/ea2c495e26115e96532efd18d495464f4482844e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
SparkQA commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-595055285 **[Test build #119353 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119353/testReport)** for PR 27728 at commit [`ea2c495`](https://github.com/apache/spark/commit/ea2c495e26115e96532efd18d495464f4482844e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
SparkQA removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-594981379 **[Test build #119354 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119354/testReport)** for PR 27780 at commit [`b6229e7`](https://github.com/apache/spark/commit/b6229e7899bbc1eb9f8eeabf7919b5a88c2e1056). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
SparkQA commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595055120 **[Test build #119354 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119354/testReport)** for PR 27780 at commit [`b6229e7`](https://github.com/apache/spark/commit/b6229e7899bbc1eb9f8eeabf7919b5a88c2e1056). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
AmplabJenkins removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595052408 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
AmplabJenkins commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595052408 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
AmplabJenkins removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595052413 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119349/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
AmplabJenkins commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595052413 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119349/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
SparkQA removed a comment on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-594977411 **[Test build #119349 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119349/testReport)** for PR 27780 at commit [`5b77ecf`](https://github.com/apache/spark/commit/5b77ecf564483d968b8681404d92fcb46997e726). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ScrapCodes edited a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
ScrapCodes edited a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent URL: https://github.com/apache/spark/pull/27800#issuecomment-595051659 @nchammas, the patch looks good to me. We can make one more improvement though. The command `dev/make-distribution.sh`'s usage help, makes it very clear that maven cli option should be at the end. Since the error output, if the user makes a mistake with the position of the `MAVEN_CLI_OPTIONS` is very misleading. So this patch is helpful ! The improvement, I would recommend is, is the user gives a misspelled CLI option, the error output is still misleading, it might as well be good to fix the error output. e.g. `./dev/make-distribution.sh --pip -r --tgz` Produces the error: `+ VERSION=' -X,--debug Produce execution debug output'` Which is not at all helpful. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ScrapCodes edited a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
ScrapCodes edited a comment on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent URL: https://github.com/apache/spark/pull/27800#issuecomment-595051659 @nchammas, the patch looks good to me. We can make one more improvement though. The command `dev/make-distribution.sh`'s usage help, makes it very clear that maven cli option should be at the end. Since the error output, if the user makes a mistake with the position of the `MAVEN_CLI_OPTIONS` is very misleading. So this patch is helpful ! The improvement, I would recommend is, if the user gives a misspelled CLI option, the error output is still misleading, it might as well be good to fix the error output. e.g. `./dev/make-distribution.sh --pip -r --tgz` Produces the error: `+ VERSION=' -X,--debug Produce execution debug output'` Which is not at all helpful. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots
SparkQA commented on issue #27780: [SPARK-31026] [SQL] [test-hive1.2] Parquet predicate pushdown on columns with dots URL: https://github.com/apache/spark/pull/27780#issuecomment-595051518 **[Test build #119349 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119349/testReport)** for PR 27780 at commit [`5b77ecf`](https://github.com/apache/spark/commit/5b77ecf564483d968b8681404d92fcb46997e726). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ScrapCodes commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent
ScrapCodes commented on issue #27800: [SPARK-31041][BUILD] Make arguments to make-distribution.sh position-independent URL: https://github.com/apache/spark/pull/27800#issuecomment-595051659 @nchammas, the patch looks good to me. We can make one more improvement though. The command `dev/make-distribution.sh`'s usage help, makes it very clear that maven cli option should be at the end. Since the error output if the user makes a mistake with the position of the `MAVEN_CLI_OPTIONS` is very misleading. So this patch is helpful ! The improvement, I would recommend is, is the user gives a misspelled CLI option, the error output is still misleading, it might as well be good to fix the error output. e.g. `./dev/make-distribution.sh --pip -r --tgz` Produces the error: `+ VERSION=' -X,--debug Produce execution debug output'` Which is not at all helpful. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27759: [SPARK-31008][SQL]Support json_array_length function
AmplabJenkins removed a comment on issue #27759: [SPARK-31008][SQL]Support json_array_length function URL: https://github.com/apache/spark/pull/27759#issuecomment-595048017 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27759: [SPARK-31008][SQL]Support json_array_length function
AmplabJenkins removed a comment on issue #27759: [SPARK-31008][SQL]Support json_array_length function URL: https://github.com/apache/spark/pull/27759#issuecomment-595048026 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24104/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
