[GitHub] [spark] AmplabJenkins removed a comment on issue #22029: [SPARK-24395][SQL] IN operator should return NULL when comparing struct with NULL fields
AmplabJenkins removed a comment on issue #22029: [SPARK-24395][SQL] IN operator should return NULL when comparing struct with NULL fields URL: https://github.com/apache/spark/pull/22029#issuecomment-514491232 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13186/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22029: [SPARK-24395][SQL] IN operator should return NULL when comparing struct with NULL fields
AmplabJenkins commented on issue #22029: [SPARK-24395][SQL] IN operator should return NULL when comparing struct with NULL fields URL: https://github.com/apache/spark/pull/22029#issuecomment-514491225 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22029: [SPARK-24395][SQL] IN operator should return NULL when comparing struct with NULL fields
AmplabJenkins removed a comment on issue #22029: [SPARK-24395][SQL] IN operator should return NULL when comparing struct with NULL fields URL: https://github.com/apache/spark/pull/22029#issuecomment-514491225 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22029: [SPARK-24395][SQL] IN operator should return NULL when comparing struct with NULL fields
AmplabJenkins commented on issue #22029: [SPARK-24395][SQL] IN operator should return NULL when comparing struct with NULL fields URL: https://github.com/apache/spark/pull/22029#issuecomment-514491232 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13186/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base
HyukjinKwon closed pull request #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25161 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base
HyukjinKwon commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25161#issuecomment-514488233 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sujith71955 commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
sujith71955 commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-514487952 @maropu i updated the PR based on your comments. thanks :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit
SparkQA removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit URL: https://github.com/apache/spark/pull/25235#issuecomment-514451678 **[Test build #108070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108070/testReport)** for PR 25235 at commit [`b475774`](https://github.com/apache/spark/commit/b47577483935c51e19402a1c0c45f98526e692e6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit
AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit URL: https://github.com/apache/spark/pull/25235#issuecomment-514487048 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108070/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit
AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit URL: https://github.com/apache/spark/pull/25235#issuecomment-514487046 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit
AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit URL: https://github.com/apache/spark/pull/25235#issuecomment-514487048 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108070/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit
AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit URL: https://github.com/apache/spark/pull/25235#issuecomment-514487046 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit
SparkQA commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks do not exit URL: https://github.com/apache/spark/pull/25235#issuecomment-514486582 **[Test build #108070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108070/testReport)** for PR 25235 at commit [`b475774`](https://github.com/apache/spark/commit/b47577483935c51e19402a1c0c45f98526e692e6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
SparkQA commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-514486480 **[Test build #108079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108079/testReport)** for PR 24903 at commit [`6bb843c`](https://github.com/apache/spark/commit/6bb843c971a8fb161c9836141614593b9f8e277a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-514486050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13185/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins removed a comment on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-514486046 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-514486046 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
AmplabJenkins commented on issue #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command URL: https://github.com/apache/spark/pull/24903#issuecomment-514486050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13185/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base
AmplabJenkins removed a comment on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25161#issuecomment-514485753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108068/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base
AmplabJenkins removed a comment on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25161#issuecomment-514485751 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base
AmplabJenkins commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25161#issuecomment-514485753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108068/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base
AmplabJenkins commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25161#issuecomment-514485751 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base
SparkQA removed a comment on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25161#issuecomment-514436813 **[Test build #108068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108068/testReport)** for PR 25161 at commit [`8795d66`](https://github.com/apache/spark/commit/8795d66b189712f54a55a3b0663273fb26126a8e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base
SparkQA commented on issue #25161: [SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base URL: https://github.com/apache/spark/pull/25161#issuecomment-514485317 **[Test build #108068 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108068/testReport)** for PR 25161 at commit [`8795d66`](https://github.com/apache/spark/commit/8795d66b189712f54a55a3b0663273fb26126a8e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rxin commented on issue #25024: [SPARK-27296][SQL] User Defined Aggregators that do not ser/de on each input row
rxin commented on issue #25024: [SPARK-27296][SQL] User Defined Aggregators that do not ser/de on each input row URL: https://github.com/apache/spark/pull/25024#issuecomment-514484414 Thanks. Will take a look at it this week. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
AmplabJenkins removed a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514482998 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13184/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
AmplabJenkins removed a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514482990 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
AmplabJenkins commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514482998 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13184/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
AmplabJenkins commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514482990 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sujith71955 commented on a change in pull request #24903: [SPARK-28084][SQL] Resolving the partition column name based on the resolver in sql load command
sujith71955 commented on a change in pull request #24903: [SPARK-28084][SQL]
Resolving the partition column name based on the resolver in sql load command
URL: https://github.com/apache/spark/pull/24903#discussion_r306629137
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
##
@@ -290,6 +291,44 @@ class HiveCommandSuite extends QueryTest with
SQLTestUtils with TestHiveSingleto
checkAnswer(
sql("SELECT employeeID, employeeName FROM part_table WHERE c = '2' AND
d = '1'"),
sql("SELECT * FROM non_part_table").collect())
+
+ // check for case insensitive property of partition column name in load
command.
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+withInputFile { f =>
+ sql(s"""$loadQuery INPATH "${f.toURI}" INTO TABLE part_table
PARTITION(C="1", D="2")""")
+}
+checkAnswer(
+ sql("SELECT employeeID, employeeName FROM part_table WHERE c = '2'
AND d = '1'"),
+ sql("SELECT * FROM non_part_table").collect())
Review comment:
you want me to move this testcase to DDLParserSuite.scala?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
SparkQA commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514481898 **[Test build #108078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108078/testReport)** for PR 25022 at commit [`0d33e22`](https://github.com/apache/spark/commit/0d33e22f4cd945c13842ee9419133e6156d92a26). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] advancedxy commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD should respect SparkContext's hadoop configuration
advancedxy commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD should respect SparkContext's hadoop configuration URL: https://github.com/apache/spark/pull/25002#issuecomment-514481630 @dongjoon-hyun sorry for the delay. I added a test case which should clearly show the wrongly used case in PythonRDD. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25202: [SPARK-28289][SQL][PYTHON][TESTS] Convert and port 'union.sql' into UDF test base
HyukjinKwon commented on a change in pull request #25202: [SPARK-28289][SQL][PYTHON][TESTS] Convert and port 'union.sql' into UDF test base URL: https://github.com/apache/spark/pull/25202#discussion_r306628576 ## File path: sql/core/src/test/resources/sql-tests/results/udf/udf-union.sql.out ## @@ -0,0 +1,166 @@ +-- Automatically generated by SQLQueryTestSuite +-- Number of queries: 16 + + +-- !query 0 +CREATE OR REPLACE TEMPORARY VIEW t1 AS VALUES (1, 'a'), (2, 'b') tbl(c1, c2) +-- !query 0 schema +struct<> +-- !query 0 output + + + +-- !query 1 +CREATE OR REPLACE TEMPORARY VIEW t2 AS VALUES (1.0, 1), (2.0, 4) tbl(c1, c2) +-- !query 1 schema +struct<> +-- !query 1 output + + + +-- !query 2 +SELECT udf(c1) as c1, udf(c2) as c2 +FROM (SELECT udf(c1) as c1, udf(c2) as c2 FROM t1 +UNION ALL +SELECT udf(c1) as c1, udf(c2) as c2 FROM t1) +-- !query 2 schema +struct +-- !query 2 output +1 a +1 a +2 b +2 b + + +-- !query 3 +SELECT udf(c1) as c1, udf(c2) as c2 +FROM (SELECT udf(c1) as c1, udf(c2) as c2 FROM t1 +UNION ALL +SELECT udf(c1) as c1, udf(c2) as c2 FROM t2 +UNION ALL +SELECT udf(c1) as c1, udf(c2) as c2 FROM t2) +-- !query 3 schema +struct +-- !query 3 output +1 1 +1 1 +1 a +2 4 +2 4 +2 b + + +-- !query 4 +SELECT udf(a) as a, udf(udf(a)) as a1 +FROM (SELECT udf(0) a, udf(0) b + UNION ALL + SELECT udf(SUM(1)) a, udf(CAST(0 AS BIGINT)) b + UNION ALL SELECT udf(0) a, udf(0) b) T +-- !query 4 schema +struct +-- !query 4 output +0 0 +0 0 +1 1 Review comment: Ah, I meant to `udf(udf(a)) as a` one column alone. This test cases are being matched with existing `union.sql`. Let's match with this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] priyankagargnitk commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
priyankagargnitk commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514480991 https://issues.apache.org/jira/browse/SPARK-28491 for SQL Praser changes https://issues.apache.org/jira/browse/SPARK-28492 for python changes https://issues.apache.org/jira/browse/SPARK-28493 For R changes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Udbhav30 removed a comment on issue #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts
Udbhav30 removed a comment on issue #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts URL: https://github.com/apache/spark/pull/24601#issuecomment-504776597 Gentle ping @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream
AmplabJenkins removed a comment on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream URL: https://github.com/apache/spark/pull/25180#issuecomment-514478742 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream
AmplabJenkins commented on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream URL: https://github.com/apache/spark/pull/25180#issuecomment-514478742 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream
AmplabJenkins removed a comment on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream URL: https://github.com/apache/spark/pull/25180#issuecomment-514478745 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13183/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream
AmplabJenkins commented on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream URL: https://github.com/apache/spark/pull/25180#issuecomment-514478745 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13183/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream
SparkQA commented on issue #25180: [SPARK-28423][SQL] Merge Scan and Batch/Stream URL: https://github.com/apache/spark/pull/25180#issuecomment-514477786 **[Test build #108077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108077/testReport)** for PR 25180 at commit [`9c826f3`](https://github.com/apache/spark/commit/9c826f3995ce07aeddc1678bbea057899d5e037a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
AmplabJenkins removed a comment on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query URL: https://github.com/apache/spark/pull/25107#issuecomment-514477447 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13182/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
AmplabJenkins removed a comment on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query URL: https://github.com/apache/spark/pull/25107#issuecomment-514477444 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
AmplabJenkins commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query URL: https://github.com/apache/spark/pull/25107#issuecomment-514477447 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13182/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
AmplabJenkins commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query URL: https://github.com/apache/spark/pull/25107#issuecomment-514477444 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
cloud-fan commented on a change in pull request #25107: [SPARK-28344][SQL]
detect ambiguous self-join and fail the query
URL: https://github.com/apache/spark/pull/25107#discussion_r306624960
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/analysis/DetectAmbiguousSelfJoin.scala
##
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.analysis
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.{AnalysisException, Column, Dataset}
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Cast,
Equality, Expression, ExprId}
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Detects ambiguous self-joins, so that we can fail the query instead of
returning confusing
+ * results.
+ *
+ * Dataset column reference is simply an [[AttributeReference]] that is
returned by `Dataset#col`.
+ * Most of time we don't need to do anything special, as
[[AttributeReference]] can point to
+ * the column precisely. However, in case of self-join, the analyzer generates
+ * [[AttributeReference]] with new expr IDs for the right side plan of the
join. If the Dataset
+ * column reference points to a column in the right side plan of a self-join,
users will get
+ * unexpected result because the column reference can't match the newly
generated
+ * [[AttributeReference]].
+ */
+class DetectAmbiguousSelfJoin(conf: SQLConf) extends Rule[LogicalPlan] {
+
+ // Dataset column reference is an `AttributeReference` with 2 special
metadata.
+ private def isColumnReference(a: AttributeReference): Boolean = {
+a.metadata.contains(Dataset.ID_PREFIX) &&
a.metadata.contains(Dataset.COL_POS_PREFIX)
+ }
+
+ private case class ColumnReference(datasetId: Long, colPos: Int, exprId:
ExprId)
+
+ private def toColumnReference(a: AttributeReference): ColumnReference = {
+ColumnReference(
+ a.metadata.getLong(Dataset.ID_PREFIX),
+ a.metadata.getLong(Dataset.COL_POS_PREFIX).toInt,
+ a.exprId)
+ }
+
+ object LogicalPlanWithDatasetId {
+def unapply(p: LogicalPlan): Option[(LogicalPlan, Long)] = {
+ p.getTagValue(Dataset.DATASET_ID_TAG).map(id => p -> id)
+}
+ }
+
+ object AttrWithCast {
+def unapply(expr: Expression): Option[AttributeReference] = expr match {
+ case Cast(child, _, _) => unapply(child)
+ case a: AttributeReference => Some(a)
+ case _ => None
+}
+ }
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+if (!conf.getConf(SQLConf.FAIL_AMBIGUOUS_SELF_JOIN)) return plan
+
+// We always remove the special metadata from `AttributeReference` at the
end of this rule, so
+// Dataset column reference only exists in the root node via Dataset
transformations like
+// `Dataset#select`.
+val colRefAttrs = plan.expressions.flatMap(_.collect {
+ case a: AttributeReference if isColumnReference(a) => a
+})
+
+if (colRefAttrs.nonEmpty) {
+ val colRefs = colRefAttrs.map(toColumnReference).distinct
+ val ambiguousColRefs = mutable.HashSet.empty[ColumnReference]
+ val dsIdSet = colRefs.map(_.datasetId).toSet
+
+ plan.foreach {
+case LogicalPlanWithDatasetId(p, id) if dsIdSet.contains(id) =>
+ colRefs.foreach { ref =>
+if (id == ref.datasetId) {
+ if (ref.colPos < 0 || ref.colPos >= p.output.length) {
+throw new IllegalStateException("[BUG] Hit an invalid Dataset
column reference: " +
+ s"$ref. Please open a JIRA ticket to report it.")
+ } else {
+// When self-join happens, the analyzer asks the right side
plan to generate
+// attributes with new exprIds. If a plan of a Dataset outputs
an attribute which
+// is referred by a column reference, and this attribute has
different exprId than
+// the attribute of column reference, then the column
reference is ambiguous, as it
+// refers to a column that gets regenerated by self-join.
+val
[GitHub] [spark] SparkQA commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
SparkQA commented on issue #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query URL: https://github.com/apache/spark/pull/25107#issuecomment-514476517 **[Test build #108076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108076/testReport)** for PR 25107 at commit [`6bff6f3`](https://github.com/apache/spark/commit/6bff6f32a6a1ccd237e0f33ae9228562fb8932b2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query
cloud-fan commented on a change in pull request #25107: [SPARK-28344][SQL]
detect ambiguous self-join and fail the query
URL: https://github.com/apache/spark/pull/25107#discussion_r306624656
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -787,6 +787,13 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val FAIL_AMBIGUOUS_SELF_JOIN =
+buildConf("spark.sql.analyzer.failAmbiguousSelfJoin")
+ .doc("When true, fail the Dataset query if it contains ambiguous
self-join.")
+ .internal()
+ .booleanConf
+ .createWithDefault(true)
+
// Whether to retain group by columns or not in GroupedData.agg.
Review comment:
This is unrelated to this patch, we can fix it and other similar configs in
another PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r306621651
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
##
@@ -452,6 +452,45 @@ class StringExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
// scalastyle:on
}
+ test("overlay for byte array") {
+val input = Literal(Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9))
+checkEvaluation(new Overlay(input, Literal(Array[Byte](-1)),
+ Literal.create(6, IntegerType)), Array[Byte](1, 2, 3, 4, 5, -1, 7, 8, 9))
Review comment:
I have added tests for negative and zero. I think this behavior is very
different from ANSI SQL and other databases.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514473910 **[Test build #108075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108075/testReport)** for PR 25172 at commit [`3611711`](https://github.com/apache/spark/commit/3611711c191c0fbc2ac289907650552a913e12fa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514473525 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13181/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514473520 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514473520 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514473525 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13181/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514473138 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13180/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514473134 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514473138 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13180/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514472372 Let's file each JIRA for SparkSQL, PySpark and SparkR. Please link them each other or make an umbrella JIRA that contains all. 1. SQL parser side: `CalendarInterval` DDL type string should be supported in SparkSQL - we need a change like this https://github.com/apache/spark/pull/25189/commits/e7e2f5bc2c88958fb672d133b55e8c01b7298f40#diff-9847f5cef7cf7fbc5830fbc6b779ee10R1929 2. Python side: `CalendarInterval` type should be supported in PySpark - see https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L45-L94 We need to define the interval type class and should support presumably time delta in Python - need to check if this works with Python serde. Let's say how it works with Python UDF. See https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala 3. R side: `CalendarInterval` type should be supported in SparkR - R side does not have type class instances IIRC. So we wouldn't need to define the class - However, we should still fix R serde side as well. Presumably `difftime` instances might have to be matched with interval type in SQL sides. See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/SerDe.scala and https://github.com/apache/spark/blob/master/R/pkg/R/types.R This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514473134 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514473072 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13180/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514472372 Let's file each JIRA for SparkSQL, PySpark and SparkR. 1. SQL parser side: `CalendarInterval` DDL type string should be supported in SparkSQL - we need a change like this https://github.com/apache/spark/pull/25189/commits/e7e2f5bc2c88958fb672d133b55e8c01b7298f40#diff-9847f5cef7cf7fbc5830fbc6b779ee10R1929 2. Python side: `CalendarInterval` type should be supported in PySpark - see https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L45-L94 We need to define the interval type class and should support presumably time delta in Python - need to check if this works with Python serde. Let's say how it works with Python UDF. See https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala 3. R side: `CalendarInterval` type should be supported in SparkR - R side does not have type class instances IIRC. So we wouldn't need to define the class - However, we should still fix R serde side as well. Presumably `difftime` instances might have to be matched with interval type in SQL sides. See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/SerDe.scala and https://github.com/apache/spark/blob/master/R/pkg/R/types.R This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514472780 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108074/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514472777 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r306621651
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
##
@@ -452,6 +452,45 @@ class StringExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
// scalastyle:on
}
+ test("overlay for byte array") {
+val input = Literal(Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9))
+checkEvaluation(new Overlay(input, Literal(Array[Byte](-1)),
+ Literal.create(6, IntegerType)), Array[Byte](1, 2, 3, 4, 5, -1, 7, 8, 9))
Review comment:
I have added tests for negative and zero. I don't understand this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514472372 Let's file a JIRA for each. 1. SQL parser side: `CalendarInterval` DDL type string should be supported in SparkSQL - we need a change like this https://github.com/apache/spark/pull/25189/commits/e7e2f5bc2c88958fb672d133b55e8c01b7298f40#diff-9847f5cef7cf7fbc5830fbc6b779ee10R1929 2. Python side: `CalendarInterval` type should be supported in PySpark - see https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L45-L94 We need to define the interval type class and should support presumably time delta in Python - need to check if this works with Python serde. Let's say how it works with Python UDF. See https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala 3. R side: `CalendarInterval` type should be supported in SparkR - R side does not have type class instances IIRC. So we wouldn't need to define the class - However, we should still fix R serde side as well. Presumably `difftime` instances might have to be matched with interval type in SQL sides. See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/SerDe.scala and https://github.com/apache/spark/blob/master/R/pkg/R/types.R This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514472768 **[Test build #108074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108074/testReport)** for PR 25172 at commit [`271e5f6`](https://github.com/apache/spark/commit/271e5f677c7d11e5621a79447083be9030a83334). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514472777 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
SparkQA removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514472475 **[Test build #108074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108074/testReport)** for PR 25172 at commit [`271e5f6`](https://github.com/apache/spark/commit/271e5f677c7d11e5621a79447083be9030a83334). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514472780 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108074/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array
SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array URL: https://github.com/apache/spark/pull/25172#issuecomment-514472475 **[Test build #108074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108074/testReport)** for PR 25172 at commit [`271e5f6`](https://github.com/apache/spark/commit/271e5f677c7d11e5621a79447083be9030a83334). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
HyukjinKwon commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514472372 Let's file a JIRA for each. 1. SQL parser side: we need a change like this https://github.com/apache/spark/pull/25189/commits/e7e2f5bc2c88958fb672d133b55e8c01b7298f40#diff-9847f5cef7cf7fbc5830fbc6b779ee10R1929 2. Python side: - see https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L45-L94 We need to define the interval type class and should support presumably time delta in Python - need to check if this works with Python serde. Let's say how it works with Python UDF. See https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala 3. R side: - R side does not have type class instances IIRC. So we wouldn't need to define the class - However, we should still fix R serde side as well. Presumably `difftime` instances might have to be matched with interval type in SQL sides. See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/SerDe.scala and https://github.com/apache/spark/blob/master/R/pkg/R/types.R This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
HyukjinKwon edited a comment on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514472372 Let's file a JIRA for each. 1. SQL parser side: we need a change like this https://github.com/apache/spark/pull/25189/commits/e7e2f5bc2c88958fb672d133b55e8c01b7298f40#diff-9847f5cef7cf7fbc5830fbc6b779ee10R1929 2. Python side: - see https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L45-L94 We need to define the interval type class and should support presumably time delta in Python - need to check if this works with Python serde. Let's say how it works with Python UDF. See https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala 3. R side: - R side does not have type class instances IIRC. So we wouldn't need to define the class - However, we should still fix R serde side as well. Presumably `difftime` instances might have to be matched with interval type in SQL sides. See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/SerDe.scala and https://github.com/apache/spark/blob/master/R/pkg/R/types.R This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25232: [SPARK-28475][CORE] Add regex MetricFilter to GraphiteSink
HeartSaVioR commented on issue #25232: [SPARK-28475][CORE] Add regex MetricFilter to GraphiteSink URL: https://github.com/apache/spark/pull/25232#issuecomment-514470692 Thanks for the contribution. I'm seeing the benefit on the change as I introduced similar thing (not for Spark though), but it also ended up with advanced (complicated) supports: accepting multiple patterns, with mode either "whitelist" or "blacklist" (not both, of course). Hopefully I'm not seeing too many Dropwizard metrics in Spark so this might be good enough to start. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yiheng commented on a change in pull request #25202: [SPARK-28289][SQL][PYTHON][TESTS] Convert and port 'union.sql' into UDF test base
yiheng commented on a change in pull request #25202: [SPARK-28289][SQL][PYTHON][TESTS] Convert and port 'union.sql' into UDF test base URL: https://github.com/apache/spark/pull/25202#discussion_r306603202 ## File path: sql/core/src/test/resources/sql-tests/results/udf/udf-union.sql.out ## @@ -0,0 +1,166 @@ +-- Automatically generated by SQLQueryTestSuite +-- Number of queries: 16 + + +-- !query 0 +CREATE OR REPLACE TEMPORARY VIEW t1 AS VALUES (1, 'a'), (2, 'b') tbl(c1, c2) +-- !query 0 schema +struct<> +-- !query 0 output + + + +-- !query 1 +CREATE OR REPLACE TEMPORARY VIEW t2 AS VALUES (1.0, 1), (2.0, 4) tbl(c1, c2) +-- !query 1 schema +struct<> +-- !query 1 output + + + +-- !query 2 +SELECT udf(c1) as c1, udf(c2) as c2 +FROM (SELECT udf(c1) as c1, udf(c2) as c2 FROM t1 +UNION ALL +SELECT udf(c1) as c1, udf(c2) as c2 FROM t1) +-- !query 2 schema +struct +-- !query 2 output +1 a +1 a +2 b +2 b + + +-- !query 3 +SELECT udf(c1) as c1, udf(c2) as c2 +FROM (SELECT udf(c1) as c1, udf(c2) as c2 FROM t1 +UNION ALL +SELECT udf(c1) as c1, udf(c2) as c2 FROM t2 +UNION ALL +SELECT udf(c1) as c1, udf(c2) as c2 FROM t2) +-- !query 3 schema +struct +-- !query 3 output +1 1 +1 1 +1 a +2 4 +2 4 +2 b + + +-- !query 4 +SELECT udf(a) as a, udf(udf(a)) as a1 +FROM (SELECT udf(0) a, udf(0) b + UNION ALL + SELECT udf(SUM(1)) a, udf(CAST(0 AS BIGINT)) b + UNION ALL SELECT udf(0) a, udf(0) b) T +-- !query 4 schema +struct +-- !query 4 output +0 0 +0 0 +1 1 Review comment: As your [review comment](https://github.com/apache/spark/pull/25202#discussion_r305656980), I add a udf(udf(a)) to the select fields. So now there're two columns in the output instead of one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514470113 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13180/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469510 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469515 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108073/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514467433 **[Test build #108073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108073/testReport)** for PR 24879 at commit [`9392dad`](https://github.com/apache/spark/commit/9392dad701a8ec8cda1c3bd2dd55be0e8979da99). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469510 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469515 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108073/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469460 **[Test build #108073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108073/testReport)** for PR 24879 at commit [`9392dad`](https://github.com/apache/spark/commit/9392dad701a8ec8cda1c3bd2dd55be0e8979da99). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469262 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13179/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469257 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469257 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469262 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13179/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514469187 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13179/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25232: [SPARK-28475][CORE] Add regex MetricFilter to GraphiteSink
HeartSaVioR commented on a change in pull request #25232: [SPARK-28475][CORE]
Add regex MetricFilter to GraphiteSink
URL: https://github.com/apache/spark/pull/25232#discussion_r306618126
##
File path:
core/src/test/scala/org/apache/spark/metrics/sink/GraphiteSinkSuite.scala
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.metrics.sink
+
+import java.util.Properties
+
+import scala.collection.JavaConverters._
+
+import com.codahale.metrics._
+
+import org.apache.spark.{SecurityManager, SparkConf, SparkFunSuite}
+
+class GraphiteSinkSuite extends SparkFunSuite {
+
+ test("GraphiteSink with default MetricsFilter") {
+val props = new Properties
+props.put("host", "127.0.0.1")
+props.put("port", "54321")
+val registry = new MetricRegistry
+val securityMgr = new SecurityManager(new SparkConf(false))
+
+val sink = new GraphiteSink(props, registry, securityMgr)
+
+val gauge = new Gauge[Double] {
+ override def getValue: Double = 1.23
+}
+sink.registry.register("gauge", gauge)
+sink.registry.register("anothergauge", gauge)
+sink.registry.register("streaminggauge", gauge)
+
+val metricKeys = sink.registry.getGauges(sink.filter).keySet.asScala
+
+assert(metricKeys.equals(Set("gauge", "anothergauge", "streaminggauge")),
+ "Should contain all metrics registered")
+ }
+
+ test("GraphiteSink with regex MetricsFilter") {
+val props = new Properties
+props.put("host", "127.0.0.1")
+props.put("port", "54321")
+props.put("regex", "streaming")
Review comment:
It would be ideal to have regex pattern on test case, as we support regex.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] priyankagargnitk commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package
priyankagargnitk commented on issue #25022: [SPARK-24695][SQL] Move `CalendarInterval` to org.apache.spark.sql.types package URL: https://github.com/apache/spark/pull/25022#issuecomment-514468532 @HyukjinKwon: What action is needed to be done at my side for SQL parser / R / Python changes? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514467914 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13178/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514467909 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514467909 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514467914 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13178/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514467868 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13178/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514467433 **[Test build #108073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108073/testReport)** for PR 24879 at commit [`9392dad`](https://github.com/apache/spark/commit/9392dad701a8ec8cda1c3bd2dd55be0e8979da99). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514466621 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13179/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #25237: [SPARK-28489][SS] Fix a bug that KafkaOffsetRangeCalculator.getRanges may drop offsets
HeartSaVioR edited a comment on issue #25237: [SPARK-28489][SS] Fix a bug that KafkaOffsetRangeCalculator.getRanges may drop offsets URL: https://github.com/apache/spark/pull/25237#issuecomment-514465712 Hmm... I'm now reading comment on getRanges. I'm not sure `numPartitions` is actually `minPartitions` (so some typos on javadoc - maybe better to fix them here), but if they're same, below comment would say it's OK to have more or less partitions than `minPartitions`. > The number of Spark tasks will be *approximately* `numPartitions`. It can be less or more depending on rounding errors or Kafka partitions that didn't receive any new data. https://github.com/apache/spark/blob/d67b98ea016e9b714bef68feaac108edd08159c9/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala#L32-L46 Please ignore my review comments if the javadoc meant it. Looks great. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #25237: [SPARK-28489][SS] Fix a bug that KafkaOffsetRangeCalculator.getRanges may drop offsets
HeartSaVioR edited a comment on issue #25237: [SPARK-28489][SS] Fix a bug that KafkaOffsetRangeCalculator.getRanges may drop offsets URL: https://github.com/apache/spark/pull/25237#issuecomment-514465712 Hmm... I'm now reading comment on getRanges. I'm not sure `numPartitions` is actually `minPartitions` (so some typos on javadoc - maybe better to fix them here), but if they're same, below comment would say the method doesn't guarantee returning count of partitions is not necessary to be equal or greater than `minPartitions`. > The number of Spark tasks will be *approximately* `numPartitions`. It can be less or more depending on rounding errors or Kafka partitions that didn't receive any new data. https://github.com/apache/spark/blob/d67b98ea016e9b714bef68feaac108edd08159c9/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala#L32-L46 Please ignore my review comments if the javadoc meant it. Looks great. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25237: [SPARK-28489][SS] Fix a bug that KafkaOffsetRangeCalculator.getRanges may drop offsets
HeartSaVioR commented on issue #25237: [SPARK-28489][SS] Fix a bug that KafkaOffsetRangeCalculator.getRanges may drop offsets URL: https://github.com/apache/spark/pull/25237#issuecomment-514465712 Hmm... I'm now reading comment on getRanges. I'm not sure `numPartitions` is actually `minPartitions` (so some typos on javadoc), but if they're same, below comment would say it's OK to have more or less partitions than `minPartitions`. > The number of Spark tasks will be *approximately* `numPartitions`. It can be less or more depending on rounding errors or Kafka partitions that didn't receive any new data. https://github.com/apache/spark/blob/d67b98ea016e9b714bef68feaac108edd08159c9/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala#L32-L46 Please ignore my review comments if the javadoc meant it. Looks great. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514463445 **[Test build #108072 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108072/testReport)** for PR 24879 at commit [`45c8bc5`](https://github.com/apache/spark/commit/45c8bc5b4a80a7b182186e114bab2fb7affb98ed). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514465569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108072/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514465569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108072/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
