[GitHub] [spark] SparkQA commented on pull request #30066: [SPARK-XXX][INFRA] Use pre-built image at GitHub Action SparkR job
SparkQA commented on pull request #30066: URL: https://github.com/apache/spark/pull/30066#issuecomment-710756510 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34543/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30073: [SPARK-33170][SQL] Add SQL config to control fast-fail behavior in FileFormatWriter
AmplabJenkins removed a comment on pull request #30073: URL: https://github.com/apache/spark/pull/30073#issuecomment-710756428 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30073: [SPARK-33170][SQL] Add SQL config to control fast-fail behavior in FileFormatWriter
AmplabJenkins commented on pull request #30073: URL: https://github.com/apache/spark/pull/30073#issuecomment-710756428 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30073: [SPARK-33170][SQL] Add SQL config to control fast-fail behavior in FileFormatWriter
SparkQA commented on pull request #30073: URL: https://github.com/apache/spark/pull/30073#issuecomment-710756237 **[Test build #129939 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129939/testReport)** for PR 30073 at commit [`434666c`](https://github.com/apache/spark/commit/434666ce612a8d8e61335fcc0883dd3b965c143c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30073: [SPARK-33170][SQL] Add SQL config to control fast-fail behavior in FileFormatWriter
SparkQA commented on pull request #30073: URL: https://github.com/apache/spark/pull/30073#issuecomment-710756039 **[Test build #129925 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129925/testReport)** for PR 30073 at commit [`5a1c030`](https://github.com/apache/spark/commit/5a1c03076db37a22426e6e69373f7d6566da021a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30073: [SPARK-33170][SQL] Add SQL config to control fast-fail behavior in FileFormatWriter
SparkQA removed a comment on pull request #30073: URL: https://github.com/apache/spark/pull/30073#issuecomment-710714629 **[Test build #129925 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129925/testReport)** for PR 30073 at commit [`5a1c030`](https://github.com/apache/spark/commit/5a1c03076db37a22426e6e69373f7d6566da021a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #30073: [SPARK-33170][SQL] Add SQL config to control fast-fail behavior in FileFormatWriter
viirya commented on a change in pull request #30073:
URL: https://github.com/apache/spark/pull/30073#discussion_r506811761
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2052,6 +2052,20 @@ object SQLConf {
.stringConf
.createWithDefault("")
+ val FASTFAIL_ON_FILEFORMAT_OUTPUT =
+buildConf("spark.sql.execution.fastFailOnFileFormatOutput")
+ .internal()
+ .doc("Whether to fast fail task execution when writing output to
FileFormat datasource. " +
+"If this is enabled, in `FileFormatWriter` we will catch
`FileAlreadyExistsException` " +
+"and fast fail output task without further task retry. Only enabling
this if you know " +
+"the `FileAlreadyExistsException` of the output task is unrecoverable,
i.e., further " +
+"task attempts won't be able to success. If the
`FileAlreadyExistsException` might be " +
+"recoverable, you should keep this as disabled and let Spark to retry
output tasks. " +
+"This is disabled by default.")
+ .version("3.1.0")
Review comment:
Ok. I changed it to `3.0.2`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AmplabJenkins removed a comment on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710754644 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34541/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AmplabJenkins removed a comment on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710754639 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
SparkQA commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710754632 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34541/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AmplabJenkins commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710754639 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30073: [SPARK-33170][SQL] Add SQL config to control fast-fail behavior in FileFormatWriter
dongjoon-hyun commented on a change in pull request #30073:
URL: https://github.com/apache/spark/pull/30073#discussion_r506803363
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2052,6 +2052,20 @@ object SQLConf {
.stringConf
.createWithDefault("")
+ val FASTFAIL_ON_FILEFORMAT_OUTPUT =
+buildConf("spark.sql.execution.fastFailOnFileFormatOutput")
+ .internal()
+ .doc("Whether to fast fail task execution when writing output to
FileFormat datasource. " +
+"If this is enabled, in `FileFormatWriter` we will catch
`FileAlreadyExistsException` " +
+"and fast fail output task without further task retry. Only enabling
this if you know " +
+"the `FileAlreadyExistsException` of the output task is unrecoverable,
i.e., further " +
+"task attempts won't be able to success. If the
`FileAlreadyExistsException` might be " +
+"recoverable, you should keep this as disabled and let Spark to retry
output tasks. " +
+"This is disabled by default.")
+ .version("3.1.0")
Review comment:
Then, I guess `3.0.2` might be better here. Given that 3.1.0 is
rescheduled, 3.0.2 will arrive before or at the same time because it's easier
than 3.1.0 release.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30057: [SPARK-32838][SQL]Check DataSource insert command path with actual path
AmplabJenkins removed a comment on pull request #30057: URL: https://github.com/apache/spark/pull/30057#issuecomment-710752871 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30057: [SPARK-32838][SQL]Check DataSource insert command path with actual path
SparkQA commented on pull request #30057: URL: https://github.com/apache/spark/pull/30057#issuecomment-710752865 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34540/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30057: [SPARK-32838][SQL]Check DataSource insert command path with actual path
AmplabJenkins commented on pull request #30057: URL: https://github.com/apache/spark/pull/30057#issuecomment-710752871 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
SparkQA commented on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710752631 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34542/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins commented on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710752639 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins removed a comment on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710752639 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30074: [MINOR][SS][DOCS] Update Structured Streaming guide doc and update code typo
dongjoon-hyun commented on pull request #30074: URL: https://github.com/apache/spark/pull/30074#issuecomment-710752231 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30066: [SPARK-XXX][INFRA] Use pre-built image at GitHub Action SparkR job
SparkQA commented on pull request #30066: URL: https://github.com/apache/spark/pull/30066#issuecomment-710751167 **[Test build #129938 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129938/testReport)** for PR 30066 at commit [`1d7d1bc`](https://github.com/apache/spark/commit/1d7d1bc4e8609f4bb18dbac0576a7ee3d8cc8f0a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
AmplabJenkins removed a comment on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710751015 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
AmplabJenkins commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710751015 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
SparkQA commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710751008 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34539/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
SparkQA commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710751043 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34541/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
SparkQA commented on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710750473 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34542/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30057: [SPARK-32838][SQL]Check DataSource insert command path with actual path
SparkQA commented on pull request #30057: URL: https://github.com/apache/spark/pull/30057#issuecomment-710750269 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34540/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
AmplabJenkins removed a comment on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710748848 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129932/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
SparkQA removed a comment on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710738272 **[Test build #129932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129932/testReport)** for PR 30027 at commit [`d5947e1`](https://github.com/apache/spark/commit/d5947e1dea2ecb28bab7e6a5abb1eda80ff2b5ef). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
AmplabJenkins removed a comment on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710748841 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
AmplabJenkins commented on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710748841 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
SparkQA commented on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710748828 **[Test build #129932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129932/testReport)** for PR 30027 at commit [`d5947e1`](https://github.com/apache/spark/commit/d5947e1dea2ecb28bab7e6a5abb1eda80ff2b5ef). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
AmplabJenkins removed a comment on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710748253 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129931/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
SparkQA commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710748357 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34539/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AmplabJenkins removed a comment on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710748246 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
AmplabJenkins commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710748251 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AmplabJenkins commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710748246 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
AmplabJenkins removed a comment on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710748251 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
SparkQA removed a comment on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710729510 **[Test build #129931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129931/testReport)** for PR 30076 at commit [`e5af8e1`](https://github.com/apache/spark/commit/e5af8e12112533150321a292a261489b3523e23a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
SparkQA commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710748237 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34538/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
SparkQA commented on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710748050 **[Test build #129937 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129937/testReport)** for PR 30075 at commit [`fdb1aed`](https://github.com/apache/spark/commit/fdb1aede9f7f444d65b1e470901cba2aec7efbb0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
SparkQA commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710748095 **[Test build #129931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129931/testReport)** for PR 30076 at commit [`e5af8e1`](https://github.com/apache/spark/commit/e5af8e12112533150321a292a261489b3523e23a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
AmplabJenkins removed a comment on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710747307 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
SparkQA commented on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710747303 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34537/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
AmplabJenkins commented on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710747307 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client
AmplabJenkins removed a comment on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-710746711 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129928/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client
AmplabJenkins commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-710746708 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client
AmplabJenkins removed a comment on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-710746708 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client
SparkQA removed a comment on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-710722789 **[Test build #129928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129928/testReport)** for PR 29843 at commit [`507270b`](https://github.com/apache/spark/commit/507270bbd9d2ae24c6d4750e3931d1738d8b0141). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client
SparkQA commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-710746490 **[Test build #129928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129928/testReport)** for PR 29843 at commit [`507270b`](https://github.com/apache/spark/commit/507270bbd9d2ae24c6d4750e3931d1738d8b0141). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
SparkQA commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710745778 **[Test build #129936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129936/testReport)** for PR 29881 at commit [`6e67c7b`](https://github.com/apache/spark/commit/6e67c7b9fcf9f5c66cb899f5f9000a5598492831). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #30072: [SPARK-33173][CORE][TESTS] Use `eventually` to check `numOnTaskFailed` in PluginContainerSuite
dongjoon-hyun closed pull request #30072: URL: https://github.com/apache/spark/pull/30072 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30072: [SPARK-33173][CORE][TESTS] Use `eventually` to check `numOnTaskFailed` in PluginContainerSuite
dongjoon-hyun commented on pull request #30072: URL: https://github.com/apache/spark/pull/30072#issuecomment-710745499 Thank you, @viirya , @HyukjinKwon , @mridulm . Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
SparkQA commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710745392 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34538/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30057: [SPARK-32838][SQL]Check DataSource insert command path with actual path
SparkQA commented on pull request #30057: URL: https://github.com/apache/spark/pull/30057#issuecomment-710744791 **[Test build #129935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129935/testReport)** for PR 30057 at commit [`9d9ab88`](https://github.com/apache/spark/commit/9d9ab8839ae68d77a48d610983171a3e9126afc1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
SparkQA commented on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710743730 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34537/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins removed a comment on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710743325 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129929/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins commented on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710743322 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins removed a comment on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710743322 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
SparkQA removed a comment on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710724052 **[Test build #129929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129929/testReport)** for PR 30075 at commit [`fa072b1`](https://github.com/apache/spark/commit/fa072b1322e6e5b06ab83475760a3bc0d7e74ec6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
SparkQA commented on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710743180 **[Test build #129929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129929/testReport)** for PR 30075 at commit [`fa072b1`](https://github.com/apache/spark/commit/fa072b1322e6e5b06ab83475760a3bc0d7e74ec6). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class AuthRpcHandler extends AbstractAuthRpcHandler ` * `public class SaslRpcHandler extends AbstractAuthRpcHandler ` * `public abstract class AbstractAuthRpcHandler extends RpcHandler ` * ` logDebug(s\" + cloning instance of REPL class $capturingClassName\")` * ` // class InnerFoo ` * ` // found intra class call to $iw.$anonfun$closure$1(L$iw;I)Lscala/collection/immutable/IndexedSeq;` * ` // found intra class call to $iw$InnerFoo$1.$anonfun$innerClosure$1(L$iw$InnerFoo$1;I)Lscala/collection/immutable/IndexedSeq;` * ` // found intra class call to $iw$InnerFoo$1.$anonfun$innerClosure$2(L$iw$InnerFoo$1;I)Ljava/lang/String;` * ` // found intra class call to $iw$InnerFoo$1.$anonfun$innerClosure$2(L$iw$InnerFoo$1;I)Ljava/lang/String;` * ` //(class java.lang.Object,Set())` * ` //(class $iw,Set(topLevelValue))` * `// starting closure (in class T)` * `// we need to track calls from \"inner closure\" to outer classes relative to it (class T, A, B)` * `logTrace(s\"found intra class call to $ownerExternalName.$name$desc\")` * `logDebug(s\"found inner class $ownerExternalName\")` * `public class SparkAvroKeyOutputFormat extends AvroKeyOutputFormat ` * ` static class SparkRecordWriterFactory extends RecordWriterFactory ` * `case class UnresolvedHaving(` * `case class First(child: Expression, ignoreNulls: Boolean)` * `case class Last(child: Expression, ignoreNulls: Boolean)` * `case class Size(child: Expression, legacySizeOfNull: Boolean)` * ` class TimestampParser(fastDateFormat: FastDateFormat) ` * `class SQLExecutionRDD(` * `case class ArrowEvalPython(` * `case class BatchEvalPython(` * `trait CreateHiveTableAsSelectBase extends DataWritingCommand ` * `case class CreateHiveTableAsSelectCommand(` * `case class OptimizedCreateHiveTableAsSelectCommand(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
SparkQA commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710742615 **[Test build #129934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129934/testReport)** for PR 30076 at commit [`ee16690`](https://github.com/apache/spark/commit/ee166901bbaa3500a9fa8704f2a5598a8ce4b6d7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on a change in pull request #30073: [SPARK-33170][SQL] Add SQL config to control fast-fail behavior in FileFormatWriter
zsxwing commented on a change in pull request #30073:
URL: https://github.com/apache/spark/pull/30073#discussion_r506790079
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2052,6 +2052,20 @@ object SQLConf {
.stringConf
.createWithDefault("")
+ val FASTFAIL_ON_FILEFORMAT_OUTPUT =
+buildConf("spark.sql.execution.fastFailOnFileFormatOutput")
+ .internal()
+ .doc("Whether to fast fail task execution when writing output to
FileFormat datasource. " +
+"If this is enabled, in `FileFormatWriter` we will catch
`FileAlreadyExistsException` " +
+"and fast fail output task without further task retry. Only enabling
this if you know " +
+"the `FileAlreadyExistsException` of the output task is unrecoverable,
i.e., further " +
+"task attempts won't be able to success. If the
`FileAlreadyExistsException` might be " +
+"recoverable, you should keep this as disabled and let Spark to retry
output tasks. " +
+"This is disabled by default.")
+ .version("3.1.0")
Review comment:
I think it's better to merge to branch-3.0 as well as currently user
don't have a flag for this in 3.0.x.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AmplabJenkins removed a comment on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710739619 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129933/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
SparkQA removed a comment on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710739403 **[Test build #129933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129933/testReport)** for PR 29881 at commit [`d631a75`](https://github.com/apache/spark/commit/d631a758e491fb7f5e7ffcb66f74cdef19028be8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30077: [SPARK-33131][SQL][3.0] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins removed a comment on pull request #30077: URL: https://github.com/apache/spark/pull/30077#issuecomment-710739398 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34535/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AmplabJenkins removed a comment on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710739618 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
SparkQA commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710739616 **[Test build #129933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129933/testReport)** for PR 29881 at commit [`d631a75`](https://github.com/apache/spark/commit/d631a758e491fb7f5e7ffcb66f74cdef19028be8). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30077: [SPARK-33131][SQL][3.0] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins removed a comment on pull request #30077: URL: https://github.com/apache/spark/pull/30077#issuecomment-710739395 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AmplabJenkins commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710739618 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
SparkQA commented on pull request #29881: URL: https://github.com/apache/spark/pull/29881#issuecomment-710739403 **[Test build #129933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129933/testReport)** for PR 29881 at commit [`d631a75`](https://github.com/apache/spark/commit/d631a758e491fb7f5e7ffcb66f74cdef19028be8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30077: [SPARK-33131][SQL][3.0] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins commented on pull request #30077: URL: https://github.com/apache/spark/pull/30077#issuecomment-710739395 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30077: [SPARK-33131][SQL][3.0] Fix grouping sets with having clause can not resolve qualified col name
SparkQA commented on pull request #30077: URL: https://github.com/apache/spark/pull/30077#issuecomment-710739390 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34535/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
AmplabJenkins removed a comment on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710738846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
SparkQA commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710738844 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34536/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
AmplabJenkins commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710738846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AngersZh commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r506788022
##
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
.doc(s"""
| Location of the jars that should be used to instantiate the
HiveMetastoreClient.
- | This property can be one of three options: "
+ | This property can be one of four options: "
| 1. "builtin"
| Use Hive ${builtinHiveVersion}, which is bundled with the Spark
assembly when
| -Phive is enabled. When this option is chosen,
| spark.sql.hive.metastore.version must be either
| ${builtinHiveVersion} or not defined.
| 2. "maven"
| Use Hive jars of specified version downloaded from Maven
repositories.
- | 3. A classpath in the standard format for both Hive and Hadoop.
+ | 3. "path"
+ | Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in
comma separated format
+ | support both local or remote paths, it should always be fully
qualified URL to indicate
+ | other file systems.
+ | 4. A classpath in the standard format for both Hive and Hadoop.
""".stripMargin)
.version("1.4.0")
.stringConf
.createWithDefault("builtin")
+ val HIVE_METASTORE_JARS_PATH =
buildStaticConf("spark.sql.hive.metastore.jars.path")
+.doc(s"Comma separated fully qualified URL of Hive jars, support both
local and remote paths," +
+ s"Such as: " +
+ s" 1. file://path/to/jar/xxx.jar" +
+ s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+ s" 3. /path/to/jar/ (path without URI scheme follow conf
`fs.defaultFS`'s URI schema)" +
+ s" 4. [http/https/ftp]://path/to/jar/xxx.jar" +
+ s"For URI, we can't support path wildcard, but for other URL support
nested path wildcard," +
Review comment:
> can we be more specific? e.g. `http/https/ftp` doesn't support
wildcard.
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
SparkQA commented on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710738272 **[Test build #129932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129932/testReport)** for PR 30027 at commit [`d5947e1`](https://github.com/apache/spark/commit/d5947e1dea2ecb28bab7e6a5abb1eda80ff2b5ef). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30072: [SPARK-33173][CORE][TESTS] Use `eventually` to check `numOnTaskFailed` in PluginContainerSuite
AmplabJenkins removed a comment on pull request #30072: URL: https://github.com/apache/spark/pull/30072#issuecomment-710738218 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AngersZh commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r506787897
##
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
.doc(s"""
| Location of the jars that should be used to instantiate the
HiveMetastoreClient.
- | This property can be one of three options: "
+ | This property can be one of four options: "
| 1. "builtin"
| Use Hive ${builtinHiveVersion}, which is bundled with the Spark
assembly when
| -Phive is enabled. When this option is chosen,
| spark.sql.hive.metastore.version must be either
| ${builtinHiveVersion} or not defined.
| 2. "maven"
| Use Hive jars of specified version downloaded from Maven
repositories.
- | 3. A classpath in the standard format for both Hive and Hadoop.
+ | 3. "path"
+ | Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in
comma separated format
Review comment:
> Add a period at the end?
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30072: [SPARK-33173][CORE][TESTS] Use `eventually` to check `numOnTaskFailed` in PluginContainerSuite
AmplabJenkins commented on pull request #30072: URL: https://github.com/apache/spark/pull/30072#issuecomment-710738218 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location
AngersZh commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r506787887
##
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##
@@ -396,23 +440,43 @@ private[spark] object HiveUtils extends Logging {
config = configurations,
barrierPrefixes = hiveMetastoreBarrierPrefixes,
sharedPrefixes = hiveMetastoreSharedPrefixes)
+} else if (hiveMetastoreJars == "path") {
+ // Convert to files and expand any directories.
+ val jars =
+HiveUtils.hiveMetastoreJarsPath(sqlConf)
+ .flatMap {
+case path if path.contains("\\") && Utils.isWindows =>
+ addLocalHiveJars(new File(path))
+case path =>
+ DataSource.checkAndGlobPathIfNecessary(
+pathStrings = Seq(path),
+hadoopConf = hadoopConf,
+checkEmptyGlobPath = true,
+checkFilesExist = false,
+enableGlobbing = true
+ ).map(_.toUri.toURL)
+ }
+
+ logInfo(
+s"Initializing HiveMetastoreConnection version $hiveMetastoreVersion "
+
+ s"using path: ${jars.mkString(";")}")
+ new IsolatedClientLoader(
+version = metaVersion,
+sparkConf = conf,
+hadoopConf = hadoopConf,
+execJars = jars.toSeq,
+config = configurations,
+isolationOn = true,
+barrierPrefixes = hiveMetastoreBarrierPrefixes,
+sharedPrefixes = hiveMetastoreSharedPrefixes)
} else {
// Convert to files and expand any directories.
val jars =
hiveMetastoreJars
.split(File.pathSeparator)
- .flatMap {
- case path if new File(path).getName == "*" =>
-val files = new File(path).getParentFile.listFiles()
-if (files == null) {
- logWarning(s"Hive jar path '$path' does not exist.")
- Nil
-} else {
-
files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
-}
- case path =>
-new File(path) :: Nil
-}
+ .flatMap { path =>
+addLocalHiveJars(new File(path))
+ }
.map(_.toURI.toURL)
Review comment:
> This line can be removed, as `addLocalHiveJars` already returns URL
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30072: [SPARK-33173][CORE][TESTS] Use `eventually` to check `numOnTaskFailed` in PluginContainerSuite
SparkQA removed a comment on pull request #30072: URL: https://github.com/apache/spark/pull/30072#issuecomment-710711396 **[Test build #129923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129923/testReport)** for PR 30072 at commit [`c5f9c1e`](https://github.com/apache/spark/commit/c5f9c1ecb6cd7e39c43d40be23189c03b24a7f4f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30072: [SPARK-33173][CORE][TESTS] Use `eventually` to check `numOnTaskFailed` in PluginContainerSuite
SparkQA commented on pull request #30072: URL: https://github.com/apache/spark/pull/30072#issuecomment-710737954 **[Test build #129923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129923/testReport)** for PR 30072 at commit [`c5f9c1e`](https://github.com/apache/spark/commit/c5f9c1ecb6cd7e39c43d40be23189c03b24a7f4f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
AmplabJenkins removed a comment on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710701405 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129901/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory
AngersZh commented on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710737270 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30077: [SPARK-33131][SQL][3.0] Fix grouping sets with having clause can not resolve qualified col name
SparkQA commented on pull request #30077: URL: https://github.com/apache/spark/pull/30077#issuecomment-710736697 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34535/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
SparkQA commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710735984 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34536/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #29977: [SPARK-33088][CORE] Enhance ExecutorPlugin API to include callbacks on task start and end events
mridulm commented on a change in pull request #29977:
URL: https://github.com/apache/spark/pull/29977#discussion_r506786690
##
File path:
core/src/test/scala/org/apache/spark/internal/plugin/PluginContainerSuite.scala
##
@@ -129,6 +129,38 @@ class PluginContainerSuite extends SparkFunSuite with
BeforeAndAfterEach with Lo
assert(TestSparkPlugin.driverPlugin != null)
}
+ test("SPARK-33088: executor tasks trigger plugin calls") {
+val conf = new SparkConf()
+ .setAppName(getClass().getName())
+ .set(SparkLauncher.SPARK_MASTER, "local[1]")
+ .set(PLUGINS, Seq(classOf[TestSparkPlugin].getName()))
+
+sc = new SparkContext(conf)
+sc.parallelize(1 to 10, 2).count()
+
+assert(TestSparkPlugin.executorPlugin.numOnTaskStart == 2)
+assert(TestSparkPlugin.executorPlugin.numOnTaskSucceeded == 2)
+assert(TestSparkPlugin.executorPlugin.numOnTaskFailed == 0)
+ }
+
+ test("SPARK-33088: executor failed tasks trigger plugin calls") {
+val conf = new SparkConf()
+ .setAppName(getClass().getName())
+ .set(SparkLauncher.SPARK_MASTER, "local[1]")
+ .set(PLUGINS, Seq(classOf[TestSparkPlugin].getName()))
+
+sc = new SparkContext(conf)
+try {
+ sc.parallelize(1 to 10, 2).foreach(i => throw new RuntimeException)
+} catch {
+ case t: Throwable => // ignore exception
+}
+
+assert(TestSparkPlugin.executorPlugin.numOnTaskStart == 2)
+assert(TestSparkPlugin.executorPlugin.numOnTaskSucceeded == 0)
+assert(TestSparkPlugin.executorPlugin.numOnTaskFailed == 2)
Review comment:
Thanks @dongjoon-hyun
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28026: [SPARK-31257][SQL] Unify create table syntax
AmplabJenkins removed a comment on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-710734894 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129924/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28026: [SPARK-31257][SQL] Unify create table syntax
AmplabJenkins removed a comment on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-710734892 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28026: [SPARK-31257][SQL] Unify create table syntax
SparkQA removed a comment on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-710711413 **[Test build #129924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129924/testReport)** for PR 28026 at commit [`e81fa5c`](https://github.com/apache/spark/commit/e81fa5cfad3d33647569de5011f0b66b6ef9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28026: [SPARK-31257][SQL] Unify create table syntax
AmplabJenkins commented on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-710734892 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client
SparkQA commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-710734640 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34533/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client
AmplabJenkins removed a comment on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-710734648 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28026: [SPARK-31257][SQL] Unify create table syntax
SparkQA commented on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-710734788 **[Test build #129924 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129924/testReport)** for PR 28026 at commit [`e81fa5c`](https://github.com/apache/spark/commit/e81fa5cfad3d33647569de5011f0b66b6ef9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SerdeInfo(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29843: [SPARK-29250][BUILD] Upgrade to Hadoop 3.2.1 and move to shaded client
AmplabJenkins commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-710734648 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #30076: [SPARK-32862][SS] Left semi stream-stream join
viirya commented on pull request #30076: URL: https://github.com/apache/spark/pull/30076#issuecomment-710731815 cc @HeartSaVioR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins commented on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710731267 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
AmplabJenkins removed a comment on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710731267 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30075: [SPARK-33131][SQL][2.4] Fix grouping sets with having clause can not resolve qualified col name
SparkQA commented on pull request #30075: URL: https://github.com/apache/spark/pull/30075#issuecomment-710731259 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34534/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
