[GitHub] [spark] tgravescs commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window
tgravescs commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607842053 merging to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
SparkQA commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code URL: https://github.com/apache/spark/pull/28081#issuecomment-607842318 **[Test build #120717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120717/testReport)** for PR 28081 at commit [`eb27c3a`](https://github.com/apache/spark/commit/eb27c3acfcfe3b5e4ac86c6720bc02a287c4f584). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
SparkQA removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code URL: https://github.com/apache/spark/pull/28081#issuecomment-607711320 **[Test build #120717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120717/testReport)** for PR 28081 at commit [`eb27c3a`](https://github.com/apache/spark/commit/eb27c3acfcfe3b5e4ac86c6720bc02a287c4f584). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] asfgit closed pull request #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window
asfgit closed pull request #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code URL: https://github.com/apache/spark/pull/28081#issuecomment-607843401 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120717/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code URL: https://github.com/apache/spark/pull/28081#issuecomment-607843401 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120717/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code URL: https://github.com/apache/spark/pull/28081#issuecomment-607843390 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code URL: https://github.com/apache/spark/pull/28081#issuecomment-607843390 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
tgravescs commented on a change in pull request #27207: [SPARK-18886][CORE]
Make Locality wait time measure resource under utilization due to delay
scheduling.
URL: https://github.com/apache/spark/pull/27207#discussion_r402310761
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -543,6 +543,11 @@ package object config {
.version("1.2.0")
.fallbackConf(DYN_ALLOCATION_SCHEDULER_BACKLOG_TIMEOUT)
+ private[spark] val LEGACY_LOCALITY_WAIT_RESET =
+ConfigBuilder("spark.locality.wait.legacyResetOnTaskLaunch")
Review comment:
yes please add the .doc and .version - it would be nice to get into 3.0.0.
I don't know that I agree with the internal tag. Maybe its being used
differently then I thought. Internal to me means it's not meant for end user,
this one could be if users find issues with the new algorithm. I'm definitely
fine with not documenting it and it seems it has that side affect so maybe not
a big deal.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26674: [SPARK-30059][CORE]Stop AsyncEventQueue when interrupted in dispatch
AmplabJenkins commented on issue #26674: [SPARK-30059][CORE]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674#issuecomment-607846556 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26674: [SPARK-30059][CORE]Stop AsyncEventQueue when interrupted in dispatch
AmplabJenkins removed a comment on issue #26674: [SPARK-30059][CORE]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674#issuecomment-558499389 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
tgravescs commented on a change in pull request #28085:
[SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
URL: https://github.com/apache/spark/pull/28085#discussion_r402312734
##
File path:
core/src/main/scala/org/apache/spark/resource/TaskResourceRequests.scala
##
@@ -37,6 +38,8 @@ private[spark] class TaskResourceRequests() extends
Serializable {
def requests: Map[String, TaskResourceRequest] = _taskResources.asScala.toMap
+ def requestsJMap: JMap[String, TaskResourceRequest] =
_taskResources.asScala.asJava
Review comment:
yes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] asfgit closed pull request #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling
asfgit closed pull request #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling URL: https://github.com/apache/spark/pull/28053 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling
tgravescs commented on issue #28053: [SPARK-29153][CORE]Add ability to merge resource profiles within a stage with Stage Level Scheduling URL: https://github.com/apache/spark/pull/28053#issuecomment-607848328 merged to master. Thanks @dongjoon-hyun @Ngone51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
AmplabJenkins commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607798873 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120716/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
AmplabJenkins commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607798860 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
AmplabJenkins commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607801398 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120719/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
AmplabJenkins removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607800874 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25420/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
AmplabJenkins commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607801391 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
SparkQA removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607730976 **[Test build #120719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120719/testReport)** for PR 28100 at commit [`e32cc30`](https://github.com/apache/spark/commit/e32cc30e470edc149427e0988a3c20c2220303c2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
AmplabJenkins removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607800857 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
SparkQA commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607800751 **[Test build #120719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120719/testReport)** for PR 28100 at commit [`e32cc30`](https://github.com/apache/spark/commit/e32cc30e470edc149427e0988a3c20c2220303c2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window
AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607815600 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window
AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607815607 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120718/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window
AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607815600 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window
AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607815607 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120718/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
SparkQA removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607667622 **[Test build #120716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120716/testReport)** for PR 28097 at commit [`3c684a5`](https://github.com/apache/spark/commit/3c684a5f58407ff42371d8a003254f46e0566fc2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
SparkQA commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607797891 **[Test build #120716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120716/testReport)** for PR 28097 at commit [`3c684a5`](https://github.com/apache/spark/commit/3c684a5f58407ff42371d8a003254f46e0566fc2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
AmplabJenkins removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607798873 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120716/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
AmplabJenkins removed a comment on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607798860 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
cloud-fan commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607799550 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
SparkQA commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607800148 **[Test build #120722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120722/testReport)** for PR 28097 at commit [`6ba5d18`](https://github.com/apache/spark/commit/6ba5d187436ea0d307ff8bfd5780539a57084548). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
AmplabJenkins commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607800874 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25420/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
AmplabJenkins commented on issue #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097#issuecomment-607800857 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
AmplabJenkins removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607801391 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
AmplabJenkins removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607801398 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120719/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window
SparkQA commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607814571 **[Test build #120718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120718/testReport)** for PR 27943 at commit [`f9be82e`](https://github.com/apache/spark/commit/f9be82e249088ae083e2ac84b94be8e69bf50304). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window
SparkQA removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-607727006 **[Test build #120718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120718/testReport)** for PR 27943 at commit [`f9be82e`](https://github.com/apache/spark/commit/f9be82e249088ae083e2ac84b94be8e69bf50304). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] manuzhang commented on issue #24618: [SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill
manuzhang commented on issue #24618: [SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill URL: https://github.com/apache/spark/pull/24618#issuecomment-607820865 @amuraru May I ask how would you set those thresholds in regard to `spark.executor.memory` ? Would, say, `0.8 * spark.executor.memory` be a good candidate for those values ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
SparkQA commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607850276 **[Test build #120723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120723/testReport)** for PR 28085 at commit [`1071f40`](https://github.com/apache/spark/commit/1071f404baf2a32f270fcb0ac813d9fae763485d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607850898 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25421/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
AmplabJenkins commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607850887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
AmplabJenkins commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607850898 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25421/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-607850887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
SparkQA commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607854256 **[Test build #120724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120724/testReport)** for PR 28100 at commit [`e32cc30`](https://github.com/apache/spark/commit/e32cc30e470edc149427e0988a3c20c2220303c2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
Ngone51 commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607853755 jenkins, retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
AmplabJenkins commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607855199 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
AmplabJenkins commented on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607855210 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25422/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
AmplabJenkins removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607855210 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25422/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
AmplabJenkins removed a comment on issue #28100: [SPARK-31249][CORE]Fix flaky CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied URL: https://github.com/apache/spark/pull/28100#issuecomment-607855199 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan opened a new pull request #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
cloud-fan opened a new pull request #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102 ### What changes were proposed in this pull request? Write Spark version into Avro file metadata ### Why are the changes needed? The version info is very useful for backward compatibility. This is also done in parquet/orc. ### Does this PR introduce any user-facing change? no ### How was this patch tested? new test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
cloud-fan commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607858651 cc @MaxGekk @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
SparkQA commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607858705 **[Test build #120725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120725/testReport)** for PR 28102 at commit [`5aaf2db`](https://github.com/apache/spark/commit/5aaf2db549e3b6d441ea01fed6bb0440f5eff02d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607859479 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607859479 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607859490 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25423/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607859490 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25423/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #28098: [SPARK-30973][SQL]ScriptTransformationExec should wait for the termination
Ngone51 commented on issue #28098: [SPARK-30973][SQL]ScriptTransformationExec should wait for the termination URL: https://github.com/apache/spark/pull/28098#issuecomment-607860263 The error is not from `ScriptTransformationExec` but Scala test framework. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination …
cloud-fan commented on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination … URL: https://github.com/apache/spark/pull/27724#issuecomment-607863142 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination …
AmplabJenkins commented on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination … URL: https://github.com/apache/spark/pull/27724#issuecomment-607863948 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination …
AmplabJenkins commented on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination … URL: https://github.com/apache/spark/pull/27724#issuecomment-607863963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25424/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination …
AmplabJenkins removed a comment on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination … URL: https://github.com/apache/spark/pull/27724#issuecomment-592001890 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination …
cloud-fan commented on a change in pull request #27724: [SPARK-30973][SQL]
ScriptTransformationExec should wait for the termination …
URL: https://github.com/apache/spark/pull/27724#discussion_r402335000
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
##
@@ -161,6 +161,7 @@ case class ScriptTransformationExec(
if (scriptOutputReader != null) {
if (scriptOutputReader.next(scriptOutputWritable) <= 0) {
+ proc.waitFor()
Review comment:
shall we have a timeout here? just in case the process hangs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
SparkQA commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607864467 **[Test build #120725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120725/testReport)** for PR 28102 at commit [`5aaf2db`](https://github.com/apache/spark/commit/5aaf2db549e3b6d441ea01fed6bb0440f5eff02d). * This patch **fails Java style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class SparkAvroKeyOutputFormat extends AvroKeyOutputFormat ` * ` static class SparkRecordWriterFactory extends RecordWriterFactory ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination …
AmplabJenkins removed a comment on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination … URL: https://github.com/apache/spark/pull/27724#issuecomment-607863948 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination …
AmplabJenkins removed a comment on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination … URL: https://github.com/apache/spark/pull/27724#issuecomment-607863963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25424/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607864523 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120725/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607864513 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
SparkQA removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607858705 **[Test build #120725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120725/testReport)** for PR 28102 at commit [`5aaf2db`](https://github.com/apache/spark/commit/5aaf2db549e3b6d441ea01fed6bb0440f5eff02d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607864513 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time
cloud-fan commented on a change in pull request #28101: [SPARK-31328][SQL] Fix
rebasing of overlapped local timestamps during daylight saving time
URL: https://github.com/apache/spark/pull/28101#discussion_r402336635
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -1030,7 +1033,13 @@ object DateTimeUtils {
cal.get(Calendar.SECOND),
(Math.floorMod(micros, MICROS_PER_SECOND) * NANOS_PER_MICROS).toInt)
.plusDays(cal.get(Calendar.DAY_OF_MONTH) - 1)
-instantToMicros(localDateTime.atZone(ZoneId.systemDefault).toInstant)
+val zonedDateTime = localDateTime.atZone(ZoneId.systemDefault)
+val adjustedZdt = if (cal.get(Calendar.DST_OFFSET) == 0) {
Review comment:
can we add some comment here? people may not understand what
`cal.get(Calendar.DST_OFFSET) == 0` means
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time
cloud-fan commented on a change in pull request #28101: [SPARK-31328][SQL] Fix
rebasing of overlapped local timestamps during daylight saving time
URL: https://github.com/apache/spark/pull/28101#discussion_r402337523
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
##
@@ -821,4 +821,20 @@ class DateTimeUtilsSuite extends SparkFunSuite with
Matchers with SQLHelper {
days += 1
}
}
+
+ test("rebasing overlapped timestamps during daylight saving time") {
Review comment:
can we add JIRA ID?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-607864523 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120725/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate
HyukjinKwon commented on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate URL: https://github.com/apache/spark/pull/28089#issuecomment-607866353 Will take a look tomorrow This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination …
SparkQA commented on issue #27724: [SPARK-30973][SQL] ScriptTransformationExec should wait for the termination … URL: https://github.com/apache/spark/pull/27724#issuecomment-607867530 **[Test build #120726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120726/testReport)** for PR 27724 at commit [`99010e0`](https://github.com/apache/spark/commit/99010e0a020c7e72cfc0ac89e50b8a5b819a65ba). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] prakharjain09 commented on a change in pull request #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
prakharjain09 commented on a change in pull request #27864: [SPARK-20732][CORE]
Decommission cache blocks to other executors when an executor is decommissioned
URL: https://github.com/apache/spark/pull/27864#discussion_r402343206
##
File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
##
@@ -1761,6 +1776,57 @@ private[spark] class BlockManager(
blocksToRemove.size
}
+ def decommissionBlockManager(): Unit = {
+if (!blockManagerDecommissioning) {
+ logInfo("Starting block manager decommissioning process")
+ blockManagerDecommissioning = true
+ decommissionManager = Some(new BlockManagerDecommissionManager)
+ decommissionManager.foreach(_.start())
+}
+ }
+
+ /**
+ * Tries to offload all cached RDD blocks from this BlockManager to peer
BlockManagers
+ * Visible for testing
+ */
+ def offloadRddCacheBlocks(): Unit = {
+val replicateBlocksInfo =
master.getReplicateInfoForRDDBlocks(blockManagerId)
+
+if (replicateBlocksInfo.nonEmpty) {
+ logInfo(s"Need to replicate ${replicateBlocksInfo.size} blocks " +
+s"for block manager decommissioning")
+ // Refresh peer list once before starting replication
+ getPeers(true)
+}
+
+// Maximum number of storage replication failure which replicateBlock can
handle
+// before giving up for one block
Review comment:
@dongjoon-hyun Renamed to
"spark.storage.decommission.maxReplicationFailuresPerBlock". Any suggestions on
2) ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
SparkQA commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned URL: https://github.com/apache/spark/pull/27864#issuecomment-607871644 **[Test build #120727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120727/testReport)** for PR 27864 at commit [`076dd67`](https://github.com/apache/spark/commit/076dd671d9a9ac3c8d5926279f54517b8b39426a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
AmplabJenkins commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned URL: https://github.com/apache/spark/pull/27864#issuecomment-607872436 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
AmplabJenkins removed a comment on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned URL: https://github.com/apache/spark/pull/27864#issuecomment-607872436 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
AmplabJenkins removed a comment on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned URL: https://github.com/apache/spark/pull/27864#issuecomment-607872448 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25425/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
AmplabJenkins commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned URL: https://github.com/apache/spark/pull/27864#issuecomment-607872448 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25425/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
SparkQA commented on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-608078757 **[Test build #120723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120723/testReport)** for PR 28085 at commit [`1071f40`](https://github.com/apache/spark/commit/1071f404baf2a32f270fcb0ac813d9fae763485d). * This patch **fails from timeout after a configured wait of `400m`**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Huang-yi-3456 commented on a change in pull request #28002: [SPARK-31233][Core] Enhance RpcTimeoutException Log Message
Huang-yi-3456 commented on a change in pull request #28002: [SPARK-31233][Core]
Enhance RpcTimeoutException Log Message
URL: https://github.com/apache/spark/pull/28002#discussion_r402583376
##
File path: core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala
##
@@ -190,6 +190,47 @@ abstract class RpcEnvSuite extends SparkFunSuite with
BeforeAndAfterAll {
}
}
+ test("SPARK-31233: ask rpcEndpointRef in client mode timeout") {
+var remoteRef: RpcEndpointRef = null
+env.setupEndpoint("ask-remotely-server", new RpcEndpoint {
+ override val rpcEnv = env
+ override def receiveAndReply(context: RpcCallContext):
PartialFunction[Any, Unit] = {
+case Register(ref) =>
+ remoteRef = ref
+ context.reply("okay")
+case msg: String =>
+ context.reply(msg)
+ }
+})
+val conf = new SparkConf()
+val anotherEnv = createRpcEnv(conf, "remote", 0, clientMode = true)
+// Use anotherEnv to find out the RpcEndpointRef
+val rpcEndpointRef = anotherEnv.setupEndpointRef(env.address,
"ask-remotely-server")
+// Register a rpcEndpointRef in anotherEnv
+val anotherRef = anotherEnv.setupEndpoint("receiver", new RpcEndpoint {
+ override val rpcEnv = anotherEnv
+ override def receiveAndReply(context: RpcCallContext):
PartialFunction[Any, Unit] = {
+case _ =>
+ Thread.sleep(1200)
+ context.reply("okay")
+ }
+})
+try {
+ val reply = rpcEndpointRef.askSync[String](Register(anotherRef))
+ assert("okay" === reply)
+ val answer = remoteRef.ask[String]("msg",
+RpcTimeout(conf, Seq("spark.rpc.askTimeout", "spark.network.timeout"),
"1s"))
+ val thrown = intercept[Exception] {
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28038: [SPARK-31208][CORE] Add an expiremental cleanShuffleDependencies
AmplabJenkins removed a comment on issue #28038: [SPARK-31208][CORE] Add an expiremental cleanShuffleDependencies URL: https://github.com/apache/spark/pull/28038#issuecomment-608093404 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120736/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time
AmplabJenkins removed a comment on issue #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time URL: https://github.com/apache/spark/pull/28101#issuecomment-608093623 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time
AmplabJenkins removed a comment on issue #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time URL: https://github.com/apache/spark/pull/28101#issuecomment-608093638 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120731/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28103: [SPARK-31253][SQL][followup] Add metrics to AQE shuffle reader
AmplabJenkins removed a comment on issue #28103: [SPARK-31253][SQL][followup] Add metrics to AQE shuffle reader URL: https://github.com/apache/spark/pull/28103#issuecomment-608115571 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28103: [SPARK-31253][SQL][followup] Add metrics to AQE shuffle reader
SparkQA removed a comment on issue #28103: [SPARK-31253][SQL][followup] Add metrics to AQE shuffle reader URL: https://github.com/apache/spark/pull/28103#issuecomment-607987790 **[Test build #120733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120733/testReport)** for PR 28103 at commit [`8739b3c`](https://github.com/apache/spark/commit/8739b3c1ee67ddab3b87c4ca067344ce65f773a2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28103: [SPARK-31253][SQL][followup] Add metrics to AQE shuffle reader
SparkQA commented on issue #28103: [SPARK-31253][SQL][followup] Add metrics to AQE shuffle reader URL: https://github.com/apache/spark/pull/28103#issuecomment-608124243 **[Test build #120733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120733/testReport)** for PR 28103 at commit [`8739b3c`](https://github.com/apache/spark/commit/8739b3c1ee67ddab3b87c4ca067344ce65f773a2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
AmplabJenkins commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608172028 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
AmplabJenkins commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608172031 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25441/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608172031 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25441/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608172028 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
maropu commented on a change in pull request #28104: [SPARK-31331][SQL][DOCS]
Document Spark integration with Hive UDFs/UDAFs/UDTFs
URL: https://github.com/apache/spark/pull/28104#discussion_r402689144
##
File path: docs/sql-ref-functions-udf-hive.md
##
@@ -0,0 +1,64 @@
+---
+layout: global
+title: Integration with Hive UDFs/UDAFs/UDTFs
+displayTitle: Integration with Hive UDFs/UDAFs/UDTFs
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+### Description
+
+Spark SQL supports integration of Hive UDFs, UDAFs and UDTFs. Similar to Spark
UDFs and UDAFs, Hive UDFs work on a single row as input and generate a single
row as output, while Hive UDAFs operate on multiple rows and return a single
aggregated row as a result. In addition, Hive also supports UDTFs (User Defined
Tabular Functions) that act on one row as input and return multiple rows as
output. To use Hive UDFs/UDAFs/UTFs, the user should register them in Spark,
and then use them in Spark SQL queries.
+
+### Examples
+
+
+
+// Register a Hive UDF and use it in Spark SQL
+// Scala
+sql(s"CREATE TEMPORARY FUNCTION testUDF AS
'org.apache.spark.sql.hive.execution.PairUDF'")
+sql("SELECT testUDF(pair) FROM hiveUDFTestTable")
+
+// Register a Hive UDAF and use it in Spark SQL
+// Scala
+sql(
+"""
+|CREATE TEMPORARY FUNCTION test_avg
+|AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage'
+""".stripMargin)
+sql("SELECT test_avg(1), test_avg(substr(value,5)) FROM src")
+
+// Register a Hive UDTF and use it in Spark SQL
+// Scala
+// GenericUDTFCount2 outputs the number of rows seen, twice.
+// The function source code can be found at:
+// https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide+UDTF
+sql(s"ADD JAR ${hiveContext.getHiveFile("TestUDTF.jar").getCanonicalPath}")
+sql(
+"""
+|CREATE TEMPORARY FUNCTION udtf_count2
+|AS 'org.apache.spark.sql.hive.execution.GenericUDTFCount2'
+""".stripMargin)
Review comment:
ditto: indent
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
viirya commented on a change in pull request #28104: [SPARK-31331][SQL][DOCS]
Document Spark integration with Hive UDFs/UDAFs/UDTFs
URL: https://github.com/apache/spark/pull/28104#discussion_r402693704
##
File path: docs/sql-ref-functions-udf-hive.md
##
@@ -0,0 +1,64 @@
+---
+layout: global
+title: Integration with Hive UDFs/UDAFs/UDTFs
+displayTitle: Integration with Hive UDFs/UDAFs/UDTFs
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+### Description
+
+Spark SQL supports integration of Hive UDFs, UDAFs and UDTFs. Similar to Spark
UDFs and UDAFs, Hive UDFs work on a single row as input and generate a single
row as output, while Hive UDAFs operate on multiple rows and return a single
aggregated row as a result. In addition, Hive also supports UDTFs (User Defined
Tabular Functions) that act on one row as input and return multiple rows as
output. To use Hive UDFs/UDAFs/UTFs, the user should register them in Spark,
and then use them in Spark SQL queries.
+
+### Examples
+
+
+
+// Register a Hive UDF and use it in Spark SQL
+// Scala
+sql(s"CREATE TEMPORARY FUNCTION testUDF AS
'org.apache.spark.sql.hive.execution.PairUDF'")
+sql("SELECT testUDF(pair) FROM hiveUDFTestTable")
+
+// Register a Hive UDAF and use it in Spark SQL
+// Scala
+sql(
+"""
+|CREATE TEMPORARY FUNCTION test_avg
+|AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage'
+""".stripMargin)
+sql("SELECT test_avg(1), test_avg(substr(value,5)) FROM src")
Review comment:
Should we use UDAF with GROUP BY?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
AmplabJenkins removed a comment on issue #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#issuecomment-608079116 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120723/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Huang-yi-3456 commented on a change in pull request #28002: [SPARK-31233][Core] Enhance RpcTimeoutException Log Message
Huang-yi-3456 commented on a change in pull request #28002: [SPARK-31233][Core]
Enhance RpcTimeoutException Log Message
URL: https://github.com/apache/spark/pull/28002#discussion_r402624630
##
File path: core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala
##
@@ -190,6 +190,47 @@ abstract class RpcEnvSuite extends SparkFunSuite with
BeforeAndAfterAll {
}
}
+ test("SPARK-31233: ask rpcEndpointRef in client mode timeout") {
+var remoteRef: RpcEndpointRef = null
+env.setupEndpoint("ask-remotely-server", new RpcEndpoint {
+ override val rpcEnv = env
+ override def receiveAndReply(context: RpcCallContext):
PartialFunction[Any, Unit] = {
+case Register(ref) =>
+ remoteRef = ref
+ context.reply("okay")
+case msg: String =>
+ context.reply(msg)
+ }
+})
+val conf = new SparkConf()
+val anotherEnv = createRpcEnv(conf, "remote", 0, clientMode = true)
+// Use anotherEnv to find out the RpcEndpointRef
+val rpcEndpointRef = anotherEnv.setupEndpointRef(env.address,
"ask-remotely-server")
+// Register a rpcEndpointRef in anotherEnv
+val anotherRef = anotherEnv.setupEndpoint("receiver", new RpcEndpoint {
+ override val rpcEnv = anotherEnv
+ override def receiveAndReply(context: RpcCallContext):
PartialFunction[Any, Unit] = {
+case _ =>
+ Thread.sleep(1200)
+ context.reply("okay")
+ }
+})
+try {
+ val reply = rpcEndpointRef.askSync[String](Register(anotherRef))
+ assert("okay" === reply)
+ val answer = remoteRef.ask[String]("msg",
+RpcTimeout(conf, Seq("spark.rpc.askTimeout", "spark.network.timeout"),
"1s"))
+ val thrown = intercept[Exception] {
+ThreadUtils.awaitResult(answer, Duration(1300, MILLISECONDS))
+ }
+ assert(thrown.getClass.equals(classOf[RpcTimeoutException]))
+ assert(!thrown.getMessage.contains("null"))
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate
AmplabJenkins removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate URL: https://github.com/apache/spark/pull/28089#issuecomment-608149636 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25439/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate
AmplabJenkins removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate URL: https://github.com/apache/spark/pull/28089#issuecomment-608149633 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608173864 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
