[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data typ
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526462894 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data typ
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526462899 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109931/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526462894 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526462899 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109931/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
SparkQA removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526425369 **[Test build #109931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109931/testReport)** for PR 25458 at commit [`abe9a84`](https://github.com/apache/spark/commit/abe9a8431f0b7f5cd403e54b31834aecce66c524). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
SparkQA commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526462461 **[Test build #109931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109931/testReport)** for PR 25458 at commit [`abe9a84`](https://github.com/apache/spark/commit/abe9a8431f0b7f5cd403e54b31834aecce66c524). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq'
AmplabJenkins removed a comment on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq' URL: https://github.com/apache/spark/pull/25627#issuecomment-526458347 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq'
AmplabJenkins commented on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq' URL: https://github.com/apache/spark/pull/25627#issuecomment-526459457 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526458256 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109936/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526458253 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq'
AmplabJenkins removed a comment on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq' URL: https://github.com/apache/spark/pull/25627#issuecomment-526457930 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq'
AmplabJenkins commented on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq' URL: https://github.com/apache/spark/pull/25627#issuecomment-526458347 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526458256 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109936/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526433143 **[Test build #109936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109936/testReport)** for PR 20965 at commit [`17706c3`](https://github.com/apache/spark/commit/17706c3b5d62ac30ac004aebaf5c0c118243e116). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526458253 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526458143 **[Test build #109936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109936/testReport)** for PR 20965 at commit [`17706c3`](https://github.com/apache/spark/commit/17706c3b5d62ac30ac004aebaf5c0c118243e116). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq'
AmplabJenkins commented on issue #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq' URL: https://github.com/apache/spark/pull/25627#issuecomment-526457930 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xianyinxin opened a new pull request #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq'
xianyinxin opened a new pull request #25627: [SPARK-28923][SQL] Deduplicate the codes 'multipartIdentifier' and 'identifierSeq' URL: https://github.com/apache/spark/pull/25627 ### What changes were proposed in this pull request? In `sqlbase.g4`, `multipartIdentifier` and `identifierSeq` have the same functionality. We'd better deduplicate them. ### Why are the changes needed? Deduplicate the codes which have the same function. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tnachen commented on a change in pull request #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
tnachen commented on a change in pull request #25614: [WIP][SPARK-28887][K8S]
Executor pod status fix
URL: https://github.com/apache/spark/pull/25614#discussion_r319357937
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
##
@@ -42,32 +43,47 @@ object ExecutorPodsSnapshot extends Logging {
ExecutorPodsSnapshot(toStatesByExecutorId(executorPods))
}
- def apply(): ExecutorPodsSnapshot = ExecutorPodsSnapshot(Map.empty[Long,
ExecutorPodState])
+ def apply(): ExecutorPodsSnapshot = ExecutorPodsSnapshot(Map.empty[Long,
ExecutorState])
- private def toStatesByExecutorId(executorPods: Seq[Pod]): Map[Long,
ExecutorPodState] = {
+ private def toStatesByExecutorId(executorPods: Seq[Pod]): Map[Long,
ExecutorState] = {
executorPods.map { pod =>
(pod.getMetadata.getLabels.get(SPARK_EXECUTOR_ID_LABEL).toLong,
toState(pod))
}.toMap
}
- private def toState(pod: Pod): ExecutorPodState = {
+ private def toState(pod: Pod): ExecutorState = {
if (isDeleted(pod)) {
- PodDeleted(pod)
+ ExecutorPodDeleted(pod)
} else {
val phase = pod.getStatus.getPhase.toLowerCase(Locale.ROOT)
phase match {
case "pending" =>
- PodPending(pod)
+ ExecutorPending(pod)
case "running" =>
- PodRunning(pod)
+ // Checking executor container status is not terminated
Review comment:
Can we add a test for this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data typ
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526453071 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data typ
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526453076 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109927/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526453076 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109927/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526453071 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
SparkQA removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526414104 **[Test build #109927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109927/testReport)** for PR 25458 at commit [`dacd46b`](https://github.com/apache/spark/commit/dacd46b3856060ba1792779815d3e938468abc5a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
SparkQA commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526452793 **[Test build #109927 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109927/testReport)** for PR 25458 at commit [`dacd46b`](https://github.com/apache/spark/commit/dacd46b3856060ba1792779815d3e938468abc5a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #25611: [SPARK-28901][SQL] SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI
AngersZh commented on issue #25611: [SPARK-28901][SQL] SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI URL: https://github.com/apache/spark/pull/25611#issuecomment-526449905 @juliuszsompolski There are some conflicts in the process. Changed a lot to cover all problems you have mentioned， include call ` close() `before setState `FINISHED`， Thanks for review again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25588: [SPARK-28000][SQL][TEST] Port comments.sql
dongjoon-hyun commented on a change in pull request #25588: [SPARK-28000][SQL][TEST] Port comments.sql URL: https://github.com/apache/spark/pull/25588#discussion_r319351993 ## File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/comments.sql ## @@ -0,0 +1,48 @@ +-- +-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group +-- +-- +-- COMMENTS +-- https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/comments.sql Review comment: Shall we use `BETA3` from now? I believe it will be the same with `BETA2`. - https://github.com/postgres/postgres/blob/REL_12_BETA3/src/test/regress/sql/comments.sql This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
dongjoon-hyun commented on a change in pull request #25614:
[WIP][SPARK-28887][K8S] Executor pod status fix
URL: https://github.com/apache/spark/pull/25614#discussion_r319351543
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodStates.scala
##
@@ -18,20 +18,20 @@ package org.apache.spark.scheduler.cluster.k8s
import io.fabric8.kubernetes.api.model.Pod
-sealed trait ExecutorPodState {
+sealed trait ExecutorState {
Review comment:
@jinxingwang . Could you make another PR for this renaming?
This PR seems to have two orthogonal themes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
dongjoon-hyun commented on a change in pull request #25614:
[WIP][SPARK-28887][K8S] Executor pod status fix
URL: https://github.com/apache/spark/pull/25614#discussion_r319351543
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodStates.scala
##
@@ -18,20 +18,20 @@ package org.apache.spark.scheduler.cluster.k8s
import io.fabric8.kubernetes.api.model.Pod
-sealed trait ExecutorPodState {
+sealed trait ExecutorState {
Review comment:
Hi, @jinxingwang . Could you make another PR for this renaming?
This PR seems to have two orthogonal themes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
dongjoon-hyun commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526449061 Thank you for making a PR, @jinxingwang . Could you fix the scala style? You can check with `dev/scalastyle`. ``` [error] /home/jenkins/workspace/SparkPullRequestBuilder/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala:25:18: Instead of importing implicits in scala.collection.JavaConversions._, import [error] scala.collection.JavaConverters._ and use .asScala / .asJava methods [error] /home/jenkins/workspace/SparkPullRequestBuilder/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala:25:0: collection.JavaConversions._ should be in group 3rdParty, not spark. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow
viirya commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow URL: https://github.com/apache/spark/pull/25625#issuecomment-526447635 No problem at all! thanks @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow
dongjoon-hyun commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow URL: https://github.com/apache/spark/pull/25625#issuecomment-526446430 Oh.. Resovling conflicts seem not good. It removes the PR template and shows me as a co-author. Sorry about that, @viirya . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526446039 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109929/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526446035 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526446039 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109929/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25625: [SPARK-28920][INFRA] Set up java version for github workflow
dongjoon-hyun closed pull request #25625: [SPARK-28920][INFRA] Set up java version for github workflow URL: https://github.com/apache/spark/pull/25625 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526415595 **[Test build #109929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109929/testReport)** for PR 20965 at commit [`67673db`](https://github.com/apache/spark/commit/67673db1349bb59d4a4917a249170b821ee02041). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526446035 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526445912 **[Test build #109929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109929/testReport)** for PR 20965 at commit [`67673db`](https://github.com/apache/spark/commit/67673db1349bb59d4a4917a249170b821ee02041). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow
AmplabJenkins removed a comment on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow URL: https://github.com/apache/spark/pull/25625#issuecomment-526445262 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109930/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow
AmplabJenkins removed a comment on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow URL: https://github.com/apache/spark/pull/25625#issuecomment-526445255 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow
AmplabJenkins commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow URL: https://github.com/apache/spark/pull/25625#issuecomment-526445262 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109930/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow
AmplabJenkins commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow URL: https://github.com/apache/spark/pull/25625#issuecomment-526445255 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow
SparkQA removed a comment on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow URL: https://github.com/apache/spark/pull/25625#issuecomment-526417252 **[Test build #109930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109930/testReport)** for PR 25625 at commit [`4fe4854`](https://github.com/apache/spark/commit/4fe4854dd5eb217b60783a1ed8cb8af0ec4e3424). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow
SparkQA commented on issue #25625: [SPARK-28920][INFRA] Set up java version for github workflow URL: https://github.com/apache/spark/pull/25625#issuecomment-526444966 **[Test build #109930 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109930/testReport)** for PR 25625 at commit [`4fe4854`](https://github.com/apache/spark/commit/4fe4854dd5eb217b60783a1ed8cb8af0ec4e3424). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #25611: [SPARK-28901][SQL] SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI
AngersZh commented on a change in pull request #25611: [SPARK-28901][SQL]
SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI
URL: https://github.com/apache/spark/pull/25611#discussion_r319345622
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala
##
@@ -239,6 +239,15 @@ object HiveThriftServer2 extends Logging {
executionList(id).state = ExecutionState.COMPILED
}
+def onStatementCanceled(id: String): Unit = {
+ synchronized {
+executionList(id).finishTimestamp = System.currentTimeMillis
+executionList(id).state = ExecutionState.CANCELED
+totalRunning -= 1
Review comment:
@juliuszsompolski if we call close() before FINISH, onOperationClose()
won't do
`totalRunning -= 1`, also a bug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25610: [SPARK-28899][SQL][TEST] merge the testing in-memory v2 catalogs from catalyst and core
cloud-fan commented on issue #25610: [SPARK-28899][SQL][TEST] merge the testing in-memory v2 catalogs from catalyst and core URL: https://github.com/apache/spark/pull/25610#issuecomment-526441695 @rdblue congrats and thanks for merging my PR! Hopefully this is your first Spark PR merging :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wenxuanguan commented on a change in pull request #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming
wenxuanguan commented on a change in pull request #25618:
[SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming
URL: https://github.com/apache/spark/pull/25618#discussion_r319342797
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataWriter.scala
##
@@ -18,16 +18,160 @@
package org.apache.spark.sql.kafka010
import java.{util => ju}
+import java.util.concurrent.atomic.AtomicInteger
+
+import com.google.common.cache._
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.Attribute
import org.apache.spark.sql.sources.v2.writer._
+import org.apache.spark.util.Utils
+
+/**
+ * A [[WriterCommitMessage]] for Kafka commit message.
+ * @param transactionalId Unique transactionalId for each producer.
+ * @param epoch Transactional epoch.
+ * @param producerId Transactional producerId for producer, got when init
transaction.
+ */
+private[kafka010] case class ProducerTransactionMetaData(
+transactionalId: String,
+epoch: Short,
+producerId: Long)
+ extends WriterCommitMessage
+
+/**
+ * Emtpy commit message for resume transaction.
+ */
+private case object EmptyCommitMessage extends WriterCommitMessage
+
+private[kafka010] case object ProducerTransactionMetaData {
+ val VERSION = 1
+
+ def toTransactionId(
+ executorId: String,
+ taskIndex: String,
+ transactionalIdSuffix: String): String = {
+toTransactionId(toProducerIdentity(executorId, taskIndex),
transactionalIdSuffix)
+ }
+
+ def toTransactionId(producerIdentity: String, transactionalIdSuffix:
String): String = {
+s"$producerIdentity||$transactionalIdSuffix"
+ }
+
+ def toTransactionalIdSuffix(transactionalId: String): String = {
+transactionalId.split("\\|\\|", 2)(1)
+ }
+
+ def toProducerIdentity(transactionalId: String): String = {
+transactionalId.split("\\|\\|", 2)(0)
+ }
+
+ def toExecutorId(transactionalId: String): String = {
+val producerIdentity = toProducerIdentity(transactionalId)
+producerIdentity.split("-", 2)(0)
+ }
+
+ def toTaskIndex(transactionalId: String): String = {
+val producerIdentity = toProducerIdentity(transactionalId)
+producerIdentity.split("-", 2)(1)
+ }
+
+ def toProducerIdentity(executorId: String, taskIndex: String): String = {
+s"$executorId-$taskIndex"
+ }
+}
+
+/**
+ * A [[DataWriter]] for Kafka transactional writing. One data writer will be
created
+ * in each partition to process incoming rows.
+ *
+ * @param targetTopic The topic that this data writer is targeting. If None,
topic will be inferred
+ *from a `topic` field in the incoming data.
+ * @param producerParams Parameters to use for the Kafka producer.
+ * @param inputSchema The attributes in the input data.
+ */
+private[kafka010] class KafkaTransactionDataWriter(
+targetTopic: Option[String],
+producerParams: ju.Map[String, Object],
+inputSchema: Seq[Attribute])
+ extends KafkaRowWriter(inputSchema, targetTopic) with
DataWriter[InternalRow] {
+
+ private lazy val producer = {
+val kafkaProducer = CachedKafkaProducer.getOrCreate(producerParams)
Review comment:
I think caching logic is ok and we can control producer creation per task,
and also failover with transactional.id in producerParams.
Transaction producer is not thread safe, so what I do is one producer per
task in one micro-batch, and in next batch reused the created producer instead
of recreate one since transaction is complete in every micro-batch. With
producerParams, transactional.id is different between tasks in one micro-batch,
but same in the next micro-batch.
And if task number is same for every executor in every micro-batch, no more
producer will be created except the first micro-batch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE
AmplabJenkins commented on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE URL: https://github.com/apache/spark/pull/25502#issuecomment-526438195 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE
AmplabJenkins removed a comment on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE URL: https://github.com/apache/spark/pull/25502#issuecomment-526438195 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE
AmplabJenkins commented on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE URL: https://github.com/apache/spark/pull/25502#issuecomment-526438201 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109922/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE
AmplabJenkins removed a comment on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE URL: https://github.com/apache/spark/pull/25502#issuecomment-526438201 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109922/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE
SparkQA commented on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE URL: https://github.com/apache/spark/pull/25502#issuecomment-526437869 **[Test build #109922 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109922/testReport)** for PR 25502 at commit [`1f82198`](https://github.com/apache/spark/commit/1f821986e7458ca4b87c898d6c3c6d73b0ac78ad). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE
SparkQA removed a comment on issue #25502: [SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE URL: https://github.com/apache/spark/pull/25502#issuecomment-526397155 **[Test build #109922 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109922/testReport)** for PR 25502 at commit [`1f82198`](https://github.com/apache/spark/commit/1f821986e7458ca4b87c898d6c3c6d73b0ac78ad). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #25611: [SPARK-28901][SQL] SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI
AngersZh commented on a change in pull request #25611: [SPARK-28901][SQL]
SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI
URL: https://github.com/apache/spark/pull/25611#discussion_r319341975
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
##
@@ -275,6 +279,7 @@ private[hive] class SparkExecuteStatementOperation(
override def cancel(): Unit = {
logInfo(s"Cancel '$statement' with $statementId")
cleanup(OperationState.CANCELED)
+HiveThriftServer2.listener.onStatementCanceled(statementId)
Review comment:
> This can call `onStatementCanceled()` before the thread launching
`execute()` gets to `onStatementStarted()`, which will result in the listener
throwing an NoSuchElementException on executionList(id).
> `onStatementStarted()` could be moved to `runInternal()` in the main
thread to prevent that, but then the `catch` block there has to add
`onStatementError()`. It then would also be good to extend the `try` block to
the start of `execute()`
add a new status ExecutionState.PREPARED to fix this problem, call
onStatementPrepared in runInternal().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #25611: [SPARK-28901][SQL] SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI
AngersZh commented on a change in pull request #25611: [SPARK-28901][SQL]
SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI
URL: https://github.com/apache/spark/pull/25611#discussion_r319341975
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
##
@@ -275,6 +279,7 @@ private[hive] class SparkExecuteStatementOperation(
override def cancel(): Unit = {
logInfo(s"Cancel '$statement' with $statementId")
cleanup(OperationState.CANCELED)
+HiveThriftServer2.listener.onStatementCanceled(statementId)
Review comment:
add a new status ExecutionState.PREPARED to fix this problem, call
onStatementPrepared in runInternal().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming
HeartSaVioR edited a comment on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming URL: https://github.com/apache/spark/pull/25618#issuecomment-526436430 Spark doesn't have semantics of 2PC natively as you've seen DSv2 API - If I understand correctly, Spark HDFS sink doesn't leverage 2PC. Previously it used temporal directory - let all tasks write to that directory, and driver move that directory to final destination only when all tasks succeed to write. It leverages the fact that "rename" is atomic, so it didn't support "exactly-once" if underlying filesystem doesn't support atomic renaming. Now it leverages metadata - let all tasks write files, and pass the list of files (path) written to driver. When driver receives all list of written files from all tasks, driver writes overall list of files to metadata. So exactly-once for HDFS is only guaranteed when "Spark" reads the output which is aware of metadata information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming
HeartSaVioR edited a comment on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming URL: https://github.com/apache/spark/pull/25618#issuecomment-526436430 Spark doesn't have semantics of 2PC natively as you've seen DSv2 API - Spark HDFS sink doesn't leverage 2PC. If I understand correctly, previously it used temporal directory - let all tasks write to that directory, and driver move that directory to final destination only when all tasks succeed to write. It leverages the fact that "rename" is atomic, so it didn't support "exactly-once" if underlying filesystem doesn't support atomic renaming. Now it leverages metadata - let all tasks write files, and pass the list of files (path) written to driver. When driver receives all list of written files from all tasks, driver writes overall list of files to metadata. So exactly-once for HDFS is only guaranteed when "Spark" reads the output which is aware of metadata information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25622: [SPARK-28915][CORE]The new keyword is not used when instantiating the WorkerOffer.
HyukjinKwon commented on a change in pull request #25622:
[SPARK-28915][CORE]The new keyword is not used when instantiating the
WorkerOffer.
URL: https://github.com/apache/spark/pull/25622#discussion_r319341164
##
File path:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##
@@ -270,7 +270,7 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
val activeExecutors = executorDataMap.filterKeys(executorIsAlive)
val workOffers = activeExecutors.map {
case (id, executorData) =>
-new WorkerOffer(id, executorData.executorHost,
executorData.freeCores,
+WorkerOffer(id, executorData.executorHost, executorData.freeCores,
Review comment:
I think we don't have to change this - it's going to cause a lot of
conflicts when we backport but the gain here is virtually nothing. Either way
is legitimate.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration()
AmplabJenkins removed a comment on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration() URL: https://github.com/apache/spark/pull/25616#issuecomment-526435912 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109928/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming
HeartSaVioR commented on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming URL: https://github.com/apache/spark/pull/25618#issuecomment-526436430 Spark doesn't have semantics of 2PC natively as you've seen DSv2 API - Spark HDFS sink doesn't leverage 2PC. If I understand correctly, previously it used temporal directory - let all tasks write to that directory, and driver move that directory to final destination only when all tasks succeed to write. Now it leverages metadata - let all tasks write files, and pass the list of files (path) written to driver. When driver receives all list of written files from all tasks, driver writes overall list of files to metadata. So exactly-once for HDFS is only guaranteed when "Spark" reads the output which is aware of metadata information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration()
AmplabJenkins removed a comment on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration() URL: https://github.com/apache/spark/pull/25616#issuecomment-526435907 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration()
SparkQA removed a comment on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration() URL: https://github.com/apache/spark/pull/25616#issuecomment-526415600 **[Test build #109928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109928/testReport)** for PR 25616 at commit [`50d6d75`](https://github.com/apache/spark/commit/50d6d7558d098a8423de0ae7570b4863c778dd14). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #25611: [SPARK-28901][SQL] SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI
AngersZh commented on a change in pull request #25611: [SPARK-28901][SQL]
SparkThriftServer's Cancel SQL Operation show it in JDBC Tab UI
URL: https://github.com/apache/spark/pull/25611#discussion_r319341101
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
##
@@ -275,6 +279,7 @@ private[hive] class SparkExecuteStatementOperation(
override def cancel(): Unit = {
logInfo(s"Cancel '$statement' with $statementId")
cleanup(OperationState.CANCELED)
+HiveThriftServer2.listener.onStatementCanceled(statementId)
Review comment:
This can truly happen, especially when running SQL asynchronously， I add a
new STATUS of ExecutStatus.PREPARED in runInternal(), then onStatementStart
just change status, this can fix this problem.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xianyinxin commented on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2
xianyinxin commented on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2 URL: https://github.com/apache/spark/pull/25626#issuecomment-526436021 cc @cloud-fan @rdblue This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25619: [SPARK-28911][SS]unify kafka source option pattern
HyukjinKwon commented on issue #25619: [SPARK-28911][SS]unify kafka source option pattern URL: https://github.com/apache/spark/pull/25619#issuecomment-526436037 Yea, let's don't change. You gotta also keep the compatibility and the namespace-like convention seems fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration()
AmplabJenkins commented on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration() URL: https://github.com/apache/spark/pull/25616#issuecomment-526435907 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration()
AmplabJenkins commented on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration() URL: https://github.com/apache/spark/pull/25616#issuecomment-526435912 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109928/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration()
SparkQA commented on issue #25616: [SPARK-28907][CORE] Review invalid usage of new Configuration() URL: https://github.com/apache/spark/pull/25616#issuecomment-526435727 **[Test build #109928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109928/testReport)** for PR 25616 at commit [`50d6d75`](https://github.com/apache/spark/commit/50d6d7558d098a8423de0ae7570b4863c778dd14). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow
AmplabJenkins removed a comment on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow URL: https://github.com/apache/spark/pull/25624#issuecomment-526434565 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109924/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow
AmplabJenkins removed a comment on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow URL: https://github.com/apache/spark/pull/25624#issuecomment-526434558 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow
AmplabJenkins commented on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow URL: https://github.com/apache/spark/pull/25624#issuecomment-526434565 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109924/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow
AmplabJenkins commented on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow URL: https://github.com/apache/spark/pull/25624#issuecomment-526434558 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow
dongjoon-hyun closed pull request #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow URL: https://github.com/apache/spark/pull/25624 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow
SparkQA commented on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow URL: https://github.com/apache/spark/pull/25624#issuecomment-526434153 **[Test build #109924 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109924/testReport)** for PR 25624 at commit [`04bb2b0`](https://github.com/apache/spark/commit/04bb2b09be91605287f232a378c69c3930adcdb2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526434276 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14963/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow
dongjoon-hyun commented on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow URL: https://github.com/apache/spark/pull/25624#issuecomment-526434325 Thank you all. Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526434276 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14963/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow
SparkQA removed a comment on issue #25624: [SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow URL: https://github.com/apache/spark/pull/25624#issuecomment-526408778 **[Test build #109924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109924/testReport)** for PR 25624 at commit [`04bb2b0`](https://github.com/apache/spark/commit/04bb2b09be91605287f232a378c69c3930adcdb2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526434268 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526434268 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
AmplabJenkins removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526433875 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
AmplabJenkins removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526433879 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109934/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
SparkQA removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526431535 **[Test build #109934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109934/testReport)** for PR 25623 at commit [`ac0b9e9`](https://github.com/apache/spark/commit/ac0b9e94aa75128debc4e2cfef11b5e7e1953584). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
AmplabJenkins commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526433875 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
SparkQA commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526433794 **[Test build #109934 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109934/testReport)** for PR 25623 at commit [`ac0b9e9`](https://github.com/apache/spark/commit/ac0b9e94aa75128debc4e2cfef11b5e7e1953584). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
AmplabJenkins commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526433879 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109934/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
beliefer commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#discussion_r319339008 ## File path: docs/configuration.md ## @@ -250,17 +250,17 @@ of the most common options to set are: spark.executor.memoryOverhead executorMemory * 0.10, with minimum of 384 -Amount of non-heap memory to be allocated per executor process in cluster mode, in MiB unless +Amount of additional memory to be allocated per executor process in cluster mode, in MiB unless otherwise specified. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the executor size (typically 6-10%). This option is currently supported on YARN and Kubernetes. -Note: Non-heap memory includes off-heap memory -(when spark.memory.offHeap.enabled=true) and memory used by other executor processes -(e.g. python process that goes with a PySpark executor) and memory used by other non-executor -processes running in the same container. The maximum memory size of container to running executor -is determined by the sum of spark.executor.memoryOverhead and -spark.executor.memory. +Note: Additional memory includes PySpark executor memory Review comment: I know the meaning of this PR. Maybe the new idea is a way. As I know, the origin decision is to unify all the different part. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data
maropu commented on a change in pull request #25458: [SPARK-27931][SQL] Accept
"true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim
input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#discussion_r319338912
##
File path: sql/core/src/test/resources/sql-tests/results/pgSQL/boolean.sql.out
##
@@ -39,7 +39,7 @@ SELECT boolean(' f ') AS `false`
-- !query 4 schema
struct
-- !query 4 output
-NULL
+false
Review comment:
I found the four places to remove.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526433143 **[Test build #109936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109936/testReport)** for PR 20965 at commit [`17706c3`](https://github.com/apache/spark/commit/17706c3b5d62ac30ac004aebaf5c0c118243e116). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data
maropu commented on a change in pull request #25458: [SPARK-27931][SQL] Accept
"true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim
input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#discussion_r319338780
##
File path: sql/core/src/test/resources/sql-tests/results/pgSQL/boolean.sql.out
##
@@ -39,7 +39,7 @@ SELECT boolean(' f ') AS `false`
-- !query 4 schema
struct
-- !query 4 output
-NULL
+false
Review comment:
plz remove the comments, too:
https://github.com/apache/spark/blob/d502c80404c398d852dfa5f86a0e87c104a6286e/sql/core/src/test/resources/sql-tests/inputs/pgSQL/boolean.sql#L25
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526432709 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14962/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] younggyuchun edited a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
younggyuchun edited a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526430106 Thank you All @cloud-fan @HyukjinKwon @dongjoon-hyun @wangyum @dongjoon-hyun @gatorsmile @maropu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526432706 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
AmplabJenkins removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526432651 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
AmplabJenkins removed a comment on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526432656 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14961/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
AmplabJenkins commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526432651 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526432706 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold
AmplabJenkins commented on issue #25623: [SPARK-28786][DOC][SQL][FOLLOW-UP] Change "Related Statements" to bold URL: https://github.com/apache/spark/pull/25623#issuecomment-526432656 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14961/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2
AmplabJenkins removed a comment on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2 URL: https://github.com/apache/spark/pull/25626#issuecomment-526431477 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
