[GitHub] dongjoon-hyun commented on a change in pull request #23456: [SPARK-26538][SQL] Set default precision and scale for elements of postgres numeric array
dongjoon-hyun commented on a change in pull request #23456: [SPARK-26538][SQL]
Set default precision and scale for elements of postgres numeric array
URL: https://github.com/apache/spark/pull/23456#discussion_r246661712
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala
##
@@ -60,7 +60,12 @@ private object PostgresDialect extends JdbcDialect {
case "bytea" => Some(BinaryType)
case "timestamp" | "timestamptz" | "time" | "timetz" => Some(TimestampType)
case "date" => Some(DateType)
-case "numeric" | "decimal" => Some(DecimalType.bounded(precision, scale))
+case "numeric" | "decimal" => if (precision > 0) {
+ Some(DecimalType.bounded(precision, scale))
+} else {
+ // SPARK-26538: handle numeric without explicit precision and scale.
+ Some(DecimalType. SYSTEM_DEFAULT)
+}
Review comment:
Hi, @a-shkarupin . Thank you for your first contribution.
Could you follow the existing succinct style? What I mean is having two
`case "numeric" | "decimal"`s.
-
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L215-L217
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23456: [SPARK-26538][SQL] Set default precision and scale for elements of postgres numeric array
dongjoon-hyun commented on a change in pull request #23456: [SPARK-26538][SQL]
Set default precision and scale for elements of postgres numeric array
URL: https://github.com/apache/spark/pull/23456#discussion_r246661712
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala
##
@@ -60,7 +60,12 @@ private object PostgresDialect extends JdbcDialect {
case "bytea" => Some(BinaryType)
case "timestamp" | "timestamptz" | "time" | "timetz" => Some(TimestampType)
case "date" => Some(DateType)
-case "numeric" | "decimal" => Some(DecimalType.bounded(precision, scale))
+case "numeric" | "decimal" => if (precision > 0) {
+ Some(DecimalType.bounded(precision, scale))
+} else {
+ // SPARK-26538: handle numeric without explicit precision and scale.
+ Some(DecimalType. SYSTEM_DEFAULT)
+}
Review comment:
Hi, @a-shkarupin .
Could you follow the existing succinct style? What I mean is having two
`case "numeric" | "decimal"`s.
-
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L215-L217
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23456: [SPARK-26538][SQL] Set default precision and scale for elements of postgres numeric array
dongjoon-hyun commented on a change in pull request #23456: [SPARK-26538][SQL]
Set default precision and scale for elements of postgres numeric array
URL: https://github.com/apache/spark/pull/23456#discussion_r246661712
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala
##
@@ -60,7 +60,12 @@ private object PostgresDialect extends JdbcDialect {
case "bytea" => Some(BinaryType)
case "timestamp" | "timestamptz" | "time" | "timetz" => Some(TimestampType)
case "date" => Some(DateType)
-case "numeric" | "decimal" => Some(DecimalType.bounded(precision, scale))
+case "numeric" | "decimal" => if (precision > 0) {
+ Some(DecimalType.bounded(precision, scale))
+} else {
+ // SPARK-26538: handle numeric without explicit precision and scale.
+ Some(DecimalType. SYSTEM_DEFAULT)
+}
Review comment:
Hi, @a-shkarupin .
Could you follow the existing style? What I mean is having two `case
"numeric" | "decimal"`s.
-
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L215-L217
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23456: [SPARK-26538][SQL] Set default precision and scale for elements of postgres numeric array
dongjoon-hyun commented on a change in pull request #23456: [SPARK-26538][SQL]
Set default precision and scale for elements of postgres numeric array
URL: https://github.com/apache/spark/pull/23456#discussion_r246661712
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala
##
@@ -60,7 +60,12 @@ private object PostgresDialect extends JdbcDialect {
case "bytea" => Some(BinaryType)
case "timestamp" | "timestamptz" | "time" | "timetz" => Some(TimestampType)
case "date" => Some(DateType)
-case "numeric" | "decimal" => Some(DecimalType.bounded(precision, scale))
+case "numeric" | "decimal" => if (precision > 0) {
+ Some(DecimalType.bounded(precision, scale))
+} else {
+ // SPARK-26538: handle numeric without explicit precision and scale.
+ Some(DecimalType. SYSTEM_DEFAULT)
+}
Review comment:
Hi, @a-shkarupin .
Could you follow the existing style?
-
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L215-L217
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
AmplabJenkins removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453001362 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6885/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
AmplabJenkins commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453001355 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
SparkQA commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453001321 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/6885/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
AmplabJenkins removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453001355 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
AmplabJenkins commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453001362 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6885/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22093: [SPARK-25100][CORE] Fix no registering TaskCommitMessage bug
AmplabJenkins removed a comment on issue #22093: [SPARK-25100][CORE] Fix no registering TaskCommitMessage bug URL: https://github.com/apache/spark/pull/22093#issuecomment-412548219 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
SparkQA commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453000850 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/6885/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22093: [SPARK-25100][CORE] Fix no registering TaskCommitMessage bug
AmplabJenkins commented on issue #22093: [SPARK-25100][CORE] Fix no registering TaskCommitMessage bug URL: https://github.com/apache/spark/pull/22093#issuecomment-453000918 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
AmplabJenkins commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453000313 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101008/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
AmplabJenkins removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453000313 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101008/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
SparkQA removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-452997827 **[Test build #101008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101008/testReport)** for PR 23504 at commit [`4400077`](https://github.com/apache/spark/commit/4400077ff8daf162cf7d1f7adcfc1bf5b77017f0). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23502: [SPARK-22128][CORE][BUILD] Add `paranamer` dependency to `core` module
dongjoon-hyun commented on a change in pull request #23502: [SPARK-22128][CORE][BUILD] Add `paranamer` dependency to `core` module URL: https://github.com/apache/spark/pull/23502#discussion_r246659697 ## File path: core/pom.xml ## @@ -33,6 +33,10 @@ Spark Project Core http://spark.apache.org/ + + com.thoughtworks.paranamer Review comment: Thank you for review, @felixcheung . This is not a new dependency. - https://github.com/apache/spark/blob/master/pom.xml#L1948-L1952 The license is here. - https://github.com/apache/spark/blob/master/LICENSE-binary#L414 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
AmplabJenkins commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453000310 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
AmplabJenkins removed a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453000310 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
SparkQA commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-453000250 **[Test build #101008 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101008/testReport)** for PR 23504 at commit [`4400077`](https://github.com/apache/spark/commit/4400077ff8daf162cf7d1f7adcfc1bf5b77017f0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23502: [SPARK-22128][CORE][BUILD] Add `paranamer` dependency to `core` module
dongjoon-hyun commented on a change in pull request #23502: [SPARK-22128][CORE][BUILD] Add `paranamer` dependency to `core` module URL: https://github.com/apache/spark/pull/23502#discussion_r246659697 ## File path: core/pom.xml ## @@ -33,6 +33,10 @@ Spark Project Core http://spark.apache.org/ + + com.thoughtworks.paranamer Review comment: This is not a new dependency. - https://github.com/apache/spark/blob/master/pom.xml#L1948-L1952 The license is here. - https://github.com/apache/spark/blob/master/LICENSE-binary#L414 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] LantaoJin commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
LantaoJin commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-452999783 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun closed pull request #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure
dongjoon-hyun closed pull request #23505: [SPARK-26491][K8S][FOLLOWUP] Fix
compile failure
URL: https://github.com/apache/spark/pull/23505
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala
index cc89683949010..250eaab9e9808 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala
@@ -68,7 +68,7 @@ private[spark] class KubernetesTestComponents(defaultClient:
DefaultKubernetesCl
.set("spark.executors.instances", "1")
.set("spark.app.name", "spark-test-app")
.set("spark.ui.enabled", "true")
- .set(IS_TESTING, false)
+ .set(IS_TESTING.key, "false")
.set("spark.kubernetes.submission.waitAppCompletion", "false")
.set("spark.kubernetes.authenticate.driver.serviceAccountName",
serviceAccountName)
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] LantaoJin commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
LantaoJin commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-452999732 Fails by [Flaky Test: HiveClientSuites](https://issues.apache.org/jira/browse/SPARK-23622) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure
dongjoon-hyun commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure URL: https://github.com/apache/spark/pull/23505#issuecomment-452999024 Thank you for review and approval, @felixcheung . Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode
AmplabJenkins removed a comment on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode URL: https://github.com/apache/spark/pull/19616#issuecomment-452997772 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23327: [SPARK-26222][SQL] Track file listing time
AmplabJenkins removed a comment on issue #23327: [SPARK-26222][SQL] Track file listing time URL: https://github.com/apache/spark/pull/23327#issuecomment-452997438 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101005/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode
AmplabJenkins removed a comment on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode URL: https://github.com/apache/spark/pull/19616#issuecomment-452997774 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100997/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode
AmplabJenkins commented on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode URL: https://github.com/apache/spark/pull/19616#issuecomment-452997772 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23327: [SPARK-26222][SQL] Track file listing time
SparkQA removed a comment on issue #23327: [SPARK-26222][SQL] Track file listing time URL: https://github.com/apache/spark/pull/23327#issuecomment-452966816 **[Test build #101005 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101005/testReport)** for PR 23327 at commit [`16d80c5`](https://github.com/apache/spark/commit/16d80c504c807d3626da1ff808f3c4c447187ea3). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode
SparkQA removed a comment on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode URL: https://github.com/apache/spark/pull/19616#issuecomment-452944227 **[Test build #100997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100997/testReport)** for PR 19616 at commit [`dc31940`](https://github.com/apache/spark/commit/dc31940ab81e0f119303caf1dc9da99928d509a7). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
SparkQA commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-452997827 **[Test build #101008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101008/testReport)** for PR 23504 at commit [`4400077`](https://github.com/apache/spark/commit/4400077ff8daf162cf7d1f7adcfc1bf5b77017f0). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode
AmplabJenkins commented on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode URL: https://github.com/apache/spark/pull/19616#issuecomment-452997774 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100997/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
felixcheung commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-452997703 Jenkins, retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23327: [SPARK-26222][SQL] Track file listing time
AmplabJenkins removed a comment on issue #23327: [SPARK-26222][SQL] Track file listing time URL: https://github.com/apache/spark/pull/23327#issuecomment-452997434 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23502: [SPARK-22128][CORE][BUILD] Add `paranamer` dependency to `core` module
felixcheung commented on a change in pull request #23502: [SPARK-22128][CORE][BUILD] Add `paranamer` dependency to `core` module URL: https://github.com/apache/spark/pull/23502#discussion_r246657269 ## File path: core/pom.xml ## @@ -33,6 +33,10 @@ Spark Project Core http://spark.apache.org/ + + com.thoughtworks.paranamer Review comment: do we need to include this in license? looks like the project has non-standard license text This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode
SparkQA commented on issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode URL: https://github.com/apache/spark/pull/19616#issuecomment-452997362 **[Test build #100997 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100997/testReport)** for PR 19616 at commit [`dc31940`](https://github.com/apache/spark/commit/dc31940ab81e0f119303caf1dc9da99928d509a7). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23327: [SPARK-26222][SQL] Track file listing time
AmplabJenkins commented on issue #23327: [SPARK-26222][SQL] Track file listing time URL: https://github.com/apache/spark/pull/23327#issuecomment-452997438 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101005/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23327: [SPARK-26222][SQL] Track file listing time
AmplabJenkins commented on issue #23327: [SPARK-26222][SQL] Track file listing time URL: https://github.com/apache/spark/pull/23327#issuecomment-452997434 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23327: [SPARK-26222][SQL] Track file listing time
SparkQA commented on issue #23327: [SPARK-26222][SQL] Track file listing time URL: https://github.com/apache/spark/pull/23327#issuecomment-452997241 **[Test build #101005 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101005/testReport)** for PR 23327 at commit [`16d80c5`](https://github.com/apache/spark/commit/16d80c504c807d3626da1ff808f3c4c447187ea3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on a change in pull request #23498: [SPARK-26580][SQL] remove Scala 2.11 hack for Scala UDF
felixcheung commented on a change in pull request #23498: [SPARK-26580][SQL] remove Scala 2.11 hack for Scala UDF URL: https://github.com/apache/spark/pull/23498#discussion_r246656972 ## File path: docs/sql-migration-guide-upgrade.md ## @@ -43,6 +43,8 @@ displayTitle: Spark SQL Upgrading Guide - Since Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Set JSON option `inferTimestamp` to `false` to disable such type inferring. + - In Spark version 2.4 and earlier, if `org.apache.spark.sql.functions.udf(Any, DataType)` gets a Scala closure with primitive-type argument, the returned UDF will return null if the input values is null. Since Spark 3.0, the UDF will return the default value of the Java type if the input value is null. For example, `val f = udf((x: Int) => x, IntegerType)`, `f($"x")` will return null in Spark 2.4 and earlier if column `x` is null, and return 0 in Spark 3.0. Review comment: should this say this is because of Scala 2.12? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on a change in pull request #20303: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
cloud-fan commented on a change in pull request #20303: [SPARK-23128][SQL] A
new approach to do adaptive execution in Spark SQL
URL: https://github.com/apache/spark/pull/20303#discussion_r246655398
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStage.scala
##
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import scala.concurrent.{ExecutionContext, Future}
+import scala.concurrent.duration.Duration
+
+import org.apache.spark.MapOutputStatistics
+import org.apache.spark.broadcast
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical.Partitioning
+import org.apache.spark.sql.execution._
+import org.apache.spark.sql.execution.exchange._
+import
org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * In adaptive execution mode, an execution plan is divided into multiple
QueryStages. Each
+ * QueryStage is a sub-tree that runs in a single stage.
+ */
+abstract class QueryStage extends UnaryExecNode {
+
+ var child: SparkPlan
+
+ // Ignore this wrapper for canonicalizing.
+ override def doCanonicalize(): SparkPlan = child.canonicalized
+
+ override def output: Seq[Attribute] = child.output
+
+ override def outputPartitioning: Partitioning = child.outputPartitioning
+
+ override def outputOrdering: Seq[SortOrder] = child.outputOrdering
+
+ /**
+ * Execute childStages and wait until all stages are completed. Use a thread
pool to avoid
+ * blocking on one child stage.
+ */
+ def executeChildStages(): Unit = {
+// Handle broadcast stages
+val broadcastQueryStages: Seq[BroadcastQueryStage] = child.collect {
+ case bqs: BroadcastQueryStageInput => bqs.childStage
+}
+val broadcastFutures = broadcastQueryStages.map { queryStage =>
+ Future { queryStage.prepareBroadcast() }(QueryStage.executionContext)
+}
+
+// Submit shuffle stages
+val executionId =
sqlContext.sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY)
+val shuffleQueryStages: Seq[ShuffleQueryStage] = child.collect {
+ case sqs: ShuffleQueryStageInput => sqs.childStage
+}
+val shuffleStageFutures = shuffleQueryStages.map { queryStage =>
+ Future {
+SQLExecution.withExecutionId(sqlContext.sparkContext, executionId) {
+ queryStage.execute()
+}
+ }(QueryStage.executionContext)
+}
+
+ThreadUtils.awaitResult(
+ Future.sequence(broadcastFutures)(implicitly,
QueryStage.executionContext), Duration.Inf)
+ThreadUtils.awaitResult(
+ Future.sequence(shuffleStageFutures)(implicitly,
QueryStage.executionContext), Duration.Inf)
+ }
+
+ /**
+ * Before executing the plan in this query stage, we execute all child
stages, optimize the plan
+ * in this stage and determine the reducer number based on the child stages'
statistics. Finally
+ * we do a codegen for this query stage and update the UI with the new plan.
+ */
+ def prepareExecuteStage(): Unit = {
+// 1. Execute childStages
+executeChildStages()
+// It is possible to optimize this stage's plan here based on the child
stages' statistics.
+
+// 2. Determine reducer number
+val queryStageInputs: Seq[ShuffleQueryStageInput] = child.collect {
+ case input: ShuffleQueryStageInput => input
+}
+val childMapOutputStatistics =
queryStageInputs.map(_.childStage.mapOutputStatistics)
+ .filter(_ != null).toArray
+if (childMapOutputStatistics.length > 0) {
+ val exchangeCoordinator = new ExchangeCoordinator(
+conf.targetPostShuffleInputSize,
+conf.minNumPostShufflePartitions)
+
+ val partitionStartIndices =
+
exchangeCoordinator.estimatePartitionStartIndices(childMapOutputStatistics)
+ child = child.transform {
+case ShuffleQueryStageInput(childStage, output, _) =>
+ ShuffleQueryStageInput(childStage, output,
Some(partitionStartIndices))
+ }
+}
+
+// 3. Codegen and update the UI
+
[GitHub] cloud-fan commented on a change in pull request #20303: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
cloud-fan commented on a change in pull request #20303: [SPARK-23128][SQL] A
new approach to do adaptive execution in Spark SQL
URL: https://github.com/apache/spark/pull/20303#discussion_r246654084
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanQueryStage.scala
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.command.ExecutedCommandExec
+import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec,
Exchange, ShuffleExchangeExec}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Divide the spark plan into multiple QueryStages. For each Exchange in the
plan, it adds a
+ * QueryStage and a QueryStageInput. If reusing Exchange is enabled, it finds
duplicated exchanges
+ * and uses the same QueryStage for all the references.
Review comment:
add a comment that this rule must be run after `EnsureRequirements`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on a change in pull request #20303: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
cloud-fan commented on a change in pull request #20303: [SPARK-23128][SQL] A
new approach to do adaptive execution in Spark SQL
URL: https://github.com/apache/spark/pull/20303#discussion_r246654954
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStage.scala
##
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import scala.concurrent.{ExecutionContext, Future}
+import scala.concurrent.duration.Duration
+
+import org.apache.spark.MapOutputStatistics
+import org.apache.spark.broadcast
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical.Partitioning
+import org.apache.spark.sql.execution._
+import org.apache.spark.sql.execution.exchange._
+import
org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * In adaptive execution mode, an execution plan is divided into multiple
QueryStages. Each
+ * QueryStage is a sub-tree that runs in a single stage.
+ */
+abstract class QueryStage extends UnaryExecNode {
+
+ var child: SparkPlan
+
+ // Ignore this wrapper for canonicalizing.
+ override def doCanonicalize(): SparkPlan = child.canonicalized
+
+ override def output: Seq[Attribute] = child.output
+
+ override def outputPartitioning: Partitioning = child.outputPartitioning
+
+ override def outputOrdering: Seq[SortOrder] = child.outputOrdering
+
+ /**
+ * Execute childStages and wait until all stages are completed. Use a thread
pool to avoid
+ * blocking on one child stage.
+ */
+ def executeChildStages(): Unit = {
+// Handle broadcast stages
+val broadcastQueryStages: Seq[BroadcastQueryStage] = child.collect {
+ case bqs: BroadcastQueryStageInput => bqs.childStage
+}
+val broadcastFutures = broadcastQueryStages.map { queryStage =>
+ Future { queryStage.prepareBroadcast() }(QueryStage.executionContext)
+}
+
+// Submit shuffle stages
+val executionId =
sqlContext.sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY)
+val shuffleQueryStages: Seq[ShuffleQueryStage] = child.collect {
+ case sqs: ShuffleQueryStageInput => sqs.childStage
+}
+val shuffleStageFutures = shuffleQueryStages.map { queryStage =>
+ Future {
+SQLExecution.withExecutionId(sqlContext.sparkContext, executionId) {
+ queryStage.execute()
+}
+ }(QueryStage.executionContext)
+}
+
+ThreadUtils.awaitResult(
+ Future.sequence(broadcastFutures)(implicitly,
QueryStage.executionContext), Duration.Inf)
+ThreadUtils.awaitResult(
+ Future.sequence(shuffleStageFutures)(implicitly,
QueryStage.executionContext), Duration.Inf)
+ }
+
+ /**
+ * Before executing the plan in this query stage, we execute all child
stages, optimize the plan
+ * in this stage and determine the reducer number based on the child stages'
statistics. Finally
+ * we do a codegen for this query stage and update the UI with the new plan.
+ */
+ def prepareExecuteStage(): Unit = {
+// 1. Execute childStages
+executeChildStages()
+// It is possible to optimize this stage's plan here based on the child
stages' statistics.
+
+// 2. Determine reducer number
+val queryStageInputs: Seq[ShuffleQueryStageInput] = child.collect {
+ case input: ShuffleQueryStageInput => input
+}
+val childMapOutputStatistics =
queryStageInputs.map(_.childStage.mapOutputStatistics)
+ .filter(_ != null).toArray
Review comment:
when `mapOutputStatistics` can be null?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
[GitHub] dongjoon-hyun edited a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
dongjoon-hyun edited a comment on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-452995851 Hi, @ramaddepally . The Kubernetes integration test failure is irrelevant to this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend
dongjoon-hyun commented on issue #23504: [SPARK-26585][K8S] Add additional integration tests for K8s Scheduler Backend URL: https://github.com/apache/spark/pull/23504#issuecomment-452995851 Hi, @ramaddepally . The failure is irrelevant to this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun edited a comment on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
dongjoon-hyun edited a comment on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#issuecomment-452995469 Hi, @Deegue . Thank you for your first contribution. Please update both PRs (this one and the previous one). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
dongjoon-hyun commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#issuecomment-452995469 Hi, @Deegue . Thank you for your first contribution. Please update this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
AmplabJenkins removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-452995005 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101001/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
dongjoon-hyun commented on a change in pull request #23506: [SPARK-26577][SQL]
Add input optimizer when reading Hive table by SparkSQL
URL: https://github.com/apache/spark/pull/23506#discussion_r246655454
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveTableScanSuite.scala
##
@@ -192,4 +192,54 @@ class HiveTableScanSuite extends HiveComparisonTest with
SQLTestUtils with TestH
case p: HiveTableScanExec => p
}.get
}
-}
+
+ test("Test the InputFormat optimizer") {
+withTable("table_old", "table_pt_old", "table_new", "table_pt_new") {
+ sql("set spark.sql.hive.fileInputFormat.enabled=true")
+ sql("set spark.sql.hive.fileInputFormat.split.maxsize=134217728")
+ sql("set spark.sql.hive.fileInputFormat.split.minsize=134217728")
+ sql(
+s"""
+ |CREATE TABLE table_old (id int)
+ |STORED AS
+ |INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
+ |OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
+ """.stripMargin)
+ sql(
+s"""
+ |CREATE TABLE table_pt_old (id int)
+ |PARTITIONED BY (a int, b int)
+ |STORED AS
+ |INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
+ |OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
+ """.stripMargin)
+ sql(
+s"""
+ |CREATE TABLE table_new (id int)
+ |STORED AS
+ |INPUTFORMAT 'org.apache.hadoop.mapreduce.lib.input.TextInputFormat'
+ |OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
+ """.stripMargin)
+ sql(
+s"""
+ |CREATE TABLE table_pt_new (id int)
+ |PARTITIONED BY (a int, b int)
+ |STORED AS
+ |INPUTFORMAT 'org.apache.hadoop.mapreduce.lib.input.TextInputFormat'
+ |OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
+ """.stripMargin)
+ intercept[Exception] {
+sql("SELECT count(1) FROM table_old")
+ }
Review comment:
We had better check the specific exception (instead of `Exception`) and the
error message. For example, the following pattern is used.
```scala
val m1 = intercept[AnalysisException] {
sql("SELECT count(1) FROM table_old")
}.getMessage
assert(m1.contains("Your error message"))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
AmplabJenkins commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-452995003 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
AmplabJenkins commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-452995005 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101001/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
AmplabJenkins removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-452995003 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
SparkQA commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-452994781 **[Test build #101001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101001/testReport)** for PR 22874 at commit [`3d9a02c`](https://github.com/apache/spark/commit/3d9a02c8a9e1ceba3f097bd76399616c136892a9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
AmplabJenkins removed a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-452994710 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
AmplabJenkins removed a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-452994716 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101000/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
SparkQA removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-452957976 **[Test build #101001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101001/testReport)** for PR 22874 at commit [`3d9a02c`](https://github.com/apache/spark/commit/3d9a02c8a9e1ceba3f097bd76399616c136892a9). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
AmplabJenkins commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-452994710 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
AmplabJenkins commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-452994716 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101000/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
dongjoon-hyun commented on a change in pull request #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#discussion_r246654460 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ## @@ -311,6 +309,35 @@ class HadoopTableReader( // Only take the value (skip the key) because Hive works only with values. rdd.map(_._2) } + + /** +* If `spark.sql.hive.fileInputFormat.enabled` is true, this function will optimize the input Review comment: nit. indentation? Also, [line 318](https://github.com/apache/spark/pull/23506/files#diff-8887a877bd52611df9aea06ccfe3a2d7R318)? Please check the style. It seems to be broken due to copy This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
SparkQA removed a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-452957052 **[Test build #101000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101000/testReport)** for PR 23208 at commit [`ee7acbc`](https://github.com/apache/spark/commit/ee7acbced67d3d084b1479b8303fe815701893ef). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
dongjoon-hyun commented on a change in pull request #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#discussion_r246654460 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ## @@ -311,6 +309,35 @@ class HadoopTableReader( // Only take the value (skip the key) because Hive works only with values. rdd.map(_._2) } + + /** +* If `spark.sql.hive.fileInputFormat.enabled` is true, this function will optimize the input Review comment: nit. indentation? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
SparkQA commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-452994268 **[Test build #101000 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101000/testReport)** for PR 23208 at commit [`ee7acbc`](https://github.com/apache/spark/commit/ee7acbced67d3d084b1479b8303fe815701893ef). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability
AmplabJenkins removed a comment on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability URL: https://github.com/apache/spark/pull/23390#issuecomment-452989298 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability
AmplabJenkins commented on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability URL: https://github.com/apache/spark/pull/23390#issuecomment-452989298 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability
AmplabJenkins removed a comment on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability URL: https://github.com/apache/spark/pull/23390#issuecomment-452989301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100999/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability
AmplabJenkins commented on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability URL: https://github.com/apache/spark/pull/23390#issuecomment-452989301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100999/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability
SparkQA removed a comment on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability URL: https://github.com/apache/spark/pull/23390#issuecomment-452950845 **[Test build #100999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100999/testReport)** for PR 23390 at commit [`bcb5667`](https://github.com/apache/spark/commit/bcb56670c4d35df880ff7b4dbe1d29d1359e7791). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability
SparkQA commented on issue #23390: [SPARK-26459][SQL] replace UpdateNullabilityInAttributeReferences with FixNullability URL: https://github.com/apache/spark/pull/23390#issuecomment-452988931 **[Test build #100999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100999/testReport)** for PR 23390 at commit [`bcb5667`](https://github.com/apache/spark/commit/bcb56670c4d35df880ff7b4dbe1d29d1359e7791). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] Deegue commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
Deegue commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#issuecomment-452988195 Thanks @gatorsmile and I wonder if it would be better to change the conf name to `spark.sql.hive.input.optimizer.enable` , `spark.sql.hive.input.split.maxsize` and `spark.sql.hive.input.split.minsize` ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
AmplabJenkins removed a comment on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#issuecomment-452986459 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
AmplabJenkins commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#issuecomment-452986740 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
AmplabJenkins removed a comment on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#issuecomment-452986402 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] Deegue commented on issue #23496: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
Deegue commented on issue #23496: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23496#issuecomment-452986433 > Let me think about the conf naming. Could you first rebase it to master branch? Now it is targeting to 2.4. Thank you, I've picked the changes to master branch and create a new PR. #23506 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
AmplabJenkins commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#issuecomment-452986459 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
AmplabJenkins commented on issue #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506#issuecomment-452986402 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] Deegue opened a new pull request #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
Deegue opened a new pull request #23506: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL URL: https://github.com/apache/spark/pull/23506 ## What changes were proposed in this pull request? When using SparkSQL, for example the ThriftServer, if we set `spark.sql.hive.fileInputFormat.enabled=true` we can optimize the InputFormat to CombineTextInputFormat automatically if it's TextInputFormat before. And we can also change the max/min size of input splits by setting, for example `spark.sql.hive.fileInputFormat.split.maxsize=268435456` `spark.sql.hive.fileInputFormat.split.minsize=134217728` Otherwise, we have to modify Hive Configs and structure of tables. And we made a test by using a Hive table with a lot of small files in HDFS and haven't combined : Before improved:  After improved:  ## How was this patch tested? Added a test. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] Deegue commented on a change in pull request #23496: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
Deegue commented on a change in pull request #23496: [SPARK-26577][SQL] Add
input optimizer when reading Hive table by SparkSQL
URL: https://github.com/apache/spark/pull/23496#discussion_r246647183
##
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
##
@@ -123,8 +123,26 @@ class HadoopTableReader(
val inputPathStr = applyFilterIfNeeded(tablePath, filterOpt)
// logDebug("Table input: %s".format(tablePath))
-val ifc = hiveTable.getInputFormatClass
+var ifc = hiveTable.getInputFormatClass
.asInstanceOf[java.lang.Class[InputFormat[Writable, Writable]]]
+if (conf.fileInputFormatEnabled) {
+ hadoopConf.set("mapreduce.input.fileinputformat.split.maxsize",
Review comment:
sorry for that, done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf
AmplabJenkins removed a comment on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf URL: https://github.com/apache/spark/pull/23503#issuecomment-452984841 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf
AmplabJenkins commented on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf URL: https://github.com/apache/spark/pull/23503#issuecomment-452984846 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100998/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf
AmplabJenkins removed a comment on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf URL: https://github.com/apache/spark/pull/23503#issuecomment-452984846 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100998/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] Deegue commented on a change in pull request #23496: [SPARK-26577][SQL] Add input optimizer when reading Hive table by SparkSQL
Deegue commented on a change in pull request #23496: [SPARK-26577][SQL] Add
input optimizer when reading Hive table by SparkSQL
URL: https://github.com/apache/spark/pull/23496#discussion_r246647067
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -561,6 +561,23 @@ object SQLConf {
.checkValues(HiveCaseSensitiveInferenceMode.values.map(_.toString))
.createWithDefault(HiveCaseSensitiveInferenceMode.INFER_AND_SAVE.toString)
+ val HIVE_FILE_INPUT_FORMAT_ENABLED =
buildConf("spark.sql.hive.fileInputFormat.enabled")
Review comment:
yes I see, done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf
AmplabJenkins commented on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf URL: https://github.com/apache/spark/pull/23503#issuecomment-452984841 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf
SparkQA removed a comment on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf URL: https://github.com/apache/spark/pull/23503#issuecomment-452945229 **[Test build #100998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100998/testReport)** for PR 23503 at commit [`e2574c7`](https://github.com/apache/spark/commit/e2574c77b399a8787182c214b53cbf054a7b97f1). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf
SparkQA commented on issue #23503: [SPARK-26584][SQL] Remove `spark.sql.orc.copyBatchToSpark` internal conf URL: https://github.com/apache/spark/pull/23503#issuecomment-452984520 **[Test build #100998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100998/testReport)** for PR 23503 at commit [`e2574c7`](https://github.com/apache/spark/commit/e2574c77b399a8787182c214b53cbf054a7b97f1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] jzhuge commented on a change in pull request #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
jzhuge commented on a change in pull request #21978: [SPARK-25006][SQL] Add
CatalogTableIdentifier.
URL: https://github.com/apache/spark/pull/21978#discussion_r246645835
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/identifiers.scala
##
@@ -18,30 +18,32 @@
package org.apache.spark.sql.catalyst
/**
- * An identifier that optionally specifies a database.
+ * An identifier that optionally specifies a database and catalog.
*
* Format (unquoted): "name" or "db.name"
* Format (quoted): "`name`" or "`db`.`name`"
*/
-sealed trait IdentifierWithDatabase {
+sealed trait IdentifierWithOptionalDatabaseAndCatalog {
val identifier: String
def database: Option[String]
+ def catalog: Option[String]
Review comment:
Default to `None`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure
dongjoon-hyun commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure URL: https://github.com/apache/spark/pull/23505#issuecomment-452983429 K8S integration test passed. cc @vanzin and @mgaido91 . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories
HeartSaVioR commented on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories URL: https://github.com/apache/spark/pull/23423#issuecomment-452976138 @kiszk @vanzin I guess I addressed review comments and the build is green. Could you please take a look again? Thanks in advance! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR edited a comment on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories
HeartSaVioR edited a comment on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories URL: https://github.com/apache/spark/pull/23423#issuecomment-452976138 @vanzin @kiszk I guess I addressed review comments and the build is green. Could you please take a look again? Thanks in advance! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories
AmplabJenkins removed a comment on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories URL: https://github.com/apache/spark/pull/23423#issuecomment-452976044 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100994/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories
AmplabJenkins commented on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories URL: https://github.com/apache/spark/pull/23423#issuecomment-452976044 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100994/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories
AmplabJenkins commented on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories URL: https://github.com/apache/spark/pull/23423#issuecomment-452976040 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories
AmplabJenkins removed a comment on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories URL: https://github.com/apache/spark/pull/23423#issuecomment-452976040 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories
SparkQA removed a comment on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories URL: https://github.com/apache/spark/pull/23423#issuecomment-452927381 **[Test build #100994 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100994/testReport)** for PR 23423 at commit [`8d61242`](https://github.com/apache/spark/commit/8d61242aa85661592fc530a7cd5a8956f6f75bb8). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories
SparkQA commented on issue #23423: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories URL: https://github.com/apache/spark/pull/23423#issuecomment-452975754 **[Test build #100994 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100994/testReport)** for PR 23423 at commit [`8d61242`](https://github.com/apache/spark/commit/8d61242aa85661592fc530a7cd5a8956f6f75bb8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] felixcheung commented on issue #23492: [SPARK-26565][BUILD] skip gpg signing/svn publish for jenkins package builds
felixcheung commented on issue #23492: [SPARK-26565][BUILD] skip gpg signing/svn publish for jenkins package builds URL: https://github.com/apache/spark/pull/23492#issuecomment-452975088 I’d defer to you and others to decide whether keeping the output artifacts can be useful... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure
AmplabJenkins commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure URL: https://github.com/apache/spark/pull/23505#issuecomment-452974845 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure
AmplabJenkins removed a comment on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure URL: https://github.com/apache/spark/pull/23505#issuecomment-452974845 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure
AmplabJenkins commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure URL: https://github.com/apache/spark/pull/23505#issuecomment-452974847 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6884/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure
AmplabJenkins removed a comment on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure URL: https://github.com/apache/spark/pull/23505#issuecomment-452974847 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6884/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure
SparkQA commented on issue #23505: [SPARK-26491][K8S][FOLLOWUP] Fix compile failure URL: https://github.com/apache/spark/pull/23505#issuecomment-452974834 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/6884/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
