[GitHub] [spark] AmplabJenkins commented on pull request #28834: [SPARK-31994][k8s] Docker image should use `https` urls for only deb.debian.org mirrors.
AmplabJenkins commented on pull request #28834: URL: https://github.com/apache/spark/pull/28834#issuecomment-644051683 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
AmplabJenkins removed a comment on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644064207 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
AmplabJenkins commented on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644064207 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
SparkQA commented on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644063697 **[Test build #124052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124052/testReport)** for PR 28832 at commit [`55e18b6`](https://github.com/apache/spark/commit/55e18b602a532adecc7f39323f6c8e5990bb528f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait
AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644077059 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124045/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait
AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644077050 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
AmplabJenkins commented on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644092796 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
AmplabJenkins removed a comment on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644092796 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
AmplabJenkins removed a comment on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644092814 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124046/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
SparkQA commented on pull request #28642: URL: https://github.com/apache/spark/pull/28642#issuecomment-644098962 **[Test build #124038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124038/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0ca9bae7c524b40c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
SparkQA removed a comment on pull request #28642: URL: https://github.com/apache/spark/pull/28642#issuecomment-643948913 **[Test build #124038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124038/testReport)** for PR 28642 at commit [`65cd324`](https://github.com/apache/spark/commit/65cd324093fac15357fb0ca9bae7c524b40c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
AmplabJenkins commented on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644050687 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
AmplabJenkins removed a comment on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644050695 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124047/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors
attilapiros commented on a change in pull request #28619:
URL: https://github.com/apache/spark/pull/28619#discussion_r440087128
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -1842,6 +1842,17 @@ package object config {
.timeConf(TimeUnit.MILLISECONDS)
.createOptional
+ private[spark] val EXECUTOR_DECOMMISSION_KILL_INTERVAL =
+ConfigBuilder("spark.executor.decommission.killInterval")
+ .doc("Duration after which a decommissioned executor will be killed
forcefully." +
+"This config is useful for cloud environments where we know in advance
when " +
+"an executor is going to go down after decommissioning signal Ex-
around 2 mins " +
Review comment:
Nit: does the Ex in the "go down after decommissioning signal Ex- around
2 mins" stands for "example"?
Sorry I have not seen this abbreviation used before and still not sure it
exists. What about "i.e." that even used in Spark documentation several times?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28834: [SPARK-31994][k8s] Docker image should use `https` urls for only deb.debian.org mirrors.
SparkQA commented on pull request #28834: URL: https://github.com/apache/spark/pull/28834#issuecomment-644051449 **[Test build #124050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124050/testReport)** for PR 28834 at commit [`9ef5b94`](https://github.com/apache/spark/commit/9ef5b946c40991c63fc241cf18b17217d4829826). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors
attilapiros commented on a change in pull request #28619:
URL: https://github.com/apache/spark/pull/28619#discussion_r440087518
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -1842,6 +1842,17 @@ package object config {
.timeConf(TimeUnit.MILLISECONDS)
.createOptional
+ private[spark] val EXECUTOR_DECOMMISSION_KILL_INTERVAL =
+ConfigBuilder("spark.executor.decommission.killInterval")
+ .doc("Duration after which a decommissioned executor will be killed
forcefully." +
+"This config is useful for cloud environments where we know in advance
when " +
+"an executor is going to go down after decommissioning signal Ex-
around 2 mins " +
+"in aws spot nodes, 1/2 hrs in spot block nodes etc. This config is
currently " +
+"used to decide what tasks running on decommission executors to
speculate")
Review comment:
Nit: Missing dot from the end of sentence.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
AmplabJenkins removed a comment on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644050687 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
SparkQA commented on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644050559 **[Test build #124047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124047/testReport)** for PR 28832 at commit [`55e18b6`](https://github.com/apache/spark/commit/55e18b602a532adecc7f39323f6c8e5990bb528f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
AmplabJenkins commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-644070471 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WangGuangxin commented on a change in pull request #28780: [SPARK-31952][SQL]Fix incorrect memory spill metric when doing Aggregate
WangGuangxin commented on a change in pull request #28780:
URL: https://github.com/apache/spark/pull/28780#discussion_r440108353
##
File path:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
##
@@ -104,11 +104,13 @@ public static UnsafeExternalSorter
createWithExistingInMemorySorter(
int initialSize,
long pageSizeBytes,
int numElementsForSpillThreshold,
- UnsafeInMemorySorter inMemorySorter) throws IOException {
+ UnsafeInMemorySorter inMemorySorter,
+ long existingMemoryConsumption) throws IOException {
UnsafeExternalSorter sorter = new UnsafeExternalSorter(taskMemoryManager,
blockManager,
serializerManager, taskContext, recordComparatorSupplier,
prefixComparator, initialSize,
pageSizeBytes, numElementsForSpillThreshold, inMemorySorter, false /*
ignored */);
sorter.spill(Long.MAX_VALUE, sorter);
+taskContext.taskMetrics().incMemoryBytesSpilled(existingMemoryConsumption);
Review comment:
cc @maropu @cloud-fan
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
AmplabJenkins removed a comment on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-644070471 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
HyukjinKwon commented on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644071006 cc @maropu, @cloud-fan, @dongjoon-hyun, @hvanhovell, @gatorsmile FYI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28834: [SPARK-31994][k8s] Docker image should use `https` urls for only deb.debian.org mirrors.
SparkQA commented on pull request #28834: URL: https://github.com/apache/spark/pull/28834#issuecomment-644074146 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28669/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28834: [SPARK-31994][k8s] Docker image should use `https` urls for only deb.debian.org mirrors.
AmplabJenkins removed a comment on pull request #28834: URL: https://github.com/apache/spark/pull/28834#issuecomment-644074165 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28834: [SPARK-31994][k8s] Docker image should use `https` urls for only deb.debian.org mirrors.
AmplabJenkins commented on pull request #28834: URL: https://github.com/apache/spark/pull/28834#issuecomment-644074165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait
SparkQA removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-643991785 **[Test build #124045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124045/testReport)** for PR 28710 at commit [`0898708`](https://github.com/apache/spark/commit/08987080d624a86379f6ec77e65571db95f36e13). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait
AmplabJenkins commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644077050 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait
SparkQA commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-644076601 **[Test build #124045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124045/testReport)** for PR 28710 at commit [`0898708`](https://github.com/apache/spark/commit/08987080d624a86379f6ec77e65571db95f36e13). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
SparkQA commented on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644091950 **[Test build #124046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124046/testReport)** for PR 28833 at commit [`3b8ddec`](https://github.com/apache/spark/commit/3b8ddecc7a1498e1e430be7de1ae76123b269454). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
SparkQA removed a comment on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-643995551 **[Test build #124046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124046/testReport)** for PR 28833 at commit [`3b8ddec`](https://github.com/apache/spark/commit/3b8ddecc7a1498e1e430be7de1ae76123b269454). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode
SparkQA commented on pull request #28746: URL: https://github.com/apache/spark/pull/28746#issuecomment-644099322 **[Test build #124054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124054/testReport)** for PR 28746 at commit [`99fad37`](https://github.com/apache/spark/commit/99fad3769c74550c7ee2be0d81d0e3541a556aa7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode
AmplabJenkins commented on pull request #28746: URL: https://github.com/apache/spark/pull/28746#issuecomment-644099791 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode
AmplabJenkins removed a comment on pull request #28746: URL: https://github.com/apache/spark/pull/28746#issuecomment-644099791 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28829: [SPARK-31992][SQL] Benchmark the EXCEPTION rebase mode
SparkQA commented on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-644108675 **[Test build #124039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124039/testReport)** for PR 28829 at commit [`16e90be`](https://github.com/apache/spark/commit/16e90bebf9314105d20c581a07120adb6d288e0b). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp
SparkQA removed a comment on pull request #28821: URL: https://github.com/apache/spark/pull/28821#issuecomment-644037334 **[Test build #124049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124049/testReport)** for PR 28821 at commit [`83be227`](https://github.com/apache/spark/commit/83be227b21faf4007df4475d75944afdb211ea84). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28829: [SPARK-31992][SQL] Benchmark the EXCEPTION rebase mode
AmplabJenkins removed a comment on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-644109622 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124039/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp
SparkQA commented on pull request #28821: URL: https://github.com/apache/spark/pull/28821#issuecomment-644109882 **[Test build #124049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124049/testReport)** for PR 28821 at commit [`83be227`](https://github.com/apache/spark/commit/83be227b21faf4007df4475d75944afdb211ea84). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp
AmplabJenkins removed a comment on pull request #28821: URL: https://github.com/apache/spark/pull/28821#issuecomment-644110363 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp
AmplabJenkins commented on pull request #28821: URL: https://github.com/apache/spark/pull/28821#issuecomment-644110363 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28834: [SPARK-31994][k8s] Docker image should use `https` urls for only deb.debian.org mirrors.
SparkQA commented on pull request #28834: URL: https://github.com/apache/spark/pull/28834#issuecomment-644062357 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28669/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
HyukjinKwon commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-644069380 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
AmplabJenkins commented on pull request #28642: URL: https://github.com/apache/spark/pull/28642#issuecomment-644100154 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition
AmplabJenkins removed a comment on pull request #28642: URL: https://github.com/apache/spark/pull/28642#issuecomment-644100154 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode
SparkQA commented on pull request #28746: URL: https://github.com/apache/spark/pull/28746#issuecomment-644104160 **[Test build #124055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124055/testReport)** for PR 28746 at commit [`ddcadfd`](https://github.com/apache/spark/commit/ddcadfdbfa1a79b2d019e5afdeff605b26d87987). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28829: [SPARK-31992][SQL] Benchmark the EXCEPTION rebase mode
AmplabJenkins commented on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-644109611 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28829: [SPARK-31992][SQL] Benchmark the EXCEPTION rebase mode
AmplabJenkins removed a comment on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-644109611 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28829: [SPARK-31992][SQL] Benchmark the EXCEPTION rebase mode
SparkQA removed a comment on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-643952127 **[Test build #124039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124039/testReport)** for PR 28829 at commit [`16e90be`](https://github.com/apache/spark/commit/16e90bebf9314105d20c581a07120adb6d288e0b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28831: [SPARK-31993][SQL] Don't split code blocks in generated code for 'concat_ws' for mixed string/array types of columns
HeartSaVioR commented on pull request #28831: URL: https://github.com/apache/spark/pull/28831#issuecomment-644120378 My bad I missed the existing test, as I target on older Spark version originally. The existing UT works with this patch, because we don't create local variables for dealing with literals (it just seems to leverage global array.) Looks like we should deal with splitting code block in any way. Let me change the PR to WIP for now and try to come up with another solution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28645: [SPARK-31826][SQL] Support composed type of case class for typed Scala UDF
HyukjinKwon commented on a change in pull request #28645:
URL: https://github.com/apache/spark/pull/28645#discussion_r440098735
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -2847,6 +2848,45 @@ class Analyzer(
}
}
+ /**
+ * Resolve the encoders for the UDF by explicitly given the attributes. We
give the
+ * attributes explicitly in order to handle the case where the data type of
the input
+ * value is not the same with the internal schema of the encoder, which
could cause
+ * data loss. For example, the encoder should not cast the input value to
Decimal(38, 18)
+ * if the actual data type is Decimal(30, 0).
+ *
+ * The resolved encoders then will be used to deserialize the internal row
to Scala value.
+ */
+ object ResolveEncodersInUDF extends Rule[LogicalPlan] {
+override def apply(plan: LogicalPlan): LogicalPlan =
plan.resolveOperatorsUp {
+ case p if !p.resolved => p // Skip unresolved nodes.
+
+ case p => p transformExpressionsUp {
+
+case udf @ ScalaUDF(_, _, inputs, encoders, _, _, _) if
encoders.nonEmpty =>
Review comment:
Shall we avoid argument matching? It's actually an anti-pattern -
https://github.com/databricks/scala-style-guide#pattern-matching
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
MaxGekk commented on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644061874 jenkins, retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
MaxGekk commented on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644061746 The issue https://github.com/apache/spark/pull/28832#issuecomment-644050559 is fixed by https://github.com/apache/spark/pull/28816 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors
attilapiros commented on a change in pull request #28619:
URL: https://github.com/apache/spark/pull/28619#discussion_r440107393
##
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
##
@@ -1042,7 +1046,19 @@ private[spark] class TaskSetManager(
// bound based on that.
logDebug("Task length threshold for speculation: " + threshold)
for (tid <- runningTasksSet) {
-foundTasks |= checkAndSubmitSpeculatableTask(tid, time, threshold)
+var speculated = checkAndSubmitSpeculatableTask(tid, time, threshold)
+if (!speculated && tidToExecutorKillTimeMapping.contains(tid)) {
Review comment:
I am just curious why this solution (introducing the kill time per task
ID map: `tidToExecutorKillTimeMapping`) is chosen over storing the kill time
per executor (`executorToKillTimeMapping` or something like that). In the later
case here would be something like:
```scala
if (!speculated && executorToKillTimeMapping.nonEmpty) {
val taskInfo = taskInfos(tid)
executorToKillTimeMapping.get(taskInfo.executorId).foreach {
executorKillTime =>
...
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
SparkQA commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-644070017 **[Test build #124053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124053/testReport)** for PR 28412 at commit [`76dbd18`](https://github.com/apache/spark/commit/76dbd184b659c2eeccba56f8201d470d939fceb4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
HyukjinKwon commented on a change in pull request #28833:
URL: https://github.com/apache/spark/pull/28833#discussion_r440111404
##
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/NullType.scala
##
@@ -32,6 +32,11 @@ class NullType private() extends DataType {
override def defaultSize: Int = 1
private[spark] override def asNullable: NullType = this
+
+ /**
+ * Readable string representation for NULL type.
+ */
+ override def simpleString: String = "void"
Review comment:
We should also override `simpleString` at `NullType` in PySpark side
manually
https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L111
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28834: [SPARK-31994][k8s] Docker image should use `https` urls for only deb.debian.org mirrors.
AmplabJenkins removed a comment on pull request #28834: URL: https://github.com/apache/spark/pull/28834#issuecomment-644074175 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28669/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp
AmplabJenkins removed a comment on pull request #28821: URL: https://github.com/apache/spark/pull/28821#issuecomment-644110376 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124049/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
AmplabJenkins commented on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644123442 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
AmplabJenkins removed a comment on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644123442 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
LantaoJin commented on a change in pull request #28833:
URL: https://github.com/apache/spark/pull/28833#discussion_r440161409
##
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/NullType.scala
##
@@ -32,6 +32,11 @@ class NullType private() extends DataType {
override def defaultSize: Int = 1
private[spark] override def asNullable: NullType = this
+
+ /**
+ * Readable string representation for NULL type.
+ */
+ override def simpleString: String = "void"
Review comment:
added, thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28833: [SPARK-20680][SQL] Spark-sql do not support for void column datatype
SparkQA commented on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-644126758 **[Test build #124056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124056/testReport)** for PR 28833 at commit [`cf0db98`](https://github.com/apache/spark/commit/cf0db989206e2d79fe747439284c181c2575551b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode
SparkQA commented on pull request #28746: URL: https://github.com/apache/spark/pull/28746#issuecomment-644130897 **[Test build #124057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124057/testReport)** for PR 28746 at commit [`a3ed5c1`](https://github.com/apache/spark/commit/a3ed5c1e213e67b6ba84a2e9b3d487cf766c2704). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28830: [SPARK-31990][SS] Use toSet.toSeq in Dataset.dropDuplicates
AmplabJenkins commented on pull request #28830: URL: https://github.com/apache/spark/pull/28830#issuecomment-644133430 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28830: [SPARK-31990][SS] Use toSet.toSeq in Dataset.dropDuplicates
AmplabJenkins removed a comment on pull request #28830: URL: https://github.com/apache/spark/pull/28830#issuecomment-644133430 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28807: [SPARK-26905][SQL] Follow the SQL:2016 reserved keywords
AmplabJenkins removed a comment on pull request #28807: URL: https://github.com/apache/spark/pull/28807#issuecomment-644141806 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28807: [SPARK-26905][SQL] Follow the SQL:2016 reserved keywords
AmplabJenkins commented on pull request #28807: URL: https://github.com/apache/spark/pull/28807#issuecomment-644141806 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28784: [SPARK-31957][SQL][test-maven] Cleanup hive scratch dir for the developer api startWithContext
AmplabJenkins removed a comment on pull request #28784: URL: https://github.com/apache/spark/pull/28784#issuecomment-644154231 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124037/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28784: [SPARK-31957][SQL][test-maven] Cleanup hive scratch dir for the developer api startWithContext
AmplabJenkins removed a comment on pull request #28784: URL: https://github.com/apache/spark/pull/28784#issuecomment-644154215 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
AmplabJenkins removed a comment on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644157275 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] prakharjain09 commented on a change in pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors
prakharjain09 commented on a change in pull request #28619:
URL: https://github.com/apache/spark/pull/28619#discussion_r440206799
##
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
##
@@ -1042,7 +1046,19 @@ private[spark] class TaskSetManager(
// bound based on that.
logDebug("Task length threshold for speculation: " + threshold)
for (tid <- runningTasksSet) {
-foundTasks |= checkAndSubmitSpeculatableTask(tid, time, threshold)
+var speculated = checkAndSubmitSpeculatableTask(tid, time, threshold)
+if (!speculated && tidToExecutorKillTimeMapping.contains(tid)) {
Review comment:
Yes - We can store executorToKillTimeMapping also. In that case, we have
to clean the map when executor is removed. For that we may have to make changes
to pass "removeExecutor" signal till TaskSetManager.
I didn't see much difference in both the approaches as total entries we will
be creating for a given executor in current approach will be equal to total
tasks running for that executor (which shouldn't be a lot).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] prakharjain09 commented on a change in pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors
prakharjain09 commented on a change in pull request #28619:
URL: https://github.com/apache/spark/pull/28619#discussion_r440207015
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -1842,6 +1842,17 @@ package object config {
.timeConf(TimeUnit.MILLISECONDS)
.createOptional
+ private[spark] val EXECUTOR_DECOMMISSION_KILL_INTERVAL =
+ConfigBuilder("spark.executor.decommission.killInterval")
+ .doc("Duration after which a decommissioned executor will be killed
forcefully." +
+"This config is useful for cloud environments where we know in advance
when " +
+"an executor is going to go down after decommissioning signal Ex-
around 2 mins " +
Review comment:
changed to "i.e.".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors
AmplabJenkins removed a comment on pull request #28619: URL: https://github.com/apache/spark/pull/28619#issuecomment-644164225 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors
AmplabJenkins commented on pull request #28619: URL: https://github.com/apache/spark/pull/28619#issuecomment-644164225 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] prakharjain09 commented on pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors
prakharjain09 commented on pull request #28619: URL: https://github.com/apache/spark/pull/28619#issuecomment-644163889 @attilapiros Thanks for the review. > One more question: the checkAndSubmitSpeculatableTask checks whether the number of running copies from the task is one: spark/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala Line 1007 in 4affa58 if (!successful(index) && copiesRunning(index) == 1 && > It could happen that we already have two running copies of the task but both are running on decommissioning executors (i.o on the same host). I am not sure whether this is worth to be considered (I mean here or maybe in a followup PR). What do you think? Yes it is possible that two copies can be running on decommissioning executors and so we will end up not speculating for it because of above checks. I felt it might be a corner case which we can take as separate/follow-up PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28830: [SPARK-31990][SS] Use toSet.toSeq in Dataset.dropDuplicates
SparkQA commented on pull request #28830: URL: https://github.com/apache/spark/pull/28830#issuecomment-644130432 **[Test build #124036 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124036/testReport)** for PR 28830 at commit [`7546ba4`](https://github.com/apache/spark/commit/7546ba4eebeee480d9a2ff8b948e900cd6023dfc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28830: [SPARK-31990][SS] Use toSet.toSeq in Dataset.dropDuplicates
SparkQA removed a comment on pull request #28830: URL: https://github.com/apache/spark/pull/28830#issuecomment-643948852 **[Test build #124036 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124036/testReport)** for PR 28830 at commit [`7546ba4`](https://github.com/apache/spark/commit/7546ba4eebeee480d9a2ff8b948e900cd6023dfc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode
Ngone51 commented on pull request #28746: URL: https://github.com/apache/spark/pull/28746#issuecomment-644130553 What do you guys think about the latest solution? @jiangxb1987 @gerashegalov This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28807: [SPARK-26905][SQL] Follow the SQL:2016 reserved keywords
SparkQA commented on pull request #28807: URL: https://github.com/apache/spark/pull/28807#issuecomment-644139809 **[Test build #124041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124041/testReport)** for PR 28807 at commit [`eeceb30`](https://github.com/apache/spark/commit/eeceb30e050c26acdb93372eef0ce14410bd0159). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28807: [SPARK-26905][SQL] Follow the SQL:2016 reserved keywords
SparkQA removed a comment on pull request #28807: URL: https://github.com/apache/spark/pull/28807#issuecomment-643965298 **[Test build #124041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124041/testReport)** for PR 28807 at commit [`eeceb30`](https://github.com/apache/spark/commit/eeceb30e050c26acdb93372eef0ce14410bd0159). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata
SparkQA removed a comment on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643987758 **[Test build #124044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124044/testReport)** for PR 28826 at commit [`0c46105`](https://github.com/apache/spark/commit/0c46105e1c25ad82dbd3c313e6c2a786c078cabe). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata
SparkQA commented on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-644147029 **[Test build #124044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124044/testReport)** for PR 28826 at commit [`0c46105`](https://github.com/apache/spark/commit/0c46105e1c25ad82dbd3c313e6c2a786c078cabe). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] guykhazma commented on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata
guykhazma commented on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-644146555 @maropu which test do you suggest to add? This is private function which is not tested anywhere also not all file formats are able to save the metadata (for example csv) so seems to me that adding a test to [SchemaPruningSuite](https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruningSuite.scala) is not the right thing. I can add a test similar to the above code snippet, if that seems ok to you where would you suggest to put it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #28786: [SPARK-31925][ML] Summary.totalIterations greater than maxIters
srowen commented on pull request #28786: URL: https://github.com/apache/spark/pull/28786#issuecomment-644146741 Merged to master. It could go in 3.0.1 too; I dont' feel strongly about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen closed pull request #28786: [SPARK-31925][ML] Summary.totalIterations greater than maxIters
srowen closed pull request #28786: URL: https://github.com/apache/spark/pull/28786 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numRowsDroppedByWatermark"
SparkQA commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-644152195 **[Test build #124040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124040/testReport)** for PR 28828 at commit [`75d12d3`](https://github.com/apache/spark/commit/75d12d31814cf774f35368bb487cf0f1ecea904d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
AmplabJenkins removed a comment on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644157284 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124052/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors
SparkQA commented on pull request #28619: URL: https://github.com/apache/spark/pull/28619#issuecomment-644163493 **[Test build #124058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124058/testReport)** for PR 28619 at commit [`d87b311`](https://github.com/apache/spark/commit/d87b311be85819ae884e2a24d94926fdd51165de). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28593: [SPARK-31710][SQL] Fail casting numeric to timestamp by default
SparkQA removed a comment on pull request #28593: URL: https://github.com/apache/spark/pull/28593#issuecomment-643969029 **[Test build #124042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124042/testReport)** for PR 28593 at commit [`8fe1960`](https://github.com/apache/spark/commit/8fe1960ef3a0c598a626b7024820b74cec787642). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28593: [SPARK-31710][SQL] Fail casting numeric to timestamp by default
SparkQA commented on pull request #28593: URL: https://github.com/apache/spark/pull/28593#issuecomment-644173741 **[Test build #124042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124042/testReport)** for PR 28593 at commit [`8fe1960`](https://github.com/apache/spark/commit/8fe1960ef3a0c598a626b7024820b74cec787642). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode
AmplabJenkins commented on pull request #28746: URL: https://github.com/apache/spark/pull/28746#issuecomment-644131514 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28746: [SPARK-31922][CORE] Fix "RpcEnv already stopped" error when exit spark-shell with local-cluster mode
AmplabJenkins removed a comment on pull request #28746: URL: https://github.com/apache/spark/pull/28746#issuecomment-644131514 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata
AmplabJenkins removed a comment on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-644148506 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata
AmplabJenkins commented on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-644148506 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numRowsDroppedByWatermark"
SparkQA removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643965274 **[Test build #124040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124040/testReport)** for PR 28828 at commit [`75d12d3`](https://github.com/apache/spark/commit/75d12d31814cf774f35368bb487cf0f1ecea904d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numRowsDroppedByWatermark"
AmplabJenkins commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-644153619 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28784: [SPARK-31957][SQL][test-maven] Cleanup hive scratch dir for the developer api startWithContext
SparkQA removed a comment on pull request #28784: URL: https://github.com/apache/spark/pull/28784#issuecomment-643948887 **[Test build #124037 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124037/testReport)** for PR 28784 at commit [`d055d60`](https://github.com/apache/spark/commit/d055d60aab356401f0f7b00b83b76dc76df0c30c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28784: [SPARK-31957][SQL][test-maven] Cleanup hive scratch dir for the developer api startWithContext
SparkQA commented on pull request #28784: URL: https://github.com/apache/spark/pull/28784#issuecomment-644153442 **[Test build #124037 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124037/testReport)** for PR 28784 at commit [`d055d60`](https://github.com/apache/spark/commit/d055d60aab356401f0f7b00b83b76dc76df0c30c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numRowsDroppedByWatermark"
AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-644153619 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28784: [SPARK-31957][SQL][test-maven] Cleanup hive scratch dir for the developer api startWithContext
AmplabJenkins commented on pull request #28784: URL: https://github.com/apache/spark/pull/28784#issuecomment-644154215 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
AmplabJenkins commented on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644157275 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28832: [SPARK-31959][SQL][TESTS][FOLLOWUP] Adopt the test "SPARK-31959: JST -> HKT at Asia/Hong_Kong in 1945" to outdated tzdb
SparkQA commented on pull request #28832: URL: https://github.com/apache/spark/pull/28832#issuecomment-644156686 **[Test build #124052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124052/testReport)** for PR 28832 at commit [`55e18b6`](https://github.com/apache/spark/commit/55e18b602a532adecc7f39323f6c8e5990bb528f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
