[GitHub] [spark] cloud-fan commented on a change in pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
cloud-fan commented on a change in pull request #28839: URL: https://github.com/apache/spark/pull/28839#discussion_r441299448 ## File path: core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala ## @@ -275,7 +275,8 @@ class BarrierTaskContextSuite

[GitHub] [spark] AmplabJenkins commented on pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-645178487 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
SparkQA removed a comment on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-645093211 **[Test build #124145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124145/testReport)** for PR 28840 at commit

[GitHub] [spark] SparkQA commented on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
SparkQA commented on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645192952 **[Test build #124155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124155/testReport)** for PR 28842 at commit

[GitHub] [spark] SparkQA commented on pull request #28846: [SPARK-32012][SQL] Incrementally create and materialize query stage to avoid unnecessary local shuffle

2020-06-17 Thread GitBox
SparkQA commented on pull request #28846: URL: https://github.com/apache/spark/pull/28846#issuecomment-645192949 **[Test build #124152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124152/testReport)** for PR 28846 at commit

[GitHub] [spark] SparkQA commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
SparkQA commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645192942 **[Test build #124150 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124150/testReport)** for PR 28835 at commit

[GitHub] [spark] SparkQA commented on pull request #28844: [SPARK-32009][ML][PySpark] Remove deprecated method BisectingKMeansModel.computeCost

2020-06-17 Thread GitBox
SparkQA commented on pull request #28844: URL: https://github.com/apache/spark/pull/28844#issuecomment-645192951 **[Test build #124156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124156/testReport)** for PR 28844 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28846: [SPARK-32012][SQL] Incrementally create and materialize query stage to avoid unnecessary local shuffle

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28846: URL: https://github.com/apache/spark/pull/28846#issuecomment-645195541 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] Ngone51 edited a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
Ngone51 edited a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-645234503 Hi @sarutak, thanks for reporting and the fix. First of all, I think it's very unlikely that we'll reach the locality wait timeout(default 3s) since it is still

[GitHub] [spark] SparkQA commented on pull request #28848: [SPARK-32003][CORE] Unregister outputs for executor on fetch failure …

2020-06-17 Thread GitBox
SparkQA commented on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-645255732 **[Test build #124160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124160/testReport)** for PR 28848 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
cloud-fan commented on a change in pull request #28840: URL: https://github.com/apache/spark/pull/28840#discussion_r441401100 ## File path: docs/sql-ref-syntax-aux-refresh-function.md ## @@ -0,0 +1,59 @@ +--- +layout: global +title: REFRESH FUNCTION +displayTitle: REFRESH

[GitHub] [spark] SparkQA commented on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
SparkQA commented on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645176056 **[Test build #124155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124155/testReport)** for PR 28842 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645176405 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645176405 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645184121 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645184484 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
SparkQA removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645183646 **[Test build #124158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124158/testReport)** for PR 28835 at commit

[GitHub] [spark] SparkQA commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
SparkQA commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645184470 **[Test build #124158 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124158/testReport)** for PR 28835 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
SparkQA removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645106412 **[Test build #124150 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124150/testReport)** for PR 28835 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28844: [SPARK-32009][ML][PySpark] Remove deprecated method BisectingKMeansModel.computeCost

2020-06-17 Thread GitBox
SparkQA removed a comment on pull request #28844: URL: https://github.com/apache/spark/pull/28844#issuecomment-645181029 **[Test build #124156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124156/testReport)** for PR 28844 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645193265 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28844: [SPARK-32009][ML][PySpark] Remove deprecated method BisectingKMeansModel.computeCost

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28844: URL: https://github.com/apache/spark/pull/28844#issuecomment-645193018 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28846: [SPARK-32012][SQL] Incrementally create and materialize query stage to avoid unnecessary local shuffle

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28846: URL: https://github.com/apache/spark/pull/28846#issuecomment-645193001 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645193060 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645193060 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28844: [SPARK-32009][ML][PySpark] Remove deprecated method BisectingKMeansModel.computeCost

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28844: URL: https://github.com/apache/spark/pull/28844#issuecomment-645193018 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28846: [SPARK-32012][SQL] Incrementally create and materialize query stage to avoid unnecessary local shuffle

2020-06-17 Thread GitBox
SparkQA removed a comment on pull request #28846: URL: https://github.com/apache/spark/pull/28846#issuecomment-645149223 **[Test build #124152 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124152/testReport)** for PR 28846 at commit

[GitHub] [spark] xuanyuanking commented on a change in pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming state store format

2020-06-17 Thread GitBox
xuanyuanking commented on a change in pull request #28725: URL: https://github.com/apache/spark/pull/28725#discussion_r441377492 ## File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingStateStoreFormatCompatibilitySuite.scala ## @@ -0,0 +1,248 @@ +/* + *

[GitHub] [spark] Ngone51 edited a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
Ngone51 edited a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-645234503 Hi @sarutak, thanks for reporting and the fix. First of all, I think it's very unlikely that we'll reach the locality wait timeout(default 3s) since it is still

[GitHub] [spark] SparkQA commented on pull request #28846: [SPARK-32012][SQL] Incrementally create and materialize query stage to avoid unnecessary local shuffle

2020-06-17 Thread GitBox
SparkQA commented on pull request #28846: URL: https://github.com/apache/spark/pull/28846#issuecomment-645198156 **[Test build #124159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124159/testReport)** for PR 28846 at commit

[GitHub] [spark] attilapiros commented on a change in pull request #28848: [SPARK-32003][CORE] Unregister outputs for executor on fetch failure …

2020-06-17 Thread GitBox
attilapiros commented on a change in pull request #28848: URL: https://github.com/apache/spark/pull/28848#discussion_r441336489 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1986,6 +1986,9 @@ private[spark] class DAGScheduler(

[GitHub] [spark] Ngone51 commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
Ngone51 commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-645245253 thanks a lot :) @sarutak This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] sarutak closed pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
sarutak closed pull request #28839: URL: https://github.com/apache/spark/pull/28839 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] attilapiros commented on pull request #28848: [SPARK-32003][CORE] Unregister outputs for executor on fetch failure …

2020-06-17 Thread GitBox
attilapiros commented on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-645254072 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] jiangxb1987 commented on a change in pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
jiangxb1987 commented on a change in pull request #28839: URL: https://github.com/apache/spark/pull/28839#discussion_r441298122 ## File path: core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala ## @@ -275,7 +275,8 @@ class BarrierTaskContextSuite

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645176412 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
SparkQA commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-645177751 **[Test build #124145 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124145/testReport)** for PR 28840 at commit

[GitHub] [spark] cloud-fan commented on pull request #27066: [SPARK-31317][SQL] Add withField method to Column

2020-06-17 Thread GitBox
cloud-fan commented on pull request #27066: URL: https://github.com/apache/spark/pull/27066#issuecomment-645209161 thanks @fqaiser94 , I like your new changes! > Internally reuse CreateNamedStruct so that existing optimizations (like predicate pushdown and schema pruning) can

[GitHub] [spark] huaxingao commented on pull request #28844: [SPARK-32009][ML][PySpark] Remove deprecated method BisectingKMeansModel.computeCost

2020-06-17 Thread GitBox
huaxingao commented on pull request #28844: URL: https://github.com/apache/spark/pull/28844#issuecomment-645179494 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] yaooqinn commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
yaooqinn commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645180015 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] viirya commented on pull request #28846: [SPARK-32012][SQL] Incrementally create and materialize query stage to avoid unnecessary local shuffle

2020-06-17 Thread GitBox
viirya commented on pull request #28846: URL: https://github.com/apache/spark/pull/28846#issuecomment-645195049 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] ulysses-you commented on pull request #28778: [SPARK-31949][SQL] Add spark.default.parallelism in SQLConf for isolated across session

2020-06-17 Thread GitBox
ulysses-you commented on pull request #28778: URL: https://github.com/apache/spark/pull/28778#issuecomment-645229971 Actually, AQE `CoalesceShufflePartitions` also use the `defaultParallelism` to detect size. List the key code. ``` val minPartitionNum =

[GitHub] [spark] attilapiros commented on a change in pull request #28848: [SPARK-32003][CORE] Unregister outputs for executor on fetch failure …

2020-06-17 Thread GitBox
attilapiros commented on a change in pull request #28848: URL: https://github.com/apache/spark/pull/28848#discussion_r441393834 ## File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ## @@ -540,6 +540,46 @@ class DAGSchedulerSuite extends

[GitHub] [spark] gatorsmile commented on a change in pull request #26624: [SPARK-8981][CORE][test-hadoop3.2][test-java11] Add MDC support in Executor

2020-06-17 Thread GitBox
gatorsmile commented on a change in pull request #26624: URL: https://github.com/apache/spark/pull/26624#discussion_r441303171 ## File path: docs/configuration.md ## @@ -2955,6 +2955,12 @@ Spark uses [log4j](http://logging.apache.org/log4j/) for logging. You can config

[GitHub] [spark] SparkQA commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
SparkQA commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645181025 **[Test build #124157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124157/testReport)** for PR 28835 at commit

[GitHub] [spark] huaxingao commented on a change in pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
huaxingao commented on a change in pull request #28840: URL: https://github.com/apache/spark/pull/28840#discussion_r441319514 ## File path: docs/sql-ref-syntax-aux-refresh-function.md ## @@ -0,0 +1,59 @@ +--- +layout: global +title: REFRESH FUNCTION +displayTitle: REFRESH

[GitHub] [spark] Ngone51 commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
Ngone51 commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-645234503 Hi @sarutak, thanks for reporting and the fix. First of all, I think it's very unlikely that we'll reach the locality wait timeout(default 3s) since it is still very long

[GitHub] [spark] Ngone51 edited a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
Ngone51 edited a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-645234503 Hi @sarutak, thanks for reporting and the fix. First of all, I think it's very unlikely that we'll reach the locality wait timeout(default 3s) since it is still

[GitHub] [spark] ulysses-you commented on a change in pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
ulysses-you commented on a change in pull request #28840: URL: https://github.com/apache/spark/pull/28840#discussion_r441375748 ## File path: docs/sql-ref-syntax-aux-refresh-function.md ## @@ -0,0 +1,59 @@ +--- +layout: global +title: REFRESH FUNCTION +displayTitle: REFRESH

[GitHub] [spark] Ngone51 edited a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
Ngone51 edited a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-645234503 Hi @sarutak, thanks for reporting and the fix. First of all, I think it's very unlikely that we'll reach the locality wait timeout(default 3s) since it is still

[GitHub] [spark] SparkQA removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
SparkQA removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645181025 **[Test build #124157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124157/testReport)** for PR 28835 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645181821 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645181821 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645181826 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
SparkQA commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645181804 **[Test build #124157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124157/testReport)** for PR 28835 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28844: [SPARK-32009][ML][PySpark] Remove deprecated method BisectingKMeansModel.computeCost

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28844: URL: https://github.com/apache/spark/pull/28844#issuecomment-645181399 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645181473 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645184484 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645184495 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] attilapiros commented on a change in pull request #28848: [SPARK-32003][CORE] Unregister outputs for executor on fetch failure …

2020-06-17 Thread GitBox
attilapiros commented on a change in pull request #28848: URL: https://github.com/apache/spark/pull/28848#discussion_r441329936 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1939,24 +1941,22 @@ private[spark] class DAGScheduler(

[GitHub] [spark] Ngone51 edited a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
Ngone51 edited a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-645234503 Hi @sarutak, thanks for reporting and the fix. First of all, I think it's very unlikely that we'll reach the locality wait timeout(default 3s) since it is still

[GitHub] [spark] cloud-fan commented on a change in pull request #27066: [SPARK-31317][SQL] Add withField method to Column

2020-06-17 Thread GitBox
cloud-fan commented on a change in pull request #27066: URL: https://github.com/apache/spark/pull/27066#discussion_r441300305 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala ## @@ -539,3 +541,82 @@ case class

[GitHub] [spark] AmplabJenkins commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645181473 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] yaooqinn commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
yaooqinn commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645181449 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] AmplabJenkins commented on pull request #28844: [SPARK-32009][ML][PySpark] Remove deprecated method BisectingKMeansModel.computeCost

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28844: URL: https://github.com/apache/spark/pull/28844#issuecomment-645181399 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28844: [SPARK-32009][ML][PySpark] Remove deprecated method BisectingKMeansModel.computeCost

2020-06-17 Thread GitBox
SparkQA commented on pull request #28844: URL: https://github.com/apache/spark/pull/28844#issuecomment-645181029 **[Test build #124156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124156/testReport)** for PR 28844 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28846: [SPARK-32012][SQL] Incrementally create and materialize query stage to avoid unnecessary local shuffle

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28846: URL: https://github.com/apache/spark/pull/28846#issuecomment-645195541 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on pull request #28778: [SPARK-31949][SQL] Add spark.default.parallelism in SQLConf for isolated across session

2020-06-17 Thread GitBox
cloud-fan commented on pull request #28778: URL: https://github.com/apache/spark/pull/28778#issuecomment-645239254 So seems we just need to add a min-partition-num config for file source? This is an automated message from

[GitHub] [spark] ulysses-you commented on pull request #28778: [SPARK-31949][SQL] Add spark.default.parallelism in SQLConf for isolated across session

2020-06-17 Thread GitBox
ulysses-you commented on pull request #28778: URL: https://github.com/apache/spark/pull/28778#issuecomment-645239785 Yas, it's a way. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] jiangxb1987 commented on a change in pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
jiangxb1987 commented on a change in pull request #28839: URL: https://github.com/apache/spark/pull/28839#discussion_r441305058 ## File path: core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala ## @@ -275,7 +275,8 @@ class BarrierTaskContextSuite

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-645178487 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan closed pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
cloud-fan closed pull request #28842: URL: https://github.com/apache/spark/pull/28842 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] cloud-fan commented on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
cloud-fan commented on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645178845 The last commit updates indentation only, I'm merging it to master, thanks! This is an automated message from

[GitHub] [spark] huaxingao commented on a change in pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
huaxingao commented on a change in pull request #28840: URL: https://github.com/apache/spark/pull/28840#discussion_r441309342 ## File path: docs/sql-ref-syntax-aux-refresh-function.md ## @@ -0,0 +1,59 @@ +--- +layout: global +title: REFRESH FUNCTION +displayTitle: REFRESH

[GitHub] [spark] SparkQA commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
SparkQA commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645183646 **[Test build #124158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124158/testReport)** for PR 28835 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645184121 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28846: [SPARK-32012][SQL] Incrementally create and materialize query stage to avoid unnecessary local shuffle

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28846: URL: https://github.com/apache/spark/pull/28846#issuecomment-645193007 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645193273 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28844: [SPARK-32009][ML][PySpark] Remove deprecated method BisectingKMeansModel.computeCost

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28844: URL: https://github.com/apache/spark/pull/28844#issuecomment-645193031 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645193067 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #28842: [SPARK-32006][SQL] Create date/timestamp formatters once before collect in `hiveResultString()`

2020-06-17 Thread GitBox
SparkQA removed a comment on pull request #28842: URL: https://github.com/apache/spark/pull/28842#issuecomment-645176056 **[Test build #124155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124155/testReport)** for PR 28842 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28846: [SPARK-32012][SQL] Incrementally create and materialize query stage to avoid unnecessary local shuffle

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28846: URL: https://github.com/apache/spark/pull/28846#issuecomment-645193001 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28835: [WIP][SPARK-31926][TESTS][FOLLOWUP]Cleanup the thread local variable of hive metastore

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28835: URL: https://github.com/apache/spark/pull/28835#issuecomment-645193265 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] attilapiros commented on pull request #28848: [SPARK-32003][CORE] Unregister outputs for executor on fetch failure …

2020-06-17 Thread GitBox
attilapiros commented on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-645194360 Build failure is unrelated. I have checked and there are some other PRs with the same error: ``` [ERROR] Failed to execute goal

[GitHub] [spark] cloud-fan commented on a change in pull request #28619: [SPARK-21040][CORE] Speculate tasks which are running on decommission executors

2020-06-17 Thread GitBox
cloud-fan commented on a change in pull request #28619: URL: https://github.com/apache/spark/pull/28619#discussion_r441376782 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1842,6 +1842,17 @@ package object config {

[GitHub] [spark] Ngone51 edited a comment on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
Ngone51 edited a comment on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-645234503 Hi @sarutak, thanks for reporting and the fix. First of all, I think it's very unlikely that we'll reach the locality wait timeout(default 3s) since it is still

[GitHub] [spark] sarutak commented on pull request #28839: [SPARK-32000][CORE][TESTS] Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread GitBox
sarutak commented on pull request #28839: URL: https://github.com/apache/spark/pull/28839#issuecomment-645244866 @Ngone51 Ah, I got it. I don't mind you opening another PR. This is an automated message from the Apache Git

[GitHub] [spark] sarutak commented on pull request #28849: [SPARK-32000][CORE][TESTS] Fix the flaky test for partially launched task in barrier-mode

2020-06-17 Thread GitBox
sarutak commented on pull request #28849: URL: https://github.com/apache/spark/pull/28849#issuecomment-645299362 LGTM. Thanks @Ngone51 ! This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-645307056 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-645307056 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] ulysses-you commented on pull request #28778: [SPARK-31949][SQL] Add spark.default.parallelism in SQLConf for isolated across session

2020-06-17 Thread GitBox
ulysses-you commented on pull request #28778: URL: https://github.com/apache/spark/pull/28778#issuecomment-645318367 @cloud-fan I will send an another pr later to do that add a min-partition-num config for file source.

[GitHub] [spark] wankunde opened a new pull request #28850: [SPARK-32015][Core]Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread GitBox
wankunde opened a new pull request #28850: URL: https://github.com/apache/spark/pull/28850 ### What changes were proposed in this pull request? Remove reference from `java.lang.ApplicationShutdownHooks` to `sparkContext` when the spark context is stopped. ### Why are the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
AmplabJenkins removed a comment on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-645297787 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-645297787 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28848: [SPARK-32003][CORE] Unregister outputs for executor on fetch failure …

2020-06-17 Thread GitBox
SparkQA commented on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-645329948 **[Test build #124160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124160/testReport)** for PR 28848 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28848: [SPARK-32003][CORE] Unregister outputs for executor on fetch failure …

2020-06-17 Thread GitBox
SparkQA removed a comment on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-645255732 **[Test build #124160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124160/testReport)** for PR 28848 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28850: [SPARK-32015][Core]Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28850: URL: https://github.com/apache/spark/pull/28850#issuecomment-645333997 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28850: [SPARK-32015][Core]Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28850: URL: https://github.com/apache/spark/pull/28850#issuecomment-645333540 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
SparkQA commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-645355478 **[Test build #124164 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124164/testReport)** for PR 28840 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28840: [SPARK-31999][SQL] Add refresh function command

2020-06-17 Thread GitBox
AmplabJenkins commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-645355732 This is an automated message from the Apache Git Service. To respond to the message, please log on to

  1   2   3   4   5   >