[GitHub] [spark] karuppayya closed pull request #28686: [SPARK-31877][SQL]Avoid stats computation for Hive table

2020-06-04 Thread GitBox
karuppayya closed pull request #28686: URL: https://github.com/apache/spark/pull/28686 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] cloud-fan commented on a change in pull request #28723: [SPARK-28624][SQL][TESTS][3.0] Run date.sql via Thrift Server

2020-06-04 Thread GitBox
cloud-fan commented on a change in pull request #28723: URL: https://github.com/apache/spark/pull/28723#discussion_r435695747 ## File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala ## @@ -68,8 +68,6 @@ class

[GitHub] [spark] karuppayya commented on pull request #28662: [SPARK-31850][SQL]Prevent DetermineTableStats from computing stats multiple times for same table

2020-06-04 Thread GitBox
karuppayya commented on pull request #28662: URL: https://github.com/apache/spark/pull/28662#issuecomment-639264681 The above condition is already present. But we return a **copy** of relation(code:

[GitHub] [spark] HyukjinKwon commented on pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
HyukjinKwon commented on pull request #28732: URL: https://github.com/apache/spark/pull/28732#issuecomment-639252018 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] HyukjinKwon closed pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
HyukjinKwon closed pull request #28732: URL: https://github.com/apache/spark/pull/28732 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28732: URL: https://github.com/apache/spark/pull/28732#issuecomment-639250427 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28732: URL: https://github.com/apache/spark/pull/28732#issuecomment-639250427 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28732: URL: https://github.com/apache/spark/pull/28732#issuecomment-639243418 **[Test build #123549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123549/testReport)** for PR 28732 at commit

[GitHub] [spark] SparkQA commented on pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
SparkQA commented on pull request #28732: URL: https://github.com/apache/spark/pull/28732#issuecomment-639250134 **[Test build #123549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123549/testReport)** for PR 28732 at commit

[GitHub] [spark] SparkQA commented on pull request #28645: [SPARK-31826][SQL] Support composed type of case class for typed Scala UDF

2020-06-04 Thread GitBox
SparkQA commented on pull request #28645: URL: https://github.com/apache/spark/pull/28645#issuecomment-639244877 **[Test build #123550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123550/testReport)** for PR 28645 at commit

[GitHub] [spark] HyukjinKwon edited a comment on pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
HyukjinKwon edited a comment on pull request #28730: URL: https://github.com/apache/spark/pull/28730#issuecomment-639243858 Merged to master and branch-3.0. I don't mind porting it back if anyone needs. I didn't here just because there's a conflict, and it's just a matter of monitoring.

[GitHub] [spark] HyukjinKwon commented on pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
HyukjinKwon commented on pull request #28730: URL: https://github.com/apache/spark/pull/28730#issuecomment-639243858 Merged to master and branch-3.0. I don't mind porting it back if anyone needs. I didn't here just because there's a conflict, and it's just a matter of monitoring.

[GitHub] [spark] AmplabJenkins commented on pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28732: URL: https://github.com/apache/spark/pull/28732#issuecomment-639243664 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28732: URL: https://github.com/apache/spark/pull/28732#issuecomment-639243664 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
SparkQA commented on pull request #28732: URL: https://github.com/apache/spark/pull/28732#issuecomment-639243418 **[Test build #123549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123549/testReport)** for PR 28732 at commit

[GitHub] [spark] HyukjinKwon closed pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
HyukjinKwon closed pull request #28730: URL: https://github.com/apache/spark/pull/28730 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] HyukjinKwon opened a new pull request #28732: [MINOR][PYTHON] Add one more newline between JVM and Python tracebacks

2020-06-04 Thread GitBox
HyukjinKwon opened a new pull request #28732: URL: https://github.com/apache/spark/pull/28732 ### What changes were proposed in this pull request? This PR proposes to add one more newline to clearly separate JVM and Python tracebacks: Before: ``` Traceback (most

[GitHub] [spark] HeartSaVioR edited a comment on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

2020-06-04 Thread GitBox
HeartSaVioR edited a comment on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-639239790 My alternative with wrapping state store is something like below: ``` class RowValidatingStateStore( underlying: StateStore, keyType:

[GitHub] [spark] HeartSaVioR commented on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

2020-06-04 Thread GitBox
HeartSaVioR commented on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-639239790 My alternative with wrapping state store is something like below: ``` class RowValidatingStateStore( underlying: StateStore, keyType:

[GitHub] [spark] HeartSaVioR edited a comment on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

2020-06-04 Thread GitBox
HeartSaVioR edited a comment on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-639200645 > @HeartSaVioR After taking a further look. Instead of dealing with the iterator, how about adding the invalidation for all state store operations in

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28730: URL: https://github.com/apache/spark/pull/28730#issuecomment-639226292 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28730: URL: https://github.com/apache/spark/pull/28730#issuecomment-639226292 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28730: URL: https://github.com/apache/spark/pull/28730#issuecomment-639151923 **[Test build #123547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123547/testReport)** for PR 28730 at commit

[GitHub] [spark] SparkQA commented on pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
SparkQA commented on pull request #28730: URL: https://github.com/apache/spark/pull/28730#issuecomment-639225813 **[Test build #123547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123547/testReport)** for PR 28730 at commit

[GitHub] [spark] holdenk commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
holdenk commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639222935 The K8s test failure appears unrelated (`- Run in client mode. *** FAILED ***`) we don't do anything with the tokens. I'll investigate more tomorrow.

[GitHub] [spark] AmplabJenkins commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639222847 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639222847 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
SparkQA commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-63971 **[Test build #123548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123548/testReport)** for PR 28708 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639183660 **[Test build #123548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123548/testReport)** for PR 28708 at commit

[GitHub] [spark] bmarcott commented on pull request #27096: [SPARK-28148][SQL] Repartition after join is not optimized away

2020-06-04 Thread GitBox
bmarcott commented on pull request #27096: URL: https://github.com/apache/spark/pull/27096#issuecomment-639221995 @cloud-fan @viirya could you help review or add a suggested reviewer here? This is an automated message from

[GitHub] [spark] HyukjinKwon edited a comment on pull request #22480: [SPARK-25473][PYTHON][SS][TEST] ForeachWriter tests failed on Python 3.6 and macOS High Sierra

2020-06-04 Thread GitBox
HyukjinKwon edited a comment on pull request #22480: URL: https://github.com/apache/spark/pull/22480#issuecomment-639212453 @pquentin, yes, it's kind of difficult to avoid in PySpark side for now. The problem isn't solely because we use `fork()` but it binds to other conditions. I didn't

[GitHub] [spark] HyukjinKwon commented on pull request #22480: [SPARK-25473][PYTHON][SS][TEST] ForeachWriter tests failed on Python 3.6 and macOS High Sierra

2020-06-04 Thread GitBox
HyukjinKwon commented on pull request #22480: URL: https://github.com/apache/spark/pull/22480#issuecomment-639212453 @pquentin, yes, it's kind of difficult to avoid in PySpark side for now. The problem isn't solely because we use `fork()` but it binds to other conditions. I didn't take a

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639202750 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639202742 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
SparkQA commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639202723 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28172/

[GitHub] [spark] AmplabJenkins commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639202742 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] holdenk commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
holdenk commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639201319 > > > So @attilapiros looking at the Jenkins console logs we aren't leaking any threads during testing (nor would I expect us to). But I'll add something to more aggressively

[GitHub] [spark] HeartSaVioR commented on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

2020-06-04 Thread GitBox
HeartSaVioR commented on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-639200645 > @HeartSaVioR After taking a further look. Instead of dealing with the iterator, how about adding the invalidation for all state store operations in StateStoreProvider?

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28720: URL: https://github.com/apache/spark/pull/28720#issuecomment-639197112 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28720: URL: https://github.com/apache/spark/pull/28720#issuecomment-639197112 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28720: URL: https://github.com/apache/spark/pull/28720#issuecomment-639149691 **[Test build #123546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123546/testReport)** for PR 28720 at commit

[GitHub] [spark] SparkQA commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx

2020-06-04 Thread GitBox
SparkQA commented on pull request #28720: URL: https://github.com/apache/spark/pull/28720#issuecomment-639196496 **[Test build #123546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123546/testReport)** for PR 28720 at commit

[GitHub] [spark] SparkQA commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
SparkQA commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639195695 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28172/

[GitHub] [spark] AmplabJenkins commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639187847 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639187847 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] skambha commented on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

2020-06-04 Thread GitBox
skambha commented on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-639187545 > @skambha You can check the integrated tests in #28725. If we delete the validation, we'll get a NPE for [this

[GitHub] [spark] github-actions[bot] commented on pull request #27172: [WIP] [SPARK-29644][SQL] Fixed ByteType JDBCUtils to map to TinyInt at write read and ShortType on read

2020-06-04 Thread GitBox
github-actions[bot] commented on pull request #27172: URL: https://github.com/apache/spark/pull/27172#issuecomment-639187219 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] SparkQA removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639134384 **[Test build #123545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123545/testReport)** for PR 28710 at commit

[GitHub] [spark] SparkQA commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
SparkQA commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639187095 **[Test build #123545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123545/testReport)** for PR 28710 at commit

[GitHub] [spark] SparkQA commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
SparkQA commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639183660 **[Test build #123548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123548/testReport)** for PR 28708 at commit

[GitHub] [spark] siknezevic commented on pull request #27246: [SPARK-30536][CORE][SQL] Sort-merge join operator spilling performance improvements

2020-06-04 Thread GitBox
siknezevic commented on pull request #27246: URL: https://github.com/apache/spark/pull/27246#issuecomment-639172725 Thank you for the comments. I will addressed them soon This is an automated message from the Apache Git

[GitHub] [spark] jacobwu123 opened a new pull request #28731: [SPARK-31909][CORE] Add SPARK_SUBMIT_OPTS to Beeline Script

2020-06-04 Thread GitBox
jacobwu123 opened a new pull request #28731: URL: https://github.com/apache/spark/pull/28731 ### What changes were proposed in this pull request? Added the SPARK_SUBMIT_OPTS environment available to beeline. ### Why are the changes needed? The beeline is not

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28731: [SPARK-31909][CORE] Add SPARK_SUBMIT_OPTS to Beeline Script

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28731: URL: https://github.com/apache/spark/pull/28731#issuecomment-639158213 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] AmplabJenkins commented on pull request #28731: [SPARK-31909][CORE] Add SPARK_SUBMIT_OPTS to Beeline Script

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28731: URL: https://github.com/apache/spark/pull/28731#issuecomment-639158594 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28731: [SPARK-31909][CORE] Add SPARK_SUBMIT_OPTS to Beeline Script

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28731: URL: https://github.com/apache/spark/pull/28731#issuecomment-639158213 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28730: URL: https://github.com/apache/spark/pull/28730#issuecomment-639152306 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28730: URL: https://github.com/apache/spark/pull/28730#issuecomment-639152306 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28730: [SPARK-31903][SQL][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
SparkQA commented on pull request #28730: URL: https://github.com/apache/spark/pull/28730#issuecomment-639151923 **[Test build #123547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123547/testReport)** for PR 28730 at commit

[GitHub] [spark] ueshin opened a new pull request #28730: [SPARK-31903][PYSPARK][R] Fix toPandas with Arrow enabled to show metrics in Query UI.

2020-06-04 Thread GitBox
ueshin opened a new pull request #28730: URL: https://github.com/apache/spark/pull/28730 ### What changes were proposed in this pull request? In `Dataset.collectAsArrowToR` and `Dataset.collectAsArrowToPython`, since the code block for `serveToStream` is run in the separate thread,

[GitHub] [spark] SparkQA commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx

2020-06-04 Thread GitBox
SparkQA commented on pull request #28720: URL: https://github.com/apache/spark/pull/28720#issuecomment-639149691 **[Test build #123546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123546/testReport)** for PR 28720 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28720: URL: https://github.com/apache/spark/pull/28720#issuecomment-639147791 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28720: URL: https://github.com/apache/spark/pull/28720#issuecomment-638456255 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] AmplabJenkins commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28720: URL: https://github.com/apache/spark/pull/28720#issuecomment-639147791 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] gatorsmile commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx

2020-06-04 Thread GitBox
gatorsmile commented on pull request #28720: URL: https://github.com/apache/spark/pull/28720#issuecomment-639147436 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435575853 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1822,108 @@ private[spark] class BlockManager( } }

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435575539 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1822,108 @@ private[spark] class BlockManager( } }

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435575094 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1822,108 @@ private[spark] class BlockManager( } }

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28729: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28729: URL: https://github.com/apache/spark/pull/28729#issuecomment-639137871 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28729: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28729: URL: https://github.com/apache/spark/pull/28729#issuecomment-639137856 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #28729: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28729: URL: https://github.com/apache/spark/pull/28729#issuecomment-639122853 **[Test build #123544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123544/testReport)** for PR 28729 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28729: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28729: URL: https://github.com/apache/spark/pull/28729#issuecomment-639137856 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28729: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server

2020-06-04 Thread GitBox
SparkQA commented on pull request #28729: URL: https://github.com/apache/spark/pull/28729#issuecomment-639137804 **[Test build #123544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123544/testReport)** for PR 28729 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28728: [SPARK-31879][SQL][test-java11] Make week-based pattern invalid for formatting too

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28728: URL: https://github.com/apache/spark/pull/28728#issuecomment-639137018 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28728: [SPARK-31879][SQL][test-java11] Make week-based pattern invalid for formatting too

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28728: URL: https://github.com/apache/spark/pull/28728#issuecomment-639137018 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28728: [SPARK-31879][SQL][test-java11] Make week-based pattern invalid for formatting too

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28728: URL: https://github.com/apache/spark/pull/28728#issuecomment-638992217 **[Test build #123538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123538/testReport)** for PR 28728 at commit

[GitHub] [spark] SparkQA commented on pull request #28728: [SPARK-31879][SQL][test-java11] Make week-based pattern invalid for formatting too

2020-06-04 Thread GitBox
SparkQA commented on pull request #28728: URL: https://github.com/apache/spark/pull/28728#issuecomment-639136125 **[Test build #123538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123538/testReport)** for PR 28728 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639134843 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639134843 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
SparkQA commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639134384 **[Test build #123545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123545/testReport)** for PR 28710 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28729: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28729: URL: https://github.com/apache/spark/pull/28729#issuecomment-639123368 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28729: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28729: URL: https://github.com/apache/spark/pull/28729#issuecomment-639123368 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28729: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server

2020-06-04 Thread GitBox
SparkQA commented on pull request #28729: URL: https://github.com/apache/spark/pull/28729#issuecomment-639122853 **[Test build #123544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123544/testReport)** for PR 28729 at commit

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r43809 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1822,108 @@ private[spark] class BlockManager( } }

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639118857 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639118857 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639026461 **[Test build #123542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123542/testReport)** for PR 28708 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639117314 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
SparkQA commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-639117770 **[Test build #123542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123542/testReport)** for PR 28708 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639117310 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639116348 **[Test build #123543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123543/testReport)** for PR 28710 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639116837 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
SparkQA commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639117298 **[Test build #123543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123543/testReport)** for PR 28710 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639117310 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639116837 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28710: [SPARK-31893][ML] Add a generic ClassificationSummary trait

2020-06-04 Thread GitBox
SparkQA commented on pull request #28710: URL: https://github.com/apache/spark/pull/28710#issuecomment-639116348 **[Test build #123543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123543/testReport)** for PR 28710 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28728: [SPARK-31879][SQL][test-java11] Make week-based pattern invalid for formatting too

2020-06-04 Thread GitBox
AmplabJenkins removed a comment on pull request #28728: URL: https://github.com/apache/spark/pull/28728#issuecomment-639112128 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28728: [SPARK-31879][SQL][test-java11] Make week-based pattern invalid for formatting too

2020-06-04 Thread GitBox
AmplabJenkins commented on pull request #28728: URL: https://github.com/apache/spark/pull/28728#issuecomment-639112128 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28728: [SPARK-31879][SQL][test-java11] Make week-based pattern invalid for formatting too

2020-06-04 Thread GitBox
SparkQA removed a comment on pull request #28728: URL: https://github.com/apache/spark/pull/28728#issuecomment-638970542 **[Test build #123537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123537/testReport)** for PR 28728 at commit

[GitHub] [spark] SparkQA commented on pull request #28728: [SPARK-31879][SQL][test-java11] Make week-based pattern invalid for formatting too

2020-06-04 Thread GitBox
SparkQA commented on pull request #28728: URL: https://github.com/apache/spark/pull/28728#issuecomment-639111050 **[Test build #123537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123537/testReport)** for PR 28728 at commit

[GitHub] [spark] viirya commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

2020-06-04 Thread GitBox
viirya commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435522183 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1822,108 @@ private[spark] class BlockManager( } }

  1   2   3   4   >