[GitHub] [spark] HyukjinKwon edited a comment on pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-16 Thread GitBox
HyukjinKwon edited a comment on pull request #29114: URL: https://github.com/apache/spark/pull/29114#issuecomment-659803975 Merged to master. Thank you so much guys <3! This is an automated message from the Apache Git

[GitHub] [spark] HyukjinKwon closed pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-16 Thread GitBox
HyukjinKwon closed pull request #29114: URL: https://github.com/apache/spark/pull/29114 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] HyukjinKwon commented on pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-16 Thread GitBox
HyukjinKwon commented on pull request #29114: URL: https://github.com/apache/spark/pull/29114#issuecomment-659803975 Merged to master. Thank you so much guys! This is an automated message from the Apache Git Service. To

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-16 Thread GitBox
HyukjinKwon commented on a change in pull request #29114: URL: https://github.com/apache/spark/pull/29114#discussion_r456188486 ## File path: python/pyspark/cloudpickle/cloudpickle.py ## @@ -0,0 +1,830 @@ +""" +This class is defined to override standard pickle functionality +

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-16 Thread GitBox
HyukjinKwon commented on a change in pull request #29114: URL: https://github.com/apache/spark/pull/29114#discussion_r456188555 ## File path: LICENSE ## @@ -229,7 +229,7 @@ BSD 3-Clause python/lib/py4j-*-src.zip -python/pyspark/cloudpickle.py

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28676: [SPARK-31869][SQL] BroadcastHashJoinExec can utilize the build side for its output partitioning

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #28676: URL: https://github.com/apache/spark/pull/28676#issuecomment-659803229 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29140: [SPARK-32145][SQL][FOLLOWUP] Fix type in the error log of SparkOperation

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29140: URL: https://github.com/apache/spark/pull/29140#issuecomment-659813256 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-659813548 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-659813548 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] GuoPhilipse commented on a change in pull request #29056: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs

2020-07-16 Thread GitBox
GuoPhilipse commented on a change in pull request #29056: URL: https://github.com/apache/spark/pull/29056#discussion_r456194748 ## File path: docs/sql-ref-syntax-ddl-create-table-hiveformat.md ## @@ -36,6 +36,14 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier

[GitHub] [spark] AmplabJenkins commented on pull request #29140: [SPARK-32145][SQL][FOLLOWUP] Fix type in the error log of SparkOperation

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29140: URL: https://github.com/apache/spark/pull/29140#issuecomment-659813256 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29140: [SPARK-32145][SQL][FOLLOWUP] Fix type in the error log of SparkOperation

2020-07-16 Thread GitBox
SparkQA removed a comment on pull request #29140: URL: https://github.com/apache/spark/pull/29140#issuecomment-659795826 **[Test build #126015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126015/testReport)** for PR 29140 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29020: [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29020: URL: https://github.com/apache/spark/pull/29020#issuecomment-659827876 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29020: [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29020: URL: https://github.com/apache/spark/pull/29020#issuecomment-659827876 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
SparkQA commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-65984 **[Test build #126025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126025/testReport)** for PR 29117 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659844490 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #29015: [SPARK-32215] Expose a (protected) /workers/kill endpoint on the MasterWebUI

2020-07-16 Thread GitBox
SparkQA commented on pull request #29015: URL: https://github.com/apache/spark/pull/29015#issuecomment-659844348 **[Test build #126012 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126012/testReport)** for PR 29015 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
SparkQA removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659834294 **[Test build #126025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126025/testReport)** for PR 29117 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659844490 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] venkata91 commented on pull request #28994: [SPARK-32170][CORE] Improve the speculation for the inefficient tasks by the task metrics.

2020-07-16 Thread GitBox
venkata91 commented on pull request #28994: URL: https://github.com/apache/spark/pull/28994#issuecomment-659869847 This is an interesting idea and a good start. Just considering the runTime of a task alone might not be useful in many cases.

[GitHub] [spark] AmplabJenkins commented on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659870258 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
SparkQA commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659870049 **[Test build #126028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126028/testReport)** for PR 29117 at commit

[GitHub] [spark] venkata91 edited a comment on pull request #28994: [SPARK-32170][CORE] Improve the speculation for the inefficient tasks by the task metrics.

2020-07-16 Thread GitBox
venkata91 edited a comment on pull request #28994: URL: https://github.com/apache/spark/pull/28994#issuecomment-659869847 This is an interesting idea and a good start. Just considering the runTime of a task alone might not be useful in many cases. Thanks!

[GitHub] [spark] SparkQA removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
SparkQA removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659848866 **[Test build #126028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126028/testReport)** for PR 29117 at commit

[GitHub] [spark] viirya commented on a change in pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-16 Thread GitBox
viirya commented on a change in pull request #29114: URL: https://github.com/apache/spark/pull/29114#discussion_r456204492 ## File path: python/pyspark/cloudpickle/cloudpickle.py ## @@ -0,0 +1,830 @@ +""" +This class is defined to override standard pickle functionality + +The

[GitHub] [spark] viirya commented on a change in pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-16 Thread GitBox
viirya commented on a change in pull request #29114: URL: https://github.com/apache/spark/pull/29114#discussion_r456204369 ## File path: LICENSE ## @@ -229,7 +229,7 @@ BSD 3-Clause python/lib/py4j-*-src.zip -python/pyspark/cloudpickle.py

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29135: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29135: URL: https://github.com/apache/spark/pull/29135#issuecomment-659833089 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-16 Thread GitBox
SparkQA removed a comment on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-659773860 **[Test build #126011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126011/testReport)** for PR 29032 at commit

[GitHub] [spark] SparkQA commented on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-16 Thread GitBox
SparkQA commented on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-659833600 **[Test build #126011 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126011/testReport)** for PR 29032 at commit

[GitHub] [spark] viirya commented on a change in pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-16 Thread GitBox
viirya commented on a change in pull request #29114: URL: https://github.com/apache/spark/pull/29114#discussion_r456207305 ## File path: LICENSE ## @@ -229,7 +229,7 @@ BSD 3-Clause python/lib/py4j-*-src.zip -python/pyspark/cloudpickle.py

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29114: [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0

2020-07-16 Thread GitBox
HyukjinKwon commented on a change in pull request #29114: URL: https://github.com/apache/spark/pull/29114#discussion_r456207338 ## File path: LICENSE ## @@ -229,7 +229,7 @@ BSD 3-Clause python/lib/py4j-*-src.zip -python/pyspark/cloudpickle.py

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28840: [SPARK-31999][SQL] Add REFRESH FUNCTION command

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-659833113 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] sap1ens commented on a change in pull request #28852: [SPARK-30616][SQL] Introduce TTL config option for SQL Metadata Cache

2020-07-16 Thread GitBox
sap1ens commented on a change in pull request #28852: URL: https://github.com/apache/spark/pull/28852#discussion_r456209080 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala ## @@ -488,6 +489,30 @@ class FileIndexSuite

[GitHub] [spark] AmplabJenkins commented on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659849344 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
SparkQA commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659848866 **[Test build #126028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126028/testReport)** for PR 29117 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-659850454 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-659850454 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29120: [SPARK-32291][SQL] COALESCE should not reduce the child parallelism if it contains a Join

2020-07-16 Thread GitBox
SparkQA commented on pull request #29120: URL: https://github.com/apache/spark/pull/29120#issuecomment-659862399 **[Test build #126021 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126021/testReport)** for PR 29120 at commit

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456191060 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala ## @@ -56,10 +65,45 @@ trait

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456191194 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkScriptTransformationExec.scala ## @@ -0,0 +1,158 @@ +/* + * Licensed to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456190975 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationExec.scala ## @@ -275,7 +252,7 @@ object

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29128: [SPARK-32329][TESTS] Rename HADOOP2_MODULE_PROFILES to HADOOP_MODULE_PROFILES

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29128: URL: https://github.com/apache/spark/pull/29128#issuecomment-659807627 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29128: [SPARK-32329][TESTS] Rename HADOOP2_MODULE_PROFILES to HADOOP_MODULE_PROFILES

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29128: URL: https://github.com/apache/spark/pull/29128#issuecomment-659807627 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] gengliangwang commented on pull request #29101: [SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-16 Thread GitBox
gengliangwang commented on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-659814952 retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-659825158 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-659824734 **[Test build #126023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126023/testReport)** for PR 29085 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-659825158 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659830392 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659830392 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29015: [SPARK-32215] Expose a (protected) /workers/kill endpoint on the MasterWebUI

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29015: URL: https://github.com/apache/spark/pull/29015#issuecomment-659845800 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29015: [SPARK-32215] Expose a (protected) /workers/kill endpoint on the MasterWebUI

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29015: URL: https://github.com/apache/spark/pull/29015#issuecomment-659845800 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun commented on pull request #29141: [SPARK-32018][SQL][2.4] UnsafeRow.setDecimal should set null with overflowed value

2020-07-16 Thread GitBox
dongjoon-hyun commented on pull request #29141: URL: https://github.com/apache/spark/pull/29141#issuecomment-659859923 Thank you, @cloud-fan ! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] gengliangwang edited a comment on pull request #29133: [SPARK-32253][INFRA] Show errors only for the sbt tests of github actions

2020-07-16 Thread GitBox
gengliangwang edited a comment on pull request #29133: URL: https://github.com/apache/spark/pull/29133#issuecomment-659821378 @HyukjinKwon I have updated the PR description. Meanwhile, I created a PR on my repo to see what the test failure log will look like:

[GitHub] [spark] c21 commented on a change in pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

2020-07-16 Thread GitBox
c21 commented on a change in pull request #29079: URL: https://github.com/apache/spark/pull/29079#discussion_r456232773 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/CoalesceBucketsInJoin.scala ## @@ -0,0 +1,175 @@ +/* + * Licensed to the

[GitHub] [spark] cloud-fan commented on a change in pull request #28852: [SPARK-30616][SQL] Introduce TTL config option for SQL Metadata Cache

2020-07-16 Thread GitBox
cloud-fan commented on a change in pull request #28852: URL: https://github.com/apache/spark/pull/28852#discussion_r456233068 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ## @@ -135,7 +136,16 @@ class SessionCatalog(

[GitHub] [spark] AmplabJenkins commented on pull request #29128: [SPARK-32329][TESTS] Rename HADOOP2_MODULE_PROFILES to HADOOP_MODULE_PROFILES

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29128: URL: https://github.com/apache/spark/pull/29128#issuecomment-659874570 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

2020-07-16 Thread GitBox
SparkQA commented on pull request #29079: URL: https://github.com/apache/spark/pull/29079#issuecomment-659874701 **[Test build #126032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126032/testReport)** for PR 29079 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29128: [SPARK-32329][TESTS] Rename HADOOP2_MODULE_PROFILES to HADOOP_MODULE_PROFILES

2020-07-16 Thread GitBox
SparkQA removed a comment on pull request #29128: URL: https://github.com/apache/spark/pull/29128#issuecomment-659810256 **[Test build #126017 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126017/testReport)** for PR 29128 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #27690: URL: https://github.com/apache/spark/pull/27690#issuecomment-659874439 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] c21 commented on a change in pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

2020-07-16 Thread GitBox
c21 commented on a change in pull request #29079: URL: https://github.com/apache/spark/pull/29079#discussion_r456233046 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/bucketing/CoalesceBucketsInJoinSuite.scala ## @@ -103,46 +119,69 @@ class

[GitHub] [spark] ulysses-you opened a new pull request #29142: [SPARK-29343][SQL][FOLLOW-UP] Add more aggregate function to support eliminate sorts.

2020-07-16 Thread GitBox
ulysses-you opened a new pull request #29142: URL: https://github.com/apache/spark/pull/29142 ### What changes were proposed in this pull request? Add more aggregate function and make these case support eliminate sorts. ### Why are the changes needed? Make

[GitHub] [spark] AmplabJenkins commented on pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29079: URL: https://github.com/apache/spark/pull/29079#issuecomment-659874311 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] c21 commented on a change in pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

2020-07-16 Thread GitBox
c21 commented on a change in pull request #29079: URL: https://github.com/apache/spark/pull/29079#discussion_r456232826 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/bucketing/CoalesceBucketsInJoinSuite.scala ## @@ -178,7 +235,16 @@ class

[GitHub] [spark] SparkQA commented on pull request #29128: [SPARK-32329][TESTS] Rename HADOOP2_MODULE_PROFILES to HADOOP_MODULE_PROFILES

2020-07-16 Thread GitBox
SparkQA commented on pull request #29128: URL: https://github.com/apache/spark/pull/29128#issuecomment-659873878 **[Test build #126017 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126017/testReport)** for PR 29128 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29140: [SPARK-32145][SQL][FOLLOWUP] Fix type in the error log of SparkOperation

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29140: URL: https://github.com/apache/spark/pull/29140#issuecomment-659796249 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] rednaxelafx commented on pull request #29124: [WIP][SPARK-31168][BUILD] Upgrade Scala to 2.12.12

2020-07-16 Thread GitBox
rednaxelafx commented on pull request #29124: URL: https://github.com/apache/spark/pull/29124#issuecomment-659796533 Just a note: both `-Yrepl-class-based` and `-Yrepl-use-magic-imports` are very nice improvements to the Scala REPL, and I'd love to see them in Spark. But they can also

[GitHub] [spark] AmplabJenkins commented on pull request #29140: [SPARK-32145][SQL][FOLLOWUP] Fix type in the error log of SparkOperation

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29140: URL: https://github.com/apache/spark/pull/29140#issuecomment-659796249 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] GuoPhilipse commented on a change in pull request #29056: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs

2020-07-16 Thread GitBox
GuoPhilipse commented on a change in pull request #29056: URL: https://github.com/apache/spark/pull/29056#discussion_r456190158 ## File path: docs/sql-ref-syntax-qry-select-case.md ## @@ -0,0 +1,123 @@ +--- +layout: global +title: CASE Clause +displayTitle: CASE Clause

[GitHub] [spark] HyukjinKwon commented on pull request #29128: [SPARK-32329][TESTS] Rename HADOOP2_MODULE_PROFILES to HADOOP_MODULE_PROFILES

2020-07-16 Thread GitBox
HyukjinKwon commented on pull request #29128: URL: https://github.com/apache/spark/pull/29128#issuecomment-659807307 Jenkins test this please This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #29101: [SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions

2020-07-16 Thread GitBox
SparkQA commented on pull request #29101: URL: https://github.com/apache/spark/pull/29101#issuecomment-659816179 **[Test build #126020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126020/testReport)** for PR 29101 at commit

[GitHub] [spark] LantaoJin commented on pull request #29120: [SPARK-32291][SQL] COALESCE should not reduce the child parallelism if it contains a Join

2020-07-16 Thread GitBox
LantaoJin commented on pull request #29120: URL: https://github.com/apache/spark/pull/29120#issuecomment-659816264 @manuzhang This is for performance improvement. COALESCE with join will reduce the SMJ operator parallelism. The user case is our internal function: download. Download the

[GitHub] [spark] maryannxue commented on a change in pull request #29134: [SPARK-32332][SQL] Support columnar exchanges when AQE is enabled

2020-07-16 Thread GitBox
maryannxue commented on a change in pull request #29134: URL: https://github.com/apache/spark/pull/29134#discussion_r456198713 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala ## @@ -37,16 +37,30 @@ import

[GitHub] [spark] gengliangwang commented on pull request #29133: [SPARK-32253][INFRA] Show errors only for the sbt tests of github actions

2020-07-16 Thread GitBox
gengliangwang commented on pull request #29133: URL: https://github.com/apache/spark/pull/29133#issuecomment-659821378 @HyukjinKwon I have updated the PR description. Meanwhile, I created a PR on my repo to see what the test failure log will look like:

[GitHub] [spark] maryannxue commented on a change in pull request #29134: [SPARK-32332][SQL] Support columnar exchanges when AQE is enabled

2020-07-16 Thread GitBox
maryannxue commented on a change in pull request #29134: URL: https://github.com/apache/spark/pull/29134#discussion_r456199456 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala ## @@ -374,12 +374,14 @@ case class

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456203832 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkScriptTransformationExec.scala ## @@ -0,0 +1,187 @@ +/* + * Licensed to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-659834162 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-659834162 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
SparkQA commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659834294 **[Test build #126025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126025/testReport)** for PR 29117 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29032: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29032: URL: https://github.com/apache/spark/pull/29032#issuecomment-659834167 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #29014: [SPARK-32199][SPARK-32198] Reduce job failures during decommissioning

2020-07-16 Thread GitBox
SparkQA removed a comment on pull request #29014: URL: https://github.com/apache/spark/pull/29014#issuecomment-659773941 **[Test build #126013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126013/testReport)** for PR 29014 at commit

[GitHub] [spark] SparkQA commented on pull request #29014: [SPARK-32199][SPARK-32198] Reduce job failures during decommissioning

2020-07-16 Thread GitBox
SparkQA commented on pull request #29014: URL: https://github.com/apache/spark/pull/29014#issuecomment-659839803 **[Test build #126013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126013/testReport)** for PR 29014 at commit

[GitHub] [spark] viirya commented on a change in pull request #28852: [SPARK-30616][SQL] Introduce TTL config option for SQL Metadata Cache

2020-07-16 Thread GitBox
viirya commented on a change in pull request #28852: URL: https://github.com/apache/spark/pull/28852#discussion_r456219067 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ## @@ -135,7 +136,16 @@ class SessionCatalog(

[GitHub] [spark] AmplabJenkins commented on pull request #29141: [SPARK-32018][SQL][2.4] UnsafeRow.setDecimal should set null with overflowed value

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29141: URL: https://github.com/apache/spark/pull/29141#issuecomment-659852368 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on pull request #29141: [SPARK-32018][SQL][2.4] UnsafeRow.setDecimal should set null with overflowed value

2020-07-16 Thread GitBox
cloud-fan commented on pull request #29141: URL: https://github.com/apache/spark/pull/29141#issuecomment-659852057 cc @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] cloud-fan opened a new pull request #29141: [SPARK-32018][SQL][2.4] UnsafeRow.setDecimal should set null with overflowed value

2020-07-16 Thread GitBox
cloud-fan opened a new pull request #29141: URL: https://github.com/apache/spark/pull/29141 backport https://github.com/apache/spark/pull/29125 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659870258 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29117: [WIP] Debug flaky pip installation test failure

2020-07-16 Thread GitBox
SparkQA commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-659870428 **[Test build #126031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126031/testReport)** for PR 29117 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-659810827 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29135: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29135: URL: https://github.com/apache/spark/pull/29135#issuecomment-659810754 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456193004 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ## @@ -712,14 +713,10 @@ class SparkSqlAstBuilder(conf:

[GitHub] [spark] nvander1 commented on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

2020-07-16 Thread GitBox
nvander1 commented on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-659811425 @ueshin I've made the modification to only take Columns and updated the note to be 3.1.0 for this overload

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-659810827 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29135: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29135: URL: https://github.com/apache/spark/pull/29135#issuecomment-659810754 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-16 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456193132 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala ## @@ -243,7 +243,8 @@ private[hive] trait HiveStrategies {

[GitHub] [spark] beliefer commented on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

2020-07-16 Thread GitBox
beliefer commented on pull request #28917: URL: https://github.com/apache/spark/pull/28917#issuecomment-659819196 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] AmplabJenkins commented on pull request #29120: [SPARK-32291][SQL] COALESCE should not reduce the child parallelism if it contains a Join

2020-07-16 Thread GitBox
AmplabJenkins commented on pull request #29120: URL: https://github.com/apache/spark/pull/29120#issuecomment-659819448 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29120: [SPARK-32291][SQL] COALESCE should not reduce the child parallelism if it contains a Join

2020-07-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29120: URL: https://github.com/apache/spark/pull/29120#issuecomment-659819448 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29120: [SPARK-32291][SQL] COALESCE should not reduce the child parallelism if it contains a Join

2020-07-16 Thread GitBox
SparkQA commented on pull request #29120: URL: https://github.com/apache/spark/pull/29120#issuecomment-659819077 **[Test build #126021 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126021/testReport)** for PR 29120 at commit

[GitHub] [spark] imback82 commented on a change in pull request #29020: [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API

2020-07-16 Thread GitBox
imback82 commented on a change in pull request #29020: URL: https://github.com/apache/spark/pull/29020#discussion_r456197496 ## File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ## @@ -868,13 +868,17 @@ private[spark] class AppStatusListener(

[GitHub] [spark] cloud-fan closed pull request #29140: [SPARK-32145][SQL][FOLLOWUP] Fix type in the error log of SparkOperation

2020-07-16 Thread GitBox
cloud-fan closed pull request #29140: URL: https://github.com/apache/spark/pull/29140 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

  1   2   3   4   5   6   7   8   9   10   >