[GitHub] [spark] SparkQA commented on pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
SparkQA commented on pull request #29166: URL: https://github.com/apache/spark/pull/29166#issuecomment-662393165 **[Test build #126331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126331/testReport)** for PR 29166 at commit

[GitHub] [spark] SparkQA commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
SparkQA commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662392996 **[Test build #126330 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126330/testReport)** for PR 29188 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662367871 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662393010 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
SparkQA commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662392331 **[Test build #126330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126330/testReport)** for PR 29188 at commit

[GitHub] [spark] HyukjinKwon commented on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
HyukjinKwon commented on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662392321 @dongjoon-hyun, do you mean we should update: https://github.com/apache/spark/blob/026b0b926dfd40038f2cee932f38b917eb25b77e/project/SparkBuild.scala#L445-L461

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662392114 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662392114 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458712933 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -754,7 +755,80 @@ class AstBuilder(conf:

[GitHub] [spark] SparkQA removed a comment on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662286321 **[Test build #126312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126312/testReport)** for PR 29104 at commit

[GitHub] [spark] SparkQA commented on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
SparkQA commented on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662391482 **[Test build #126312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126312/testReport)** for PR 29104 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662387970 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662387962 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662388154 **[Test build #126329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126329/testReport)** for PR 29085 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662387962 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662384929 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662384929 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662383656 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662383645 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
SparkQA commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662383880 **[Test build #126328 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126328/testReport)** for PR 29187 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662383645 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] Fokko commented on pull request #29180: [SPARK-17333][PYSPARK] Enable mypy on the repository

2020-07-22 Thread GitBox
Fokko commented on pull request #29180: URL: https://github.com/apache/spark/pull/29180#issuecomment-662383147 @zero323 The annotations aren't random at all. This PR fixes all the outstanding violations on the current master: ``` fokkodriesprong@Fan python % mypy

[GitHub] [spark] gaborgsomogyi commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
gaborgsomogyi commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662383093 cc @HeartSaVioR @zsxwing @xuanyuanking This is an automated message from the Apache Git Service. To

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458697351 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,343 @@ +/* + * Licensed to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458697351 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,343 @@ +/* + * Licensed to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458696650 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,343 @@ +/* + * Licensed to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29166: URL: https://github.com/apache/spark/pull/29166#issuecomment-662374731 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29166: URL: https://github.com/apache/spark/pull/29166#issuecomment-662374731 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458692902 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -754,7 +755,80 @@ class AstBuilder(conf: SQLConf)

[GitHub] [spark] Ngone51 commented on a change in pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
Ngone51 commented on a change in pull request #29166: URL: https://github.com/apache/spark/pull/29166#discussion_r458691890 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1237,20 +1250,79 @@ class Analyzer( if

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458687735 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,343 @@ +/* + * Licensed to the

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458691259 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite

[GitHub] [spark] HyukjinKwon commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662372392 @BryanCutler, @huaxingao, @ueshin, @viirya, @srowen, @dongjoon-hyun, @WeichenXu123, @zhengruifeng, @holdenk, @zero323, can you guys take a look when you are available?

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458689199 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #29188: URL: https://github.com/apache/spark/pull/29188#discussion_r458688852 ## File path: python/docs/source/conf.py ## @@ -14,12 +14,23 @@ import sys import os +import shutil # If extensions (or modules to document

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458688383 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #29188: URL: https://github.com/apache/spark/pull/29188#discussion_r458688468 ## File path: python/docs/source/_templates/class_with_docs.rst ## @@ -0,0 +1,79 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #29188: URL: https://github.com/apache/spark/pull/29188#discussion_r458688468 ## File path: python/docs/source/_templates/class_with_docs.rst ## @@ -0,0 +1,79 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662368327 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #29188: URL: https://github.com/apache/spark/pull/29188#discussion_r458686040 ## File path: .gitignore ## @@ -64,6 +64,7 @@ python/lib/pyspark.zip python/.eggs/ python/deps python/docs/_site/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662368318 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662368200 **[Test build #126317 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126317/testReport)** for PR 29160 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662316877 **[Test build #126317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126317/testReport)** for PR 29160 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662357223 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662368318 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
SparkQA commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662368464 **[Test build #126327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126327/testReport)** for PR 29187 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662367871 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon opened a new pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon opened a new pull request #29188: URL: https://github.com/apache/spark/pull/29188 ### What changes were proposed in this pull request? This PR proposes to redesign the PySpark documentation. I made a demo site to make it easier to review:

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458681551 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala ## @@ -87,17 +178,69 @@ trait

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458681628 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite +--

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662364371 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662364371 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662364401 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662362036 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662362036 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662362247 **[Test build #126326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126326/testReport)** for PR 29085 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662289378 **[Test build #126314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126314/testReport)** for PR 29185 at commit

[GitHub] [spark] SparkQA commented on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
SparkQA commented on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662361283 **[Test build #126314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126314/testReport)** for PR 29185 at commit

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458676081 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite +--

[GitHub] [spark] dongjoon-hyun closed pull request #29183: [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket

2020-07-22 Thread GitBox
dongjoon-hyun closed pull request #29183: URL: https://github.com/apache/spark/pull/29183 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] dongjoon-hyun commented on pull request #29183: [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket

2020-07-22 Thread GitBox
dongjoon-hyun commented on pull request #29183: URL: https://github.com/apache/spark/pull/29183#issuecomment-662358252 Thank you, @maropu and all. Merged to master for Apache Spark 3.1.0 on December 2020. This is an

[GitHub] [spark] AmplabJenkins commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662357223 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] gaborgsomogyi opened a new pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
gaborgsomogyi opened a new pull request #29187: URL: https://github.com/apache/spark/pull/29187 ### What changes were proposed in this pull request? `UninterruptibleThread` running functionality is baked into `KafkaOffsetReader` which can be extracted into a class. The main intention is

[GitHub] [spark] LantaoJin edited a comment on pull request #29062: [SPARK-32237][SQL] Resolve hint in CTE

2020-07-22 Thread GitBox
LantaoJin edited a comment on pull request #29062: URL: https://github.com/apache/spark/pull/29062#issuecomment-662354861 ``` WITH cte AS (SELECT /*+ REPARTITION(3) */ * FROM t) SELECT * FROM cte ``` throws `java.lang.IllegalStateException: Internal error: logical hint operator

[GitHub] [spark] LantaoJin commented on pull request #29062: [SPARK-32237][SQL] Resolve hint in CTE

2020-07-22 Thread GitBox
LantaoJin commented on pull request #29062: URL: https://github.com/apache/spark/pull/29062#issuecomment-662354861 ``` WITH cte AS (SELECT /*+ REPARTITION(3) */ * FROM t) SELECT * FROM cte ``` throws `java.lang.IllegalStateException: Internal error: logical hint operator should

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662350440 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662350440 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662346816 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662346816 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662346190 **[Test build #126325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126325/testReport)** for PR 29085 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29186: [SPARK-32386][SS][TESTS] Fix temp view leaking in Structured Streaming tests

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29186: URL: https://github.com/apache/spark/pull/29186#issuecomment-662343077 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662343043 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458654492 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662343043 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458654664 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala ## @@ -87,17 +178,69 @@ trait

[GitHub] [spark] AmplabJenkins commented on pull request #29186: [SPARK-32386][SS][TESTS] Fix temp view leaking in Structured Streaming tests

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29186: URL: https://github.com/apache/spark/pull/29186#issuecomment-662343077 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662342533 **[Test build #126324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126324/testReport)** for PR 29160 at commit

[GitHub] [spark] SparkQA commented on pull request #29186: [SPARK-32386][SS][TESTS] Fix temp view leaking in Structured Streaming tests

2020-07-22 Thread GitBox
SparkQA commented on pull request #29186: URL: https://github.com/apache/spark/pull/29186#issuecomment-662342525 **[Test build #126323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126323/testReport)** for PR 29186 at commit

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458652781 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class

[GitHub] [spark] dongjoon-hyun commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662341945 Thank you, @cloud-fan . The PR is updated to use `Seq` instead of `Map`. This is an automated message

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458651760 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -290,7 +290,7 @@ class DataFrameReader

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662339436 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662339436 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458649024 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662338839 **[Test build #126322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126322/testReport)** for PR 29160 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
cloud-fan commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458645317 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -290,7 +290,7 @@ class DataFrameReader

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662335140 **[Test build #126321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126321/testReport)** for PR 29160 at commit

[GitHub] [spark] HyukjinKwon commented on pull request #29180: [SPARK-17333][PYSPARK] Enable mypy on the repository

2020-07-22 Thread GitBox
HyukjinKwon commented on pull request #29180: URL: https://github.com/apache/spark/pull/29180#issuecomment-662334741 I also tend to think it's not a good idea to put the types into the codes partially. It would make things more difficult to manage. Let's avoid to do it for now.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662331930 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662331930 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662331774 I updated the PR according to your comment, @cloud-fan . It looks much better indeed. Thank you so much.

[GitHub] [spark] cloud-fan commented on a change in pull request #29107: [SPARK-32308][SQL] Move by-name resolution logic of unionByName from API code to analysis phase

2020-07-22 Thread GitBox
cloud-fan commented on a change in pull request #29107: URL: https://github.com/apache/spark/pull/29107#discussion_r458638512 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala ## @@ -428,7 +428,7 @@ class DataFrameSetOperationsSuite

[GitHub] [spark] cloud-fan commented on a change in pull request #29107: [SPARK-32308][SQL] Move by-name resolution logic of unionByName from API code to analysis phase

2020-07-22 Thread GitBox
cloud-fan commented on a change in pull request #29107: URL: https://github.com/apache/spark/pull/29107#discussion_r458636985 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ## @@ -683,7 +683,7 @@ abstract class SparkStrategies

[GitHub] [spark] xuanyuanking commented on pull request #28977: [WIP] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
xuanyuanking commented on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662328961 I think the A/B test shows the extra parallel test group for `hive.execution` suites takes effect: Test | Description | Scala test time |

[GitHub] [spark] cloud-fan commented on a change in pull request #29107: [SPARK-32308][SQL] Move by-name resolution logic of unionByName from API code to analysis phase

2020-07-22 Thread GitBox
cloud-fan commented on a change in pull request #29107: URL: https://github.com/apache/spark/pull/29107#discussion_r458635997 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala ## @@ -337,6 +337,10 @@ trait CheckAnalysis

[GitHub] [spark] AmplabJenkins commented on pull request #29186: [SPARK-32386][SS][TESTS] Fix temp view leaking in Structured Streaming tests

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29186: URL: https://github.com/apache/spark/pull/29186#issuecomment-662328299 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29186: [SPARK-32386][SS][TESTS] Fix temp view leaking in Structured Streaming tests

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29186: URL: https://github.com/apache/spark/pull/29186#issuecomment-662328299 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458635678 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] SparkQA commented on pull request #29186: [SPARK-32386][SS][TESTS] Fix temp view leaking in Structured Streaming tests

2020-07-22 Thread GitBox
SparkQA commented on pull request #29186: URL: https://github.com/apache/spark/pull/29186#issuecomment-662327785 **[Test build #126320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126320/testReport)** for PR 29186 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #29107: [SPARK-32308][SQL] Move by-name resolution logic of unionByName from API code to analysis phase

2020-07-22 Thread GitBox
cloud-fan commented on a change in pull request #29107: URL: https://github.com/apache/spark/pull/29107#discussion_r458634082 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala ## @@ -50,7 +50,7 @@ object

<    3   4   5   6   7   8   9   10   11   >