[GitHub] [spark] xuanyuanking opened a new pull request #29186: [SPARK-32386][SS][TESTS] Fix temp view leaking in Structured Streaming tests

2020-07-22 Thread GitBox
xuanyuanking opened a new pull request #29186: URL: https://github.com/apache/spark/pull/29186 ### What changes were proposed in this pull request? Delete the temp view created by memory source stream writer in Structured Streaming tests. ### Why are the changes needed? The

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28977: [WIP] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662324835 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29168: [WIP][SPARK-32375][SQL] Basic functionality of table catalog v2 for JDBC

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29168: URL: https://github.com/apache/spark/pull/29168#issuecomment-662324831 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on a change in pull request #29107: [SPARK-32308][SQL] Move by-name resolution logic of unionByName from API code to analysis phase

2020-07-22 Thread GitBox
cloud-fan commented on a change in pull request #29107: URL: https://github.com/apache/spark/pull/29107#discussion_r458631494 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1387,7 +1388,7 @@ class Analyzer(

[GitHub] [spark] AmplabJenkins commented on pull request #29168: [WIP][SPARK-32375][SQL] Basic functionality of table catalog v2 for JDBC

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29168: URL: https://github.com/apache/spark/pull/29168#issuecomment-662324831 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458631113 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ## @@ -713,13 +714,18 @@ class SparkSqlAstBuilder(conf:

[GitHub] [spark] AmplabJenkins commented on pull request #28977: [WIP] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662324835 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28977: [WIP] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
SparkQA commented on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662324250 **[Test build #126319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126319/testReport)** for PR 28977 at commit

[GitHub] [spark] SparkQA commented on pull request #29168: [WIP][SPARK-32375][SQL] Basic functionality of table catalog v2 for JDBC

2020-07-22 Thread GitBox
SparkQA commented on pull request #29168: URL: https://github.com/apache/spark/pull/29168#issuecomment-662324201 **[Test build #126318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126318/testReport)** for PR 29168 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458627500 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458627500 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] cloud-fan closed pull request #29176: [MINOR][SQL][TESTS] Create tables once in JDBC tests

2020-07-22 Thread GitBox
cloud-fan closed pull request #29176: URL: https://github.com/apache/spark/pull/29176 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] cloud-fan commented on pull request #29176: [MINOR][SQL][TESTS] Create tables once in JDBC tests

2020-07-22 Thread GitBox
cloud-fan commented on pull request #29176: URL: https://github.com/apache/spark/pull/29176#issuecomment-662321150 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458626216 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458626216 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] cloud-fan commented on a change in pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
cloud-fan commented on a change in pull request #29166: URL: https://github.com/apache/spark/pull/29166#discussion_r458625525 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1237,20 +1250,42 @@ class Analyzer( if

[GitHub] [spark] maropu commented on pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
maropu commented on pull request #29184: URL: https://github.com/apache/spark/pull/29184#issuecomment-662320088 late LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
cloud-fan commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458624656 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458624125 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,343 @@ +/* + * Licensed to

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458623671 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458623671 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662317577 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662317577 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458619881 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662316877 **[Test build #126317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126317/testReport)** for PR 29160 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458619881 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458619294 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458620082 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,72 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t1 AS

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458619650 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,72 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t1 AS

[GitHub] [spark] cloud-fan commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
cloud-fan commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458618745 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -114,7 +114,7 @@ class DataFrameReader

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458618057 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458617941 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458617007 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -192,7 +137,25 @@ class

[GitHub] [spark] cloud-fan closed pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
cloud-fan closed pull request #29184: URL: https://github.com/apache/spark/pull/29184 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458616565 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458615634 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -192,7 +137,25 @@ class

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458616070 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite

[GitHub] [spark] cloud-fan commented on pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
cloud-fan commented on pull request #29184: URL: https://github.com/apache/spark/pull/29184#issuecomment-662312854 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662310779 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662309777 **[Test build #126316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126316/testReport)** for PR 29160 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662310771 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662310771 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662310751 **[Test build #126316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126316/testReport)** for PR 29160 at commit

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662309777 **[Test build #126316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126316/testReport)** for PR 29160 at commit

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458612159 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala ## @@ -87,17 +178,69 @@ trait

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458611727 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,72 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t1 AS

[GitHub] [spark] Ngone51 commented on a change in pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
Ngone51 commented on a change in pull request #29166: URL: https://github.com/apache/spark/pull/29166#discussion_r458609757 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1237,20 +1250,43 @@ class Analyzer( if

[GitHub] [spark] dongjoon-hyun removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-660845154 Retest this please. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on a change in pull request #29160: URL: https://github.com/apache/spark/pull/29160#discussion_r458609575 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ## @@ -211,6 +211,7 @@ class DataFrameReader

[GitHub] [spark] dongjoon-hyun commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662307263 Hi, @cloud-fan and @HyukjinKwon . This PR aims to fix this problem completely. Please review the approach.

[GitHub] [spark] constzhou commented on a change in pull request #29094: [SPARK-24983][SQL] limit number of leaf expressions in a single project when collapse project to prevent driver oom

2020-07-22 Thread GitBox
constzhou commented on a change in pull request #29094: URL: https://github.com/apache/spark/pull/29094#discussion_r458607522 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CollapseProjectSuite.scala ## @@ -121,6 +121,17 @@ class

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458608523 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,72 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t1 AS

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662306904 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662306904 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458608012 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala ## @@ -56,10 +69,88 @@ trait

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458608774 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,72 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t1 AS

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458606278 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458596110 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,72 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t1 AS SELECT

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29184: URL: https://github.com/apache/spark/pull/29184#issuecomment-662300728 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29184: URL: https://github.com/apache/spark/pull/29184#issuecomment-662286207 **[Test build #126308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126308/testReport)** for PR 29184 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29184: URL: https://github.com/apache/spark/pull/29184#issuecomment-662300728 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
SparkQA commented on pull request #29184: URL: https://github.com/apache/spark/pull/29184#issuecomment-662300444 **[Test build #126308 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126308/testReport)** for PR 29184 at commit

[GitHub] [spark] Ngone51 commented on pull request #28746: [SPARK-31922][CORE] logDebug "RpcEnv already stopped" error on LocalSparkCluster shutdown

2020-07-22 Thread GitBox
Ngone51 commented on pull request #28746: URL: https://github.com/apache/spark/pull/28746#issuecomment-662297775 Thank you @dongjoon-hyun and all! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] MaxGekk commented on pull request #29176: [MINOR][SQL][TESTS] Create tables once in JDBC tests

2020-07-22 Thread GitBox
MaxGekk commented on pull request #29176: URL: https://github.com/apache/spark/pull/29176#issuecomment-662294250 > I think the difference is small. Sine I am working on JDBC staff now, like #29168, I would prefer to spend less time on waiting for tests.

[GitHub] [spark] MaxGekk commented on pull request #29176: [MINOR][SQL][TESTS] Create tables once in JDBC tests

2020-07-22 Thread GitBox
MaxGekk commented on pull request #29176: URL: https://github.com/apache/spark/pull/29176#issuecomment-662292929 > Creating and dropping table for each test case seems more reasonable ... 1. The test in the suite don't modify the global tables 2. If some of them will do in the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29021: [SPARK-32201][SQL] More general skew join pattern matching

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29021: URL: https://github.com/apache/spark/pull/29021#issuecomment-662289996 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] zero323 commented on pull request #29180: [SPARK-17333][PYSPARK] Enable mypy on the repository

2020-07-22 Thread GitBox
zero323 commented on pull request #29180: URL: https://github.com/apache/spark/pull/29180#issuecomment-662291534 I don't really have standing here, but I still think that putting random annotations here and there are make things exclusively worse. Other than that ‒ I'd strongly

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662290016 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662290016 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] gengliangwang edited a comment on pull request #29176: [MINOR][SQL][TESTS] Create tables once in JDBC tests

2020-07-22 Thread GitBox
gengliangwang edited a comment on pull request #29176: URL: https://github.com/apache/spark/pull/29176#issuecomment-662288227 > on my laptop, before ~20 sec, after ~12 sec for JDBCSuite I think the difference is small. Creating and dropping table for each test case seems more

[GitHub] [spark] zhengruifeng commented on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
zhengruifeng commented on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662289751 test code: ``` import org.apache.spark.HashPartitioner val data = sc.parallelize(Seq((0, 5), (3, 8), (2, 6), (0, 8), (3, 8), (1, 3)), 2) val partitioner =

[GitHub] [spark] AmplabJenkins commented on pull request #29021: [SPARK-32201][SQL] More general skew join pattern matching

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29021: URL: https://github.com/apache/spark/pull/29021#issuecomment-662289996 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29021: [SPARK-32201][SQL] More general skew join pattern matching

2020-07-22 Thread GitBox
SparkQA commented on pull request #29021: URL: https://github.com/apache/spark/pull/29021#issuecomment-662289409 **[Test build #126315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126315/testReport)** for PR 29021 at commit

[GitHub] [spark] SparkQA commented on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
SparkQA commented on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662289378 **[Test build #126314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126314/testReport)** for PR 29185 at commit

[GitHub] [spark] gengliangwang commented on pull request #29176: [MINOR][SQL][TESTS] Create tables once in JDBC tests

2020-07-22 Thread GitBox
gengliangwang commented on pull request #29176: URL: https://github.com/apache/spark/pull/29176#issuecomment-662288227 > on my laptop, before ~20 sec, after ~12 sec for JDBCSuite I think the difference is small. Creating and dropping table for each test case seems more reasonable so

[GitHub] [spark] zhengruifeng opened a new pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
zhengruifeng opened a new pull request #29185: URL: https://github.com/apache/spark/pull/29185 ### What changes were proposed in this pull request? avoid unnecessary shuffle if possible ### Why are the changes needed? In `combineByKeyWithClassTag`, there is a check to avoid

[GitHub] [spark] LantaoJin commented on pull request #29021: [SPARK-32201][SQL] More general skew join pattern matching

2020-07-22 Thread GitBox
LantaoJin commented on pull request #29021: URL: https://github.com/apache/spark/pull/29021#issuecomment-662287457 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29183: [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29183: URL: https://github.com/apache/spark/pull/29183#issuecomment-662286690 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29184: URL: https://github.com/apache/spark/pull/29184#issuecomment-662286763 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662286831 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29107: [SPARK-32308][SQL] Move by-name resolution logic of unionByName from API code to analysis phase

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29107: URL: https://github.com/apache/spark/pull/29107#issuecomment-662286804 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662286815 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29145: [SPARK-32346][SQL] Support filters pushdown in Avro datasource

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29145: URL: https://github.com/apache/spark/pull/29145#issuecomment-662286746 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29107: [SPARK-32308][SQL] Move by-name resolution logic of unionByName from API code to analysis phase

2020-07-22 Thread GitBox
SparkQA commented on pull request #29107: URL: https://github.com/apache/spark/pull/29107#issuecomment-662286329 **[Test build #126311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126311/testReport)** for PR 29107 at commit

[GitHub] [spark] SparkQA commented on pull request #29145: [SPARK-32346][SQL] Support filters pushdown in Avro datasource

2020-07-22 Thread GitBox
SparkQA commented on pull request #29145: URL: https://github.com/apache/spark/pull/29145#issuecomment-662286253 **[Test build #126310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126310/testReport)** for PR 29145 at commit

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662286346 **[Test build #126313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126313/testReport)** for PR 29085 at commit

[GitHub] [spark] leanken commented on a change in pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
leanken commented on a change in pull request #29104: URL: https://github.com/apache/spark/pull/29104#discussion_r458583091 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNullAwareHashJoinExec.scala ## @@ -0,0 +1,205 @@ +/* + * Licensed to

[GitHub] [spark] AmplabJenkins commented on pull request #29183: [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29183: URL: https://github.com/apache/spark/pull/29183#issuecomment-662286690 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
SparkQA commented on pull request #29184: URL: https://github.com/apache/spark/pull/29184#issuecomment-662286207 **[Test build #126308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126308/testReport)** for PR 29184 at commit

[GitHub] [spark] leanken commented on a change in pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
leanken commented on a change in pull request #29104: URL: https://github.com/apache/spark/pull/29104#discussion_r458582831 ## File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java ## @@ -171,6 +171,23 @@ private volatile MapIterator

[GitHub] [spark] SparkQA commented on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
SparkQA commented on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662286321 **[Test build #126312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126312/testReport)** for PR 29104 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662286815 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29184: [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29184: URL: https://github.com/apache/spark/pull/29184#issuecomment-662286763 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662286831 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] leanken commented on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
leanken commented on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662286709 > @leanken ... IMHO, this is getting close and really you have done an amazing job not just trying out all the different ideas and approaches but also paying attention to

[GitHub] [spark] MaxGekk commented on pull request #29176: [MINOR][SQL][TESTS] Create tables once in JDBC tests

2020-07-22 Thread GitBox
MaxGekk commented on pull request #29176: URL: https://github.com/apache/spark/pull/29176#issuecomment-662285931 on my laptop, before ~20 sec, after ~12 sec for `JDBCSuite` This is an automated message from the Apache Git

[GitHub] [spark] AmplabJenkins commented on pull request #29145: [SPARK-32346][SQL] Support filters pushdown in Avro datasource

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29145: URL: https://github.com/apache/spark/pull/29145#issuecomment-662286746 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29107: [SPARK-32308][SQL] Move by-name resolution logic of unionByName from API code to analysis phase

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29107: URL: https://github.com/apache/spark/pull/29107#issuecomment-662286804 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29183: [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket

2020-07-22 Thread GitBox
SparkQA commented on pull request #29183: URL: https://github.com/apache/spark/pull/29183#issuecomment-662286224 **[Test build #126309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126309/testReport)** for PR 29183 at commit

[GitHub] [spark] MaxGekk commented on pull request #29145: [SPARK-32346][SQL] Support filters pushdown in Avro datasource

2020-07-22 Thread GitBox
MaxGekk commented on pull request #29145: URL: https://github.com/apache/spark/pull/29145#issuecomment-662285340 jenkins, retest this, please This is an automated message from the Apache Git Service. To respond to the

<    4   5   6   7   8   9   10   11   >