[GitHub] spark pull request #21745: [SPARK-24781][SQL] Using a reference from Dataset...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21745#discussion_r202236189 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2387,4 +2387,25 @@ class DataFrameSuite extends QueryTest with

[GitHub] spark issue #20795: [SPARK-23486]cache the function name from the external c...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20795 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20795: [SPARK-23486]cache the function name from the ext...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20795#discussion_r202231590 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1204,16 +1207,46 @@ class Analyzer( * only

[GitHub] spark pull request #21753: [SPARK-24790][SQL] Allow complex aggregate expres...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21753#discussion_r202211733 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -586,12 +581,17 @@ class Analyzer

[GitHub] spark issue #21753: [SPARK-24790][SQL] Allow complex aggregate expressions i...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21753 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21751: [SPARK-24208][SQL][FOLLOWUP] Move test cases to proper l...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21751 Please follow the style suggestions in the future. LGTM Thanks! Merged to master --- - To unsubscribe, e

[GitHub] spark pull request #21751: [SPARK-24208][SQL][FOLLOWUP] Move test cases to p...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21751#discussion_r202191341 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala --- @@ -557,4 +558,21 @@ class AnalysisSuite extends

[GitHub] spark pull request #21751: [SPARK-24208][SQL][FOLLOWUP] Move test cases to p...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21751#discussion_r202191261 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala --- @@ -557,4 +558,21 @@ class AnalysisSuite extends

[GitHub] spark issue #20208: [SPARK-23007][SQL][TEST] Add read schema suite for file-...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20208 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20208: [SPARK-23007][SQL][TEST] Add read schema suite for file-...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20208 @dongjoon-hyun This PR is to improve the test coverage. LGTM. When the schema do not match with the schemas of underlying data source, the current error messages might be weird. This is

[GitHub] spark issue #21742: [SPARK-24768][SQL] Have a built-in AVRO data source impl...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21742 LGTM Thanks! Merged to master. @gengliangwang Please submit the follow-up PRs to resolve the sub tasks and the good comment in the PR review

[GitHub] spark pull request #20795: [SPARK-23486]cache the function name from the ext...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20795#discussion_r202175343 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/LookupFunctionsSuite.scala --- @@ -0,0 +1,107 @@ +/* + * Licensed

[GitHub] spark pull request #21753: [SPARK-24790][SQL] Allow complex aggregate expres...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21753#discussion_r202107542 --- Diff: sql/core/src/test/resources/sql-tests/results/pivot.sql.out --- @@ -192,3 +192,33 @@ struct<> -- !query 12

[GitHub] spark pull request #21753: [SPARK-24790][SQL] Allow complex aggregate expres...

2018-07-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21753#discussion_r202105958 --- Diff: sql/core/src/test/resources/sql-tests/results/pivot.sql.out --- @@ -192,3 +192,33 @@ struct<> -- !query 12

[GitHub] spark issue #21745: [SPARK-24781][SQL] Using a reference from Dataset in Fil...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21745 We might need to get rid of AnalysisBarrier in the next release. This already caused at least three regressions in 2.3

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21589 cc @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #21751: [SPARK-24208][SQL][FOLLOWUP] Move test cases to p...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21751#discussion_r201879749 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala --- @@ -557,4 +558,21 @@ class AnalysisSuite extends

[GitHub] spark issue #21736: [SPARK-24782][SQL] Simplify conf retrieval in SQL expres...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21736 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21753: [SPARK-24790][SQL] Allow complex aggregate expres...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21753#discussion_r201870662 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -586,12 +581,16 @@ class Analyzer

[GitHub] spark pull request #21184: [WIP][SPARK-24051][SQL] Replace Aliases with the ...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21184#discussion_r201769371 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2265,4 +2266,15 @@ class DataFrameSuite extends QueryTest with

[GitHub] spark issue #21737: [SPARK-24208][SQL] Fix attribute deduplication for FlatM...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21737 @mgaido91 Since 2.3.2 release will be out soon, I merge this fix to 2.3 branch. Regarding the comments of the test cases, could you submit a follow-up PR

[GitHub] spark pull request #21737: [SPARK-24208][SQL] Fix attribute deduplication fo...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21737#discussion_r201762849 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/GroupedDatasetSuite.scala --- @@ -93,4 +93,16 @@ class GroupedDatasetSuite extends QueryTest with

[GitHub] spark pull request #21737: [SPARK-24208][SQL] Fix attribute deduplication fo...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21737#discussion_r201761976 --- Diff: python/pyspark/sql/tests.py --- @@ -5925,6 +5925,22 @@ def test_invalid_args(self): 'mixture.*aggregate fun

[GitHub] spark issue #21737: [SPARK-24208][SQL] Fix attribute deduplication for FlatM...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21737 LGTM Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21737: [SPARK-24208][SQL] Fix attribute deduplication fo...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21737#discussion_r201758572 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -738,6 +738,10 @@ class Analyzer

[GitHub] spark pull request #20795: [SPARK-23486]cache the function name from the ext...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20795#discussion_r201757597 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/LookupFunctionsSuite.scala --- @@ -0,0 +1,70 @@ +/* + * Licensed

[GitHub] spark pull request #20795: [SPARK-23486]cache the function name from the ext...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20795#discussion_r201757391 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/LookupFunctionsSuite.scala --- @@ -0,0 +1,70 @@ +/* + * Licensed

[GitHub] spark issue #21745: [SPARK-24781][SQL] Using a reference from Dataset in Fil...

2018-07-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21745 Which PR caused this regression? CC @jerryshao We need to block 2.3.2 release before addressing this issue --- - To

[GitHub] spark pull request #21742: [SPARK-24768][SQL] Have a built-in AVRO data sour...

2018-07-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21742#discussion_r201569743 --- Diff: dev/sparktestsupport/modules.py --- @@ -170,6 +170,16 @@ def __hash__(self): ] ) +avro = Module( --- End diff

[GitHub] spark pull request #20208: [SPARK-23007][SQL][TEST] Add read schema suite fo...

2018-07-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20208#discussion_r201506731 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ReadSchemaTest.scala --- @@ -0,0 +1,493 @@ +/* + * Licensed to the

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-07-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21321 `catalogString ` is preferred in error messages. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21321 @HyukjinKwon The whole PR is doing the wrong things. That is why I reverted it. I do not want the others to follow this PR. For the other PRs whose main objective are not to use `simpleString

[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21722#discussion_r201181931 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala --- @@ -215,6 +215,8 @@ object Metadata { x

[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21722#discussion_r201180956 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala --- @@ -215,6 +215,8 @@ object Metadata { x

[GitHub] spark pull request #21657: [SPARK-24676][SQL] Project required data from CSV...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21657#discussion_r201172532 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala --- @@ -38,24 +38,28 @@ class UnivocityParser

[GitHub] spark pull request #20208: [SPARK-23007][SQL][TEST] Add schema evolution tes...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20208#discussion_r201170482 --- Diff: docs/sql-programming-guide.md --- @@ -815,6 +815,54 @@ should start with, they can set `basePath` in the data source options. For examp

[GitHub] spark issue #21582: [SPARK-24576][BUILD] Upgrade Apache ORC to 1.5.2

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21582 > ORC-344 Support for using Decimal64ColumnVector This can achieve better perf number? Could you share the perf g

[GitHub] spark issue #21156: [SPARK-24087][SQL] Avoid shuffle when join keys are a su...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21156 cc @maryannxue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21737: [SPARK-24208][SQL] Fix attribute deduplication fo...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21737#discussion_r201164293 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -738,6 +738,10 @@ class Analyzer

[GitHub] spark pull request #21737: [SPARK-24208][SQL] Fix attribute deduplication fo...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21737#discussion_r201163844 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -738,6 +738,10 @@ class Analyzer

[GitHub] spark issue #21728: [SPARK-24759] [SQL] No reordering keys for broadcast has...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21728 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #18544: [SPARK-21318][SQL]Improve exception message throw...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18544#discussion_r201157527 --- Diff: sql/hive/src/test/java/org/apache/spark/sql/hive/execution/UDAFEmpty.java --- @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #18544: [SPARK-21318][SQL]Improve exception message thrown by `l...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18544 cc @stanzhai Could you address my above comments? Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #18544: [SPARK-21318][SQL]Improve exception message throw...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18544#discussion_r201157378 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -129,14 +129,14 @@ private[sql] class HiveSessionCatalog

[GitHub] spark issue #18544: [SPARK-21318][SQL]Improve exception message thrown by `l...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18544 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20795: [SPARK-23486]cache the function name from the external c...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20795 ping @kevinyu98 @dilipbiswal --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21321 Let me revert the changes. Please re-submit the fix. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21321 I think the fix is wrong. We should not use simpleString but catalogString, because simpleString will do the truncation

[GitHub] spark issue #21360: [SPARK-24288] Enable preventing predicate pushdown

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21360 sorry, just updated the comment. We can add a JDBC connector option. Users can have two JDBC sources for a single JDBC table. One is with predicate pushdown; another without predicate pushdown

[GitHub] spark issue #21360: [SPARK-24288] Enable preventing predicate pushdown

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21360 @TomaszGaweda @maryannxue Let us reduce the complexity and introduce a new JDBC option for controlling the predicate pushdown. ```Scala val JDBC_FILTER_PUSHDOWN_ENABLED = buildConf

[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21682 ping @wangyum --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21728: [SPARK-24759] [SQL] No reordering keys for broadcast has...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21728 cc @cloud-fan @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21728: [SPARK-24759] [SQL] No reordering keys for broadcast has...

2018-07-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21728 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-08 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21720#discussion_r200830672 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -515,13 +515,33 @@ class Analyzer

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-08 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21720#discussion_r200831720 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -559,8 +574,8 @@ class Analyzer

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-08 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21720#discussion_r200829969 --- Diff: sql/core/src/test/resources/sql-tests/results/pivot.sql.out --- @@ -144,51 +155,162 @@ PIVOT ( sum(earnings * s) FOR course IN

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-08 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21720#discussion_r200831568 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -515,13 +515,33 @@ class Analyzer

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-08 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21720#discussion_r200831755 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -515,13 +515,33 @@ class Analyzer

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-08 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21720#discussion_r200830835 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -515,13 +515,33 @@ class Analyzer

[GitHub] spark pull request #21728: [SPARK-24759] [SQL] No reordering keys for broadc...

2018-07-07 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/21728 [SPARK-24759] [SQL] No reordering keys for broadcast hash join ## What changes were proposed in this pull request? As the implementation of the broadcast hash join is independent of the

[GitHub] spark pull request #21608: [SPARK-24626] [SQL] Improve location size calcula...

2018-07-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21608#discussion_r200704192 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala --- @@ -47,15 +48,26 @@ object CommandUtils extends Logging

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21720#discussion_r200518399 --- Diff: sql/core/src/test/resources/sql-tests/inputs/pivot.sql --- @@ -88,12 +93,12 @@ PIVOT ( ); -- pivot with aliases and projection

[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...

2018-07-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21620#discussion_r200514583 --- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/arrayJoin.sql.out --- @@ -0,0 +1,90 @@ +-- Automatically generated by

[GitHub] spark issue #21693: [SPARK-24673][SQL] scala sql function from_utc_timestamp...

2018-07-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21693 @maropu @HyukjinKwon @zzcclp @tmnd1991 Could any of you submit a follow-up PR for PySpark? --- - To unsubscribe, e-mail

[GitHub] spark pull request #21693: [SPARK-24673][SQL] scala sql function from_utc_ti...

2018-07-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21693#discussion_r200419437 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2934,6 +2934,17 @@ object functions { FromUTCTimestamp(ts.expr

[GitHub] spark issue #21655: [SPARK-24675][SQL]Rename table: validate existence of ne...

2018-07-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21655 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21712: [SPARK-22384][SQL][followup] Refine partition pruning wh...

2018-07-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21712 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21608: [SPARK-24626] [SQL] Improve location size calcula...

2018-07-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21608#discussion_r200207191 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala --- @@ -47,22 +47,34 @@ object CommandUtils extends Logging

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21608 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21716: [SPARK-17213][FOLLOW-UP] Improve the test of SPAR...

2018-07-04 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/21716 [SPARK-17213][FOLLOW-UP] Improve the test of SPARK-17213 ## What changes were proposed in this pull request? This is a minor improvement for the test of SPARK-17213 ## How was this

[GitHub] spark pull request #21655: [SPARK-24675][SQL]Rename table: validate existenc...

2018-07-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21655#discussion_r199963152 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -619,6 +619,7 @@ class SessionCatalog

[GitHub] spark pull request #21705: [SPARK-24727][SQL] Add a static config to control...

2018-07-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21705#discussion_r199935428 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala --- @@ -66,6 +66,12 @@ object StaticSQLConf

[GitHub] spark pull request #21705: [SPARK-24727][SQL] Add a static config to control...

2018-07-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21705#discussion_r199936887 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala --- @@ -66,6 +66,12 @@ object StaticSQLConf

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-07-02 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21596 What is the conclusion of this perf comparison? A regression? If we do not have a strong reason, we should avoid the upgrade. Jackson is pretty important for our JSON sources

[GitHub] spark pull request #21668: [SPARK-24690][SQL] Add a new config to control pl...

2018-07-02 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21668#discussion_r199552405 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -375,16 +375,16 @@ case class CatalogStatistics

[GitHub] spark pull request #21668: [SPARK-24690][SQL] Add a new config to control pl...

2018-07-02 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21668#discussion_r199552346 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -375,16 +375,16 @@ case class CatalogStatistics

[GitHub] spark pull request #19757: [SPARK-22529] [SQL] Relation stats should be cons...

2018-07-02 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19757#discussion_r199550269 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -366,10 +366,16 @@ case class CatalogStatistics

[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...

2018-07-02 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21682 It is a regression that was introduced in Spark 1.2. Almost 4 years ago. https://issues.apache.org/jira/browse/SPARK-4453 Thus, I think no need to document it. The pushdown can be

[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21682 @maropu We accidentally dropped the pushdown of these data types when we refactored the file formats. You can check the change history and find the PR

[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21682 cc @gengliangwang --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21682: [SPARK-24706][SQL] ByteType and ShortType support...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21682#discussion_r199390298 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -42,6 +42,14 @@ private[parquet] class

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r199389588 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -182,18 +182,20 @@ private

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r199368095 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -47,16 +47,25 @@ import

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r199364935 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -71,9 +80,22 @@ private[parquet

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r199389252 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -71,9 +80,22 @@ private[parquet

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r199356283 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -47,16 +47,25 @@ import

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r199354841 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -417,11 +417,12 @@ class

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r199365004 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -47,16 +47,25 @@ import

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-07-01 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21688 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21674: [SPARK-24696][SQL] ColumnPruning rule fails to remove ex...

2018-06-29 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21674 Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #21674: [SPARK-24696][SQL] ColumnPruning rule fails to re...

2018-06-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21674#discussion_r199315202 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala --- @@ -149,6 +149,7 @@ package object dsl

[GitHub] spark pull request #21674: [SPARK-24696][SQL] ColumnPruning rule fails to re...

2018-06-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21674#discussion_r199314983 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -2792,4 +2792,25 @@ class SQLQuerySuite extends QueryTest with

[GitHub] spark issue #21613: [SPARK-24629][SQL]thrift server memory leaks when Beelin...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21613 cc @liufengdb --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21600: [Spark-24553][WEB-UI] http 302 fixes for href redirect

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21600 Thanks! Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21389 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21602: [SPARK-24613][SQL] Cache with UDF could not be matched w...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21602 This is also a regression. Backported to 2.3 branch too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18900 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21389 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21389 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21553 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

<    7   8   9   10   11   12   13   14   15   16   >