[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21389 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21389 cc @gengliangwang Please take another look --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21590: [SPARK-24423][SQL] Add a new option for JDBC sources

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21590 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r198314687 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -109,6 +134,20 @@ class JDBCOptions

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r198313914 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala --- @@ -86,7 +86,8 @@ class

[GitHub] spark issue #21641: [SPARK-24658][SQL] Remove workaround for ANTLR bug

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21641 Based on my understanding, the issue should be fixed by https://github.com/antlr/antlr4/pull/873 , which is released in https://github.com/antlr/antlr4/releases/tag/4.5.1

[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21553#discussion_r198301742 --- Diff: python/pyspark/sql/conf.py --- @@ -64,6 +64,96 @@ def _checkType(self, obj, identifier): (identifier, obj

[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21553#discussion_r198300614 --- Diff: docs/configuration.md --- @@ -456,33 +456,6 @@ Apart from these, the following properties are also available, and may be useful from

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21598 LGTM and WFT --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21598 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21570: [SPARK-24564][TEST] Add test suite for RecordBinaryCompa...

2018-06-25 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21570 ping @JoshRosen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-25 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21594 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-25 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21594 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21631: [SPARK-24645][SQL] Skip parsing when csvColumnPruning en...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21631 Both? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21631: [SPARK-24645][SQL] Skip parsing when csvColumnPruning en...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21631 @maropu Could you confirm whether these two bugs are regressions in the master branch? --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21594 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21379 The build passed. The tests have passed in the previous run. The current tests will be killed at the midnight. LGTM Thanks! Merged to master

[GitHub] spark pull request #21603: [SPARK-17091][SQL] Add rule to convert IN predica...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21603#discussion_r197686539 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -378,6 +378,17 @@ object SQLConf { .booleanConf

[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21379 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21603: [SPARK-17091][SQL] Add rule to convert IN predica...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21603#discussion_r197681414 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -378,6 +378,17 @@ object SQLConf { .booleanConf

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21596 Can we target this to Spark 3.0, which should be the next release after Spark 2.4 release? --- - To unsubscribe, e-mail

[GitHub] spark pull request #21618: [SPARK-20408][SQL] Get the glob path in parallel ...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21618#discussion_r197657738 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -724,4 +726,35 @@ object DataSource extends

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21598 All the behavior changes need very careful reviews and discussions. Whenever we decide to make a behavior change, we should document it in the migration guide and provide a conf to revert it

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21594 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r197646687 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/QueryPlanConstraints.scala --- @@ -99,27 +100,28 @@ trait

[GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...

2018-06-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21320#discussion_r197629698 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala --- @@ -301,7 +301,6 @@ case class FileSourceScanExec

[GitHub] spark issue #21288: [SPARK-24206][SQL] Improve FilterPushdownBenchmark bench...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21288 Sure --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20345 cc @maryannxue Please take a look at this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/Pa...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21389#discussion_r197626306 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala --- @@ -0,0 +1,88 @@ +/* + * Licensed to the

[GitHub] spark pull request #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/Pa...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21389#discussion_r197626232 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala --- @@ -202,4 +204,222 @@ class FileBasedDataSourceSuite extends

[GitHub] spark pull request #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/Pa...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21389#discussion_r197626154 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala --- @@ -0,0 +1,88 @@ +/* + * Licensed to the

[GitHub] spark pull request #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/Pa...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21389#discussion_r197626122 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala --- @@ -0,0 +1,88 @@ +/* + * Licensed to the

[GitHub] spark issue #21288: [SPARK-24206][SQL] Improve FilterPushdownBenchmark bench...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21288 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20949: [SPARK-19018][SQL] Add support for custom encoding on cs...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20949 cc @MaxGekk @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21247: [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 ...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21247 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620329 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -65,13 +65,38 @@ class JDBCOptions

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620436 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -65,13 +65,38 @@ class JDBCOptions

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620455 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -109,6 +134,20 @@ class JDBCOptions

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620140 --- Diff: docs/sql-programming-guide.md --- @@ -1302,9 +1302,20 @@ the following case-insensitive options: dbtable

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620099 --- Diff: docs/sql-programming-guide.md --- @@ -1302,9 +1302,20 @@ the following case-insensitive options: dbtable

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620021 --- Diff: docs/sql-programming-guide.md --- @@ -1302,9 +1302,20 @@ the following case-insensitive options: dbtable

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620115 --- Diff: docs/sql-programming-guide.md --- @@ -1302,9 +1302,20 @@ the following case-insensitive options: dbtable

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620496 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala --- @@ -58,6 +58,10 @@ class

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620483 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -109,6 +134,20 @@ class JDBCOptions

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620091 --- Diff: docs/sql-programming-guide.md --- @@ -1302,9 +1302,20 @@ the following case-insensitive options: dbtable

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620340 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -65,13 +65,38 @@ class JDBCOptions

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r197620384 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -65,13 +65,38 @@ class JDBCOptions

[GitHub] spark pull request #21598: [SPARK-24605][SQL] size(null) returns null instea...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21598#discussion_r197618603 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -75,28 +75,47 @@ trait

[GitHub] spark pull request #21598: [SPARK-24605][SQL] size(null) returns null instea...

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21598#discussion_r197618318 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -75,28 +75,47 @@ trait

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21598 I created a JIRA https://issues.apache.org/jira/browse/SPARK-24640. We should change the behavior in 3.0. Before 3.0 release, we introduce a conf and make it configurable. The default is to keep

[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down

2018-06-23 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21623 cc @rdblue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21594 Thanks for fixing the current behavior! This is a very important fix. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21594 document the behavior changes in the `# Migration Guide` of `/docs/sql-programming-guide.md` --- - To unsubscribe, e-mail

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r197591957 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala --- @@ -490,7 +494,7 @@ class CatalogImpl(sparkSession: SparkSession

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r197591943 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala --- @@ -438,7 +440,9 @@ class CatalogImpl(sparkSession: SparkSession

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r197591886 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -204,9 +205,10 @@ case class DropTableCommand

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r197591842 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -189,8 +189,9 @@ case class DropTableCommand

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r197591661 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -2971,7 +2971,7 @@ class Dataset[T] private[sql]( * @since 1.6.0

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r197586914 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -493,7 +493,7 @@ case class TruncateTableCommand

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r197586970 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala --- @@ -105,24 +105,58 @@ class CacheManager extends Logging

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r197586499 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala --- @@ -105,24 +105,58 @@ class CacheManager extends Logging

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r197289750 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala --- @@ -105,24 +105,58 @@ class CacheManager extends Logging

[GitHub] spark pull request #21603: [SPARK-17091][SQL] Add rule to convert IN predica...

2018-06-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21603#discussion_r197523544 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -378,6 +378,17 @@ object SQLConf { .booleanConf

[GitHub] spark pull request #21610: Updates to LICENSE and NOTICE

2018-06-21 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21610#discussion_r197348104 --- Diff: NOTICE --- @@ -1,667 +1,11 @@ Apache Spark -Copyright 2014 and onwards The Apache Software Foundation. +Copyright 2014 - 20018 The

[GitHub] spark pull request #21610: Updates to LICENSE and NOTICE

2018-06-21 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21610#discussion_r197347713 --- Diff: NOTICE --- @@ -1,667 +1,11 @@ Apache Spark -Copyright 2014 and onwards The Apache Software Foundation. +Copyright 2014 - 20018 The

[GitHub] spark issue #21610: Updates to LICENSE and NOTICE

2018-06-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21610 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21587: [SPARK-24588][SS] streaming join should require HashClus...

2018-06-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21587 LGTM Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21602: [SPARK-24613][SQL] Cache with UDF could not be matched w...

2018-06-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21602 Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #21603: [SPARK-17091][SQL] Add rule to convert IN predica...

2018-06-21 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21603#discussion_r197191390 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -270,6 +270,11 @@ private[parquet

[GitHub] spark issue #21602: [SPARK-24613][SQL] Cache with UDF could not be matched w...

2018-06-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21602 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21578: [SPARK-24571][SQL] Support Char literals

2018-06-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21578 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21593: [SPARK-24578][Core] Cap sub-region's size of returned ni...

2018-06-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21593 cc @zsxwing @JoshRosen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21585 Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #21578: [SPARK-24571][SQL] Support Char literals

2018-06-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21578 @MaxGekk Could you resolve the conflict and then I will merge it? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21531: [SPARK-24521][SQL][TEST] Fix ineffective test in CachedT...

2018-06-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21531 LGTM Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21587: [SPARK-24588][SS] streaming join should require H...

2018-06-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21587#discussion_r196301308 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala --- @@ -99,16 +99,19 @@ case class

[GitHub] spark pull request #21550: [SPARK-24543][SQL] Support any type as DDL string...

2018-06-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21550#discussion_r196242923 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala --- @@ -354,8 +354,8 @@ class JsonFunctionsSuite extends QueryTest with

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21585 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21546 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21529 Adding new queries to `SQLQueryTestSuite` is the best way to do it in the current infrastructure. Do your best to cover all the join algorithms for different input data and join types

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21529 Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #21288: [SPARK-24206][SQL] Improve FilterPushdownBenchmark bench...

2018-06-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21288 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21379: [SPARK-24327][SQL] Verify and normalize a partition colu...

2018-06-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21379 LGTM except one minor comment. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #21379: [SPARK-24327][SQL] Verify and normalize a partiti...

2018-06-14 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21379#discussion_r195479735 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -309,7 +309,8 @@ class DataFrameReader private[sql](sparkSession

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21389 @maropu Just want to double check whether all the data types are not supported before this PR? Have you ran these test cases without the code changes? After this PR, the error messages are more

[GitHub] spark pull request #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/Pa...

2018-06-14 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21389#discussion_r195472992 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala --- @@ -0,0 +1,61 @@ +/* + * Licensed to the

[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21553 Could you address the comments in the original PR? - https://github.com/apache/spark/pull/21370#discussion_r194276735

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement eager evaluation...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r195316251 --- Diff: docs/configuration.md --- @@ -456,6 +456,33 @@ Apart from these, the following properties are also available, and may be useful from

[GitHub] spark issue #21379: [SPARK-24327][SQL] Add an option to quote a partition co...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21379 @maropu The fix looks good to me. Thanks for changing the solution. Could you update the PR title and description? --- - To

[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21379#discussion_r195314034 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala --- @@ -78,7 +84,17 @@ private[sql] object

[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21379#discussion_r195313861 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala --- @@ -99,10 +115,21 @@ private[sql] object

[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21379#discussion_r195313934 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala --- @@ -51,7 +53,11 @@ private[sql] object

[GitHub] spark pull request #21379: [SPARK-24327][SQL] Add an option to quote a parti...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21379#discussion_r195313083 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala --- @@ -1206,4 +1211,47 @@ class JDBCSuite extends SparkFunSuite

[GitHub] spark issue #21288: [SPARK-24206][SQL] Improve FilterPushdownBenchmark bench...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21288 @maropu Could you fix the style? BTW, based on the latest result, Parquet is generally faster than ORC. cc @dongjoon-hyun @rdblue

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21529 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21553#discussion_r195254446 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1314,6 +1314,31 @@ object SQLConf { "Other c

[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21553#discussion_r195254239 --- Diff: docs/configuration.md --- @@ -472,15 +472,17 @@ Apart from these, the following properties are also available, and may be useful 20

[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21549 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21499: [SPARK-24468][SQL] Handle negative scale when adjusting ...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21499 Thank you for fixing this! In Spark 3.0, we might introduce a mode for strict ANSI compliance. We can handle the cases in https://github.com/apache/spark/pull/20350

[GitHub] spark pull request #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow...

2018-06-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21549#discussion_r195153892 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/xml/UDFXPathUtilSuite.scala --- @@ -77,6 +77,23 @@ class

<    8   9   10   11   12   13   14   15   16   17   >