[GitHub] spark issue #15393: [HOTFIX][BUILD] Do not use contains in Option in JdbcRel...

2016-10-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15393 Actually, I hit this before. Sorry, I did not catch it when I merging the PR. Next time, I will be more careful about it. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #15393: [HOTFIX][BUILD] Do not use contains in Option in JdbcRel...

2016-10-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15393 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15393: [HOTFIX][BUILD] Do not use contains in Option in JdbcRel...

2016-10-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15393 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #15334: [SPARK-10367][SQL][WIP] Support Parquet logical type INT...

2016-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15334 Saw @viirya submitted a PR for the same issue: https://github.com/apache/spark/pull/7793 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #15388: [SPARK-17821][SQL] Support And and Or in Expression Cano...

2016-10-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15388 Why the following case can pass? ```Scala setTest(1, bUpper > 100 || aUpper <= 10 && aUpper > bUpper, bUpper > 100 ||

[GitHub] spark issue #15388: [SPARK-17821][SQL] Support And and Or in Expression Cano...

2016-10-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15388 BTW, we need to add more mixed cases in the test cases when supporting `And` and `Or` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #15388: [SPARK-17821][SQL] Support And and Or in Expression Cano...

2016-10-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15388 LGTM Forgot the precedence of `AND` is higher than `OR`. BTW, also checked SQL. It follows the same precedence order. That is, `AND` is higher than `OR`. ``` From

[GitHub] spark issue #15388: [SPARK-17821][SQL] Support And and Or in Expression Cano...

2016-10-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15388 @rxin Agree. Sorry for that. Will be more careful in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #15292: [SPARK-17719][SPARK-17776][SQL] Unify and tie up ...

2016-10-09 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15292#discussion_r82537727 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -17,47 +17,132 @@ package

[GitHub] spark issue #15399: [SPARK-17819][SQL] Support default database in connectio...

2016-10-09 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15399 First, I am not familiar with the code in this component. Thus, I am not the right person to review it. Second, when I going over the pending JIRA list, I found many bugs that are

[GitHub] spark pull request #15316: [SPARK-17751] [SQL] Remove spark.sql.eagerAnalysi...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15316#discussion_r82643571 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/AnalysisException.scala --- @@ -43,6 +43,11 @@ class AnalysisException protected[sql

[GitHub] spark pull request #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE an...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15358#discussion_r82655758 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -52,9 +52,15 @@ case class CatalogStorageFormat

[GitHub] spark issue #15316: [SPARK-17751] [SQL] Remove spark.sql.eagerAnalysis and O...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15316 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15417: [SPARK-17851][SQL][MINOR][TESTS] Update invalid test sql...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15417 Is this the only one? If you add the commands in the function `comparePlans` of `PlanTest.scala`, you can see more. ```Scala SimpleAnalyzer.checkAnalysis(plan1

[GitHub] spark issue #15190: [SPARK-17620][SQL] Determine Serde by hive.default.filef...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15190 cc @yhuai @cloud-fan Based on the above PR discussion, it sounds like this PR is ok to merge. What do you think? Thank you! --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15360 Will review this tonight. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15292: [SPARK-17719][SPARK-17776][SQL] Unify and tie up options...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15292 Thank you for working on it! Please update the PR description. Ping me when it is done. Then, I can merge it. : ) --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #15398: [SPARK-17647][SQL] Fix backslash escaping in 'LIKE' patt...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15398 Also cc @yhuai and @JoshRosen @mengxr Please check whether the changes here can satisfy what you want. Thanks! --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #15292: [SPARK-17719][SPARK-17776][SQL] Unify and tie up options...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15292 Sorry, I did not explain it in details. In this PR, we had a bug fix. We need a separate bullet in the PR description. Previously, when attempting to make a database connection, we pass

[GitHub] spark issue #15292: [SPARK-17719][SPARK-17776][SQL] Unify and tie up options...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15292 Thanks! Merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #14702: [SPARK-15694] Implement ScriptTransformation in sql/core...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14702 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-le...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15360#discussion_r82730527 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -358,50 +358,180 @@ class StatisticsSuite extends QueryTest with

[GitHub] spark pull request #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-le...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15360#discussion_r82730881 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -358,50 +358,180 @@ class StatisticsSuite extends QueryTest with

[GitHub] spark pull request #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-le...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15360#discussion_r82731661 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeColumnCommand.scala --- @@ -62,7 +62,7 @@ case class

[GitHub] spark pull request #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-le...

2016-10-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15360#discussion_r82731979 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -358,50 +358,180 @@ class StatisticsSuite extends QueryTest with

[GitHub] spark issue #15230: [SPARK-17657] [SQL] Disallow Users to Change Table Type

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15230 cc @cloud-fan Could you review it again? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15230: [SPARK-17657] [SQL] Disallow Users to Change Table Type

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15230 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15417: [SPARK-17851][SQL][MINOR][TESTS] Update invalid test sql...

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15417 You can make a try. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15398: [SPARK-17647][SQL] Fix backslash escaping in 'LIKE' patt...

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15398 I see. Agree. Will do an investigation with @jodersky Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #14847: [SPARK-17254][SQL] Filter can stop when the condition is...

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14847 @viirya This is a pretty interesting. Based on the suggestion from @rxin let us discuss it? Also cc @ioana-delaney @nsyca --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #15190: [SPARK-17620][SQL] Determine Serde by hive.default.filef...

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15190 Added them to your test cases. : ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15295: [SPARK-17720][SQL] introduce static SQL conf

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15295 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15295: [SPARK-17720][SQL] introduce static SQL conf

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15295 Merging to master! Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15434: [SPARK-17873][SQL] ALTER TABLE RENAME TO should allow us...

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15434 Just FYI. Hive allows the following changes: ```SQL ALTER TABLE db1.tbl RENAME TO db2.tbl2 ``` --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request #15230: [SPARK-17657] [SQL] Disallow Users to Change Tabl...

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15230#discussion_r82940270 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -225,6 +225,11 @@ case class AlterTableSetPropertiesCommand

[GitHub] spark issue #15230: [SPARK-17657] [SQL] Disallow Users to Change Table Type

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15230 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #15434: [SPARK-17873][SQL] ALTER TABLE RENAME TO should a...

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15434#discussion_r82944802 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -459,11 +459,20 @@ class SessionCatalog

[GitHub] spark issue #15434: [SPARK-17873][SQL] ALTER TABLE RENAME TO should allow us...

2016-10-11 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15434 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15434: [SPARK-17873][SQL] ALTER TABLE RENAME TO should allow us...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15434 cc @zhzhan Just FYI, the behavior of this DDL is different from Hive. If your team is migrating your Hive to Spark, you need to double check your scripts containing ALTER TABLE RENAME TO

[GitHub] spark pull request #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-le...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15360#discussion_r82947471 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -358,53 +358,189 @@ class StatisticsSuite extends QueryTest with

[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15360 cc @cloud-fan I do not have any more comment. Could you check this please? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request #15230: [SPARK-17657] [SQL] Disallow Users to Change Tabl...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15230#discussion_r83006784 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -225,6 +225,11 @@ case class AlterTableSetPropertiesCommand

[GitHub] spark pull request #9973: [SPARK-11989][SQL] Only use commit in JDBC data so...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/9973#discussion_r83048584 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala --- @@ -175,14 +187,18 @@ object JdbcUtils extends

[GitHub] spark pull request #9973: [SPARK-11989][SQL] Only use commit in JDBC data so...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/9973#discussion_r83049327 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala --- @@ -175,14 +187,18 @@ object JdbcUtils extends

[GitHub] spark issue #15048: [SPARK-17409] [SQL] Do Not Optimize Query in CTAS More T...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15048 Yeah. We should backport it to 2.0 Yeah. It affects both data source tables and hive serde tables. To fix it in Spark 2.0, we need to rewrite the fix since Spark 2.0 does not have a

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 Not sure whether you realize it. Since this PR changes the input parm of `Rand` and `Randn`, you also changes the external support. Now, users can do something like ```SQL

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 Since you are running mysql, the output of rand(0) is the same as rand(null)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #15048: [SPARK-17409] [SQL] Do Not Optimize Query in CTAS More T...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15048 Yeah, based on my understanding, it should cover the hive serde table. I will submit a PR to make sure it and also include the test case you provided above. Thank you! --- If your project is

[GitHub] spark issue #9973: [SPARK-11989][SQL] Only use commit in JDBC data source if...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/9973 The basic problem is multiple connections work on the same transaction. It is doable but might not be applicable as a general JDBC data source connector. Let us keep it as an open problem. If

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 I have a very general comment about the work you are working. Like what we are doing for the `LIKE` operation, we did an investigation on ANSI standard, and all the mainstream data stores

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 Let me show you an example: https://www.ibm.com/support/knowledgecenter/SSEPEK_11.0.0/sqlref/src/tpc/db2z_bif_rand.html This is the official document of `rand` in DB2 z/OS. Below is

[GitHub] spark pull request #15230: [SPARK-17657] [SQL] Disallow Users to Change Tabl...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15230#discussion_r83138864 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -111,6 +111,10 @@ private[spark] class HiveExternalCatalog

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 Unfortunately, not all the things have a standard to follow. That is why I suggested you to do a research about it. Like Oracle, it does not have such a function in their SQL-function list

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 At first, we do not strictly follow Hive. You can easily find many in Spark. I do not think this is an urgent JIRA, right? Like what @srowen replied in the JIRA, he does not think this is a bug

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 What is the behavior of `PostgreSQL`? Treating `NULL` as zero? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 Now, at least we have four options, when users setting `NULL` as a seed number for `rand`: 1. Hive/MySQL - `NULL` is equivalent to `0` 2. DB2 - when the seed is `NULL`, `rand` returns

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 @HyukjinKwon Could you document the behavior in the description of `rand` function? Checked whether we have any missing test case? Not sure whether you are also can check whether `rand` in R and

[GitHub] spark pull request #15230: [SPARK-17657] [SQL] Disallow Users to Change Tabl...

2016-10-12 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15230#discussion_r83149074 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -111,6 +111,10 @@ private[spark] class HiveExternalCatalog

[GitHub] spark issue #15458: [SPARK-17899][SQL] add a debug mode to keep raw table pr...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15458 If we turn this flag on, I am just afraid that some DDL statements might not work well. I assume this flag is really used for the our Spark SQL developers. --- If your project is set up for

[GitHub] spark pull request #15459: [SPARK-17409] [SQL] [FOLLOW-UP] Do Not Optimize Q...

2016-10-13 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/15459 [SPARK-17409] [SQL] [FOLLOW-UP] Do Not Optimize Query in CTAS More Than Once ### What changes were proposed in this pull request? This follow-up PR is for addressing the [comment](https

[GitHub] spark issue #9973: [SPARK-11989][SQL] Only use commit in JDBC data source if...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/9973 Checked it with @huaxingao who worked for JDBC driver team before. Yeah, we are unable to do it using JDBC. In my previous team, we did it using the native connection methods instead of JDBC. It

[GitHub] spark pull request #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15398#discussion_r83349251 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala --- @@ -54,20 +57,32 @@ class

[GitHub] spark pull request #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15398#discussion_r83349295 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala --- @@ -90,6 +105,28 @@ class

[GitHub] spark pull request #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15398#discussion_r83349306 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala --- @@ -90,6 +105,28 @@ class

[GitHub] spark issue #15460: [SPARK-17903][SQL] MetastoreRelation should talk to exte...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15460 LGTM except one comment about test case --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15460: [SPARK-17903][SQL] MetastoreRelation should talk ...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15460#discussion_r83349707 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -636,6 +637,13 @@ private[spark] class HiveExternalCatalog

[GitHub] spark pull request #15478: [SPARK-17899][SQL][DO-NOT-MERGE] Check the Impact...

2016-10-13 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/15478 [SPARK-17899][SQL][DO-NOT-MERGE] Check the Impact When Turning On spark.sql.debug ### What changes were proposed in this pull request? **DO NOT MERGE** This PR is just to do a

[GitHub] spark issue #15190: [SPARK-17620][SQL] Determine Serde by hive.default.filef...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15190 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #15478: [SPARK-17899][SQL][DO-NOT-MERGE] Check the Impact...

2016-10-13 Thread gatorsmile
Github user gatorsmile closed the pull request at: https://github.com/apache/spark/pull/15478 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15478: [SPARK-17899][SQL][DO-NOT-MERGE] Check the Impact When T...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15478 `ALTER TABLE`, `ANALYZE TABLE` and `CREATE TABLE LIKE` do not work well when `spark.sql.debug` is set to true. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #15458: [SPARK-17899][SQL] add a debug mode to keep raw table pr...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15458 Just FYI. https://github.com/apache/spark/pull/15478 shows at least three DDL commands do not work properly when `spark.sql.debug` is set to `true`: `ALTER TABLE`, `ANALYZE TABLE` and `CREATE

[GitHub] spark issue #15478: [SPARK-17899][SQL][DO-NOT-MERGE] Check the Impact When T...

2016-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15478 Yeah, I see. This is just for showing the impact. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #15465: [SPARK-17909][SQL] we should create table before writing...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15465 Not sure what is the reason in the original design. If we write data before creating the table, users will never get partial results. If we first create a table, and then write data

[GitHub] spark issue #15190: [SPARK-17620][SQL] Determine Serde by hive.default.filef...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15190 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15190: [SPARK-17620][SQL] Determine Serde by hive.default.filef...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15190 Merging to master! Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #15458: [SPARK-17899][SQL] add a debug mode to keep raw t...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15458#discussion_r83496528 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -915,4 +915,9 @@ object StaticSQLConf { .internal

[GitHub] spark issue #15190: [SPARK-17620][SQL] Determine Serde by hive.default.filef...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15190 It sounds like the other merged PRs made a change and impact your code. Please resolve it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #15494: [SPARK-17947] [SQL] Add Doc and Comment about spa...

2016-10-14 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/15494 [SPARK-17947] [SQL] Add Doc and Comment about spark.sql.debug ### What changes were proposed in this pull request? Just document the impact of `spark.sql.debug`: When enabling the

[GitHub] spark issue #15434: [SPARK-17873][SQL] ALTER TABLE RENAME TO should allow us...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15434 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15434: [SPARK-17873][SQL] ALTER TABLE RENAME TO should allow us...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15434 cc @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15398#discussion_r83509641 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala --- @@ -74,6 +107,31 @@ class

[GitHub] spark issue #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in 'LIKE'...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15398 As @jodersky said, SQL2003 syntax is like ``` WHERE expression [NOT] LIKE string_pattern [ESCAPE escape_sequence] ``` ``` - ESCAPE escape_sequence Allows you to search for

[GitHub] spark issue #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in 'LIKE'...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15398 @jodersky Please follow @rxin 's suggestion and submit a separate PR to support it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appe

[GitHub] spark pull request #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in...

2016-10-14 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15398#discussion_r83512274 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala --- @@ -74,6 +107,31 @@ class

[GitHub] spark pull request #15502: [SPARK-17892] [SQL] [2.0] Do Not Optimize Query i...

2016-10-15 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/15502 [SPARK-17892] [SQL] [2.0] Do Not Optimize Query in CTAS More Than Once #15048 ### What changes were proposed in this pull request? This PR is to backport https://github.com/apache/spark

[GitHub] spark issue #15494: [SPARK-17947] [SQL] Add Doc and Comment about spark.sql....

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15494 You can see the failed test cases in the PR: https://github.com/apache/spark/pull/15478 `ANALYZE TABLE` will fail due to the [checking](https://github.com/apache/spark/blob

[GitHub] spark issue #15495: [SPARK-17620][SQL] Determine Serde by hive.default.filef...

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15495 @yhuai Do you think it is good enough to merge? Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in 'LIKE'...

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15398 Also cc @yhuai , @JoshRosen and @mengxr --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in...

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15398#discussion_r83537154 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala --- @@ -74,6 +107,31 @@ class

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 Like what I mentioned above, the current change in this PR also allows `rand` to take an expression that returns a value of a built-in integer/long data type. Thus, you also need to test them

[GitHub] spark pull request #15432: [SPARK-17854][SQL] rand/randn allows null/long as...

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15432#discussion_r83537639 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala --- @@ -50,22 +54,17 @@ abstract class RDG

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 Here, I am talking about the black box testing. If you add the new capability to any external function, you should add it in the test cases. This is very fundamental when we developing

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 One more suggestion to your future PR. Whenever you submit a PR, please try to improve the test case coverage. This can help you find bugs in your codes and also benefit the whole community

[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15432 Regarding the test cases of R and PythonR, I am fine if you do not add them into the code base. However, please at least run them manually. We hit many surprise bugs in the past just because

[GitHub] spark issue #14531: [SPARK-17353] [SPARK-16943] [SPARK-16942] [SQL] Fix mult...

2016-10-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14531 Sure, will submit a PR for it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15218: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15218#discussion_r83560546 --- Diff: docs/configuration.md --- @@ -1334,6 +1334,17 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark pull request #15218: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15218#discussion_r83560691 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -61,6 +59,21 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #15218: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15218#discussion_r83561571 --- Diff: docs/configuration.md --- @@ -1334,6 +1334,17 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark pull request #15218: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15218#discussion_r83562599 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -408,4 +474,5 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #15218: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15218#discussion_r83562596 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -109,6 +109,72 @@ class TaskSchedulerImplSuite extends

<    4   5   6   7   8   9   10   11   12   13   >