spark git commit: [SPARK-15286][SQL] Make the output readable for EXPLAIN CREATE TABLE and DESC EXTENDED

2016-06-03 Thread wenchen
nch: refs/heads/master Commit: eb10b481ca51a3e010c168a1fb0baf1a94dcadfa Parents: e526913 Author: gatorsmile <gatorsm...@gmail.com> Authored: Fri Jun 3 13:56:22 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Fri Jun

spark git commit: [SPARK-15286][SQL] Make the output readable for EXPLAIN CREATE TABLE and DESC EXTENDED

2016-06-03 Thread wenchen
3070 from gatorsmile/betterExplainCatalogTable. (cherry picked from commit eb10b481ca51a3e010c168a1fb0baf1a94dcadfa) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/35765a0e T

spark git commit: [SPARK-15722][SQL] Disallow specifying schema in CTAS statement

2016-06-03 Thread wenchen
no-column. (cherry picked from commit b1cc7da3e3c36cee59825e7b95e14b00ab1fe196) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3670b2c6 Tree: http://git-wip-us.apache.org/

spark git commit: [SPARK-15722][SQL] Disallow specifying schema in CTAS statement

2016-06-03 Thread wenchen
er Commit: b1cc7da3e3c36cee59825e7b95e14b00ab1fe196 Parents: 11c83f8 Author: Andrew Or <and...@databricks.com> Authored: Fri Jun 3 14:39:41 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Fri Jun 3 14:39:41 2016 -0700

spark git commit: [SPARK-15789][SQL] Allow reserved keywords in most places

2016-06-07 Thread wenchen
How was this patch tested? I have added/move/renamed test in the catalyst `*ParserSuite`s. Author: Herman van Hovell <hvanhov...@databricks.com> Closes #13534 from hvanhovell/SPARK-15789. (cherry picked from commit 91fbc880b69bddcf5310afecc49df1102408e1f3) Signed-off-by: Wenchen Fan <wenc...@da

spark git commit: [SPARK-15789][SQL] Allow reserved keywords in most places

2016-06-07 Thread wenchen
ored: Tue Jun 7 17:01:11 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Tue Jun 7 17:01:11 2016 -0700 -- .../apache/spark/sql/catalyst/parser/SqlBase.g4 | 30 ++-- .../spar

spark git commit: Revert "[SPARK-15753][SQL] Move Analyzer stuff to Analyzer from DataFrameWriter"

2016-06-12 Thread wenchen
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f5d38c39 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f5d38c39 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f5d38c39 Branch: refs/heads/master Commit: f5d38c39255cc75325c6639561bfec1bc051f788 Parents: caebd7f Author

spark git commit: Revert "[SPARK-15753][SQL] Move Analyzer stuff to Analyzer from DataFrameWriter"

2016-06-12 Thread wenchen
19a114 Author: Wenchen Fan <wenc...@databricks.com> Authored: Sun Jun 12 16:53:39 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Sun Jun 12 16:53:39 2016 -0700 -- .../spark/sql/catalyst/analysi

spark git commit: [SPARK-15870][SQL] DataFrame can't execute after uncacheTable.

2016-06-12 Thread wenchen
st> Closes #13596 from ueshin/issues/SPARK-15870. (cherry picked from commit caebd7f2622340fc081bb9a2ea6a0b246f1e3a3f) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commi

spark git commit: [SPARK-15870][SQL] DataFrame can't execute after uncacheTable.

2016-06-12 Thread wenchen
e.org/repos/asf/spark/diff/caebd7f2 Branch: refs/heads/master Commit: caebd7f2622340fc081bb9a2ea6a0b246f1e3a3f Parents: 20b8f2c Author: Takuya UESHIN <ues...@happy-camper.st> Authored: Sun Jun 12 16:37:44 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Sun Jun 12 16:37:44 2016 -0700

spark git commit: [SPARK-15804][SQL] Include metadata in the toStructType

2016-06-09 Thread wenchen
igned-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eb9e8fc0 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/eb9e8fc0 Diff: http://git-wip-us.apache.org/repos/asf/spar

spark git commit: [SPARK-15804][SQL] Include metadata in the toStructType

2016-06-09 Thread wenchen
.@us.ibm.com> Authored: Thu Jun 9 09:50:09 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu Jun 9 09:50:09 2016 -0700 -- .../spark/sql/catalyst/expressions/package.scala | 2 +- ..

spark git commit: [SPARK-15677][SQL] Query with scalar sub-query in the SELECT list throws UnsupportedOperationException

2016-06-03 Thread wenchen
://git-wip-us.apache.org/repos/asf/spark/diff/9e2eb13c Branch: refs/heads/master Commit: 9e2eb13ca59fc7ac66c6accd49469f339700b23b Parents: 8fa00dd Author: Ioana Delaney <ioanamdela...@gmail.com> Authored: Fri Jun 3 12:04:27 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Fri Jun 3

spark git commit: [SPARK-15677][SQL] Query with scalar sub-query in the SELECT list throws UnsupportedOperationException

2016-06-03 Thread wenchen
na Delaney <ioanamdela...@gmail.com> Closes #13418 from ioana-delaney/scalarSubV2. (cherry picked from commit 9e2eb13ca59fc7ac66c6accd49469f339700b23b) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apac

spark git commit: [MINOR] Clean up several build warnings, mostly due to internal use of old accumulators

2016-06-14 Thread wenchen
46cdb2ef111 Parents: 6e8cdef Author: Sean Owen <so...@cloudera.com> Authored: Tue Jun 14 09:40:07 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Tue Jun 14 09:40:07 2016 -0700 -

spark git commit: [SPARK-15655][SQL] Fix Wrong Partition Column Order when Fetching Partitioned Tables

2016-06-14 Thread wenchen
//git-wip-us.apache.org/repos/asf/spark/diff/bc02d011 Branch: refs/heads/master Commit: bc02d011294fcd1ab07b9baf1011c3f2bdf749d9 Parents: 6151d26 Author: gatorsmile <gatorsm...@gmail.com> Authored: Tue Jun 14 09:58:06 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed

spark git commit: [SPARK-15655][SQL] Fix Wrong Partition Column Order when Fetching Partitioned Tables

2016-06-14 Thread wenchen
hor: gatorsmile <gatorsm...@gmail.com> Closes #13400 from gatorsmile/partitionedTableFetch. (cherry picked from commit bc02d011294fcd1ab07b9baf1011c3f2bdf749d9) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-15864][SQL] Fix Inconsistent Behaviors when Uncaching Non-cached Tables

2016-06-14 Thread wenchen
uncacheNonCachedTable. (cherry picked from commit df4ea6614d709ee66f1ceb966df6216b125b8ea1) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/51593704 Tree: http://git-wip-us.apache.org/repos/asf/spark/tr

spark git commit: [SPARK-15864][SQL] Fix Inconsistent Behaviors when Uncaching Non-cached Tables

2016-06-14 Thread wenchen
e66f1ceb966df6216b125b8ea1 Parents: c5b7355 Author: gatorsmile <gatorsm...@gmail.com> Authored: Tue Jun 14 11:44:37 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Tue Jun 14 11:44:37 2016 -0700 -- .../scal

spark git commit: [SPARK-15915][SQL] Logical plans should use subqueries eliminated plan when override sameResult.

2016-06-15 Thread wenchen
thored: Wed Jun 15 10:05:19 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Wed Jun 15 10:05:19 2016 -0700 -- .../spark/sql/catalyst/plans/logical/LocalRelation.scala | 11 +++ .../org/apach

spark git commit: [SPARK-15910][SQL] Check schema consistency when using Kryo encoder to convert DataFrame to Dataset

2016-06-13 Thread wenchen
f/spark/commit/7b9071ee Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7b9071ee Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7b9071ee Branch: refs/heads/master Commit: 7b9071eeaa62fd9a51d9e94cfd479224b8341517 Parents: a6babca Author: Sean Zhong <seanzh...@databricks.com> Authored: Mon

spark git commit: [SPARK-15910][SQL] Check schema consistency when using Kryo encoder to convert DataFrame to Dataset

2016-06-13 Thread wenchen
cannot cast IntegerType to BinaryType; ... ``` ## How was this patch tested? Unit test. Author: Sean Zhong <seanzh...@databricks.com> Closes #13632 from clockfly/spark-15910. (cherry picked from commit 7b9071eeaa62fd9a51d9e94cfd479224b8341517) Signed-off-by: Wenchen Fan <we

spark git commit: [SPARK-15704][SQL] add a test case in DatasetAggregatorSuite for regression testing

2016-06-05 Thread wenchen
13446 from inouehrs/fix_aggregate. (cherry picked from commit 79268aa461abd237bc4f96a7d31457c98e11798c) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/790de600 Tree: htt

spark git commit: [SPARK-15704][SQL] add a test case in DatasetAggregatorSuite for regression testing

2016-06-05 Thread wenchen
: refs/heads/master Commit: 79268aa461abd237bc4f96a7d31457c98e11798c Parents: 26c1089 Author: Hiroshi Inoue <inoue...@jp.ibm.com> Authored: Sun Jun 5 20:10:33 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed

spark git commit: [SPARK-15549][SQL] Disable bucketing when the output doesn't contain all bucketing columns

2016-05-28 Thread wenchen
...@gmail.com> Authored: Sat May 28 10:19:29 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Sat May 28 10:19:29 2016 -0700 -- .../org/apache/spark/sql/execution/ExistingRDD.scala | 13

spark git commit: [SPARK-15549][SQL] Disable bucketing when the output doesn't contain all bucketing columns

2016-05-28 Thread wenchen
is bucketed 3. the output contains all bucketing columns ## How was this patch tested? Updated test cases to reflect the changes. Author: Yadong Qi <qiyadong2...@gmail.com> Closes #13321 from watermen/SPARK-15549. (cherry picked from commit b4c32c4952f7af2733258aa4e27f21e8832c8a3a)

spark git commit: [SPARK-15647][SQL] Fix Boundary Cases in OptimizeCodegen Rule

2016-05-31 Thread wenchen
dc7d329dbd) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6347ff51 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6347ff51 Diff: http://git-wip-us.apache.

spark git commit: [SPARK-15647][SQL] Fix Boundary Cases in OptimizeCodegen Rule

2016-05-31 Thread wenchen
atorsmile <gatorsm...@gmail.com> Authored: Tue May 31 10:08:00 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Tue May 31 10:08:00 2016 -0700 -- .../sql/catalyst/optimizer/Optimizer.scala

spark git commit: [SPARK-15076][SQL] Add ReorderAssociativeOperator optimizer

2016-06-02 Thread wenchen
<dongj...@apache.org> Authored: Thu Jun 2 09:48:58 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu Jun 2 09:48:58 2016 -0700 -- .../sql/catalyst/optimizer/Optimizer.scala | 39 ...

spark git commit: [SPARK-15495][SQL] Improve the explain output for Aggregation operator

2016-06-01 Thread wenchen
this patch tested? Manual test and existing UT. Author: Sean Zhong <seanzh...@databricks.com> Closes #13363 from clockfly/verbose3. (cherry picked from commit d5012c274036463c47a751cfe9977ade3a68b668) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apa

spark git commit: [SPARK-15495][SQL] Improve the explain output for Aggregation operator

2016-06-01 Thread wenchen
epos/asf/spark/tree/d5012c27 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d5012c27 Branch: refs/heads/master Commit: d5012c274036463c47a751cfe9977ade3a68b668 Parents: 1f43562 Author: Sean Zhong <seanzh...@databricks.com> Authored: Wed Jun 1 09:58:01 2016 -0700 Committer: We

spark git commit: [SPARK-15692][SQL] Improves the explain output of several physical plans by displaying embedded logical plan in tree style

2016-06-01 Thread wenchen
commit c8fb776d4a0134c47f90272c4bd5e4bba902aae5) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/35195f6c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/35195

spark git commit: [SPARK-16024][SQL][TEST] Verify Column Comment for Data Source Tables

2016-06-22 Thread wenchen
er Commit: 9f990fa3f9e0b798d8018cf4132b93a3468f33bb Parents: 4f869f8 Author: gatorsmile <gatorsm...@gmail.com> Authored: Thu Jun 23 09:12:20 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu Jun 2

spark git commit: [SPARK-15230][SQL] distinct() does not handle column name with dot properly

2016-06-22 Thread wenchen
ec7 Parents: 37f3be5 Author: bomeng <bm...@us.ibm.com> Authored: Thu Jun 23 11:06:19 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu Jun 23 11:06:19 2016 +0800 -- sql/core/src/main/scala/org/apache/s

spark git commit: [SPARK-15230][SQL] distinct() does not handle column name with dot properly

2016-06-22 Thread wenchen
2dd88beaddf555c74d90856ab040ec7) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5b4a9a4c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5b4a9a4c Diff: http

spark git commit: [SPARK-16024][SQL][TEST] Verify Column Comment for Data Source Tables

2016-06-22 Thread wenchen
ceComment. (cherry picked from commit 9f990fa3f9e0b798d8018cf4132b93a3468f33bb) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e0a43235 Tree: http://git-wip-us.apache.org/

spark git commit: [SPARK-15776][SQL] Divide Expression inside Aggregation function is casted to wrong type

2016-06-15 Thread wenchen
/tree/9bd80ad6 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9bd80ad6 Branch: refs/heads/master Commit: 9bd80ad6bd43462d16ce24cda77cdfaa336c4e02 Parents: 049e639 Author: Sean Zhong <seanzh...@databricks.com> Authored: Wed Jun 15 14:34:15 2016 -0

spark git commit: [SPARK-15824][SQL] Execute WITH .... INSERT ... statements immediately

2016-06-15 Thread wenchen
om> Authored: Wed Jun 15 21:33:26 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Wed Jun 15 21:33:26 2016 -0700 -- .../scala/org/apache/spark/sql/Dataset.scala| 2 +- .../apache/spark/sql/sour

spark git commit: [SPARK-15824][SQL] Execute WITH .... INSERT ... statements immediately

2016-06-15 Thread wenchen
nce of this problem in the ThriftSever). ## How was this patch tested? Added a test to `InsertSuite` Author: Herman van Hovell <hvanhov...@databricks.com> Closes #13678 from hvanhovell/SPARK-15824. (cherry picked from commit b75f454f946714b93fe561055cd53b0686187d2e) Signed-off-by: Wenchen

spark git commit: [SPARK-15983][SQL] Removes FileFormat.prepareRead

2016-06-16 Thread wenchen
-0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu Jun 16 10:24:29 2016 -0700 -- .../spark/ml/source/libsvm/LibSVMRelation.scala | 33 ++-- .../sql/execution/datasources/DataSource.scala | 5

spark git commit: [SPARK-15983][SQL] Removes FileFormat.prepareRead

2016-06-16 Thread wenchen
mat` interface clean. ## How was this patch tested? Existing tests. Author: Cheng Lian <l...@databricks.com> Closes #13698 from liancheng/remove-prepare-read. (cherry picked from commit 9ea0d5e326e08b914aa46f1eec8795688a61bf74) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Proj

spark git commit: [SPARK-15776][SQL] Divide Expression inside Aggregation function is casted to wrong type

2016-06-15 Thread wenchen
ull/13524 by Sephiroth-Lin Author: Sean Zhong <seanzh...@databricks.com> Closes #13651 from clockfly/SPARK-15776. (cherry picked from commit 9bd80ad6bd43462d16ce24cda77cdfaa336c4e02) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/a

spark git commit: [SPARK-13473][SQL] Don't push predicate through project with nondeterministic field(s)

2016-02-25 Thread wenchen
ect with a nondeterministic field. Optimized query plan shouldn't change in this case. Author: Cheng Lian <l...@databricks.com> Closes #11348 from liancheng/spark-13473-no-ppd-through-nondeterministic-project-field. (cherry picked from commit 3fa6491be66dad690ca5329dd32e7c82037ae8c1) Signed-off-by

spark git commit: [SPARK-13473][SQL] Don't push predicate through project with nondeterministic field(s)

2016-02-25 Thread wenchen
ect with a nondeterministic field. Optimized query plan shouldn't change in this case. Author: Cheng Lian <l...@databricks.com> Closes #11348 from liancheng/spark-13473-no-ppd-through-nondeterministic-project-field. (cherry picked from commit 3fa6491be66dad690ca5329dd32e7c82037ae8c1) Signed-off-by

spark git commit: [SPARK-13473][SQL] Don't push predicate through project with nondeterministic field(s)

2016-02-25 Thread wenchen
tabricks.com> Authored: Thu Feb 25 20:43:03 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu Feb 25 20:43:03 2016 +0800 -- .../sql/catalyst/optimizer/Optimizer.scala | 9 +

[3/5] spark git commit: [SPARK-13928] Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2016-03-18 Thread wenchen
http://git-wip-us.apache.org/repos/asf/spark/blob/8ef3399a/core/src/test/scala/org/apache/spark/ThreadingSuite.scala -- diff --git a/core/src/test/scala/org/apache/spark/ThreadingSuite.scala

spark git commit: [SPARK-13974][SQL] sub-query names do not need to be globally unique while generate SQL

2016-03-18 Thread wenchen
SQL string, but not all the time. This PR moves the `newSubqueryName` method to `class SQLBuilder` and remove `object SQLBuilder`. also addressed 2 minor comments in https://github.com/apache/spark/pull/11696 ## How was this patch tested? existing tests. Author: Wenchen Fan <wenc...@databric

spark git commit: [SPARK-13698][SQL] Fix Analysis Exceptions when Using Backticks in Generate

2016-03-09 Thread wenchen
ip Biswal <dbis...@us.ibm.com> Authored: Wed Mar 9 21:49:37 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Wed Mar 9 21:49:37 2016 +0800 -- .../spark/sql/catalyst/parser/CatalystQl.scala | 10 +

[4/5] spark git commit: [SPARK-13928] Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2016-03-19 Thread wenchen
http://git-wip-us.apache.org/repos/asf/spark/blob/8ef3399a/core/src/main/scala/org/apache/spark/internal/Logging.scala -- diff --git a/core/src/main/scala/org/apache/spark/internal/Logging.scala

[5/5] spark git commit: [SPARK-13928] Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2016-03-19 Thread wenchen
was this patch tested? existing tests. Author: Wenchen Fan <wenc...@databricks.com> Closes #11764 from cloud-fan/logger. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8ef3399a Tree: http://git-wip-us.apache.org/rep

spark git commit: [SPARK-13732][SPARK-13797][SQL] Remove projectList from Window and Eliminate useless Window

2016-03-10 Thread wenchen
a462 Parents: 4d535d1 Author: gatorsmile <gatorsm...@gmail.com> Authored: Fri Mar 11 11:59:18 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Fri Mar 11 11:59:18 2016 +0800 -- .../spark/sql/catal

spark git commit: [SPARK-12718][SPARK-13720][SQL] SQL generation support for window functions

2016-03-10 Thread wenchen
any thanks to gatorsmile for the initial discussion and test cases! ## How was this patch tested? new tests in `LogicalPlanToSQLSuite` Author: Wenchen Fan <wenc...@databricks.com> Closes #11555 from cloud-fan/window. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http:

spark git commit: [HOT-FIX] fix compile

2016-03-10 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 6871cc8f3 -> 74c4e2651 [HOT-FIX] fix compile Fix the compilation failure introduced by https://github.com/apache/spark/pull/11555 because of a merge conflict. Author: Wenchen Fan <wenc...@databricks.com> Closes #11648 from

[2/2] spark git commit: [SPARK-13880][SPARK-13881][SQL] Rename DataFrame.scala Dataset.scala, and remove LegacyFunctions

2016-03-14 Thread wenchen
Xin <r...@databricks.com> Authored: Tue Mar 15 10:39:07 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Tue Mar 15 10:39:07 2016 +0800 -- project/MimaExcludes.scala |1 + ..

spark git commit: [MINOR][SQL] Remove some unused imports in datasources.

2016-04-12 Thread wenchen
0 Parents: 768b3d6 Author: hyukjinkwon <gurwls...@gmail.com> Authored: Wed Apr 13 10:20:03 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Wed Apr 13 10:20:03 2016 +0800 -- .../apache/s

spark git commit: [SPARK-14554][SQL][FOLLOW-UP] use checkDataset to check the result

2016-04-12 Thread wenchen
How was this patch tested? N/A Author: Wenchen Fan <wenc...@databricks.com> Closes #12346 from cloud-fan/tmp. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a5f8c9b1 Tree: http://git-wip-us.apache.org/repos/asf/s

spark git commit: [SPARK-14296][SQL] whole stage codegen support for Dataset.map

2016-04-05 Thread wenchen
ier to adapt to whole stage codegen framework. ## How was this patch tested? new test in `WholeStageCodegenSuite` Author: Wenchen Fan <wenc...@databricks.com> Closes #12087 from cloud-fan/map. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/

spark git commit: [SPARK-12719][SQL] SQL generation support for Generate

2016-03-19 Thread wenchen
596 ## How was this patch tested? new tests in `LogicalPlanToSQLSuite` Author: Wenchen Fan <wenc...@databricks.com> Closes #11696 from cloud-fan/generate. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1974d1d3 Tree: h

[1/5] spark git commit: [SPARK-13928] Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2016-03-19 Thread wenchen
Repository: spark Updated Branches: refs/heads/master ea9ca6f04 -> 8ef3399af http://git-wip-us.apache.org/repos/asf/spark/blob/8ef3399a/sql/hive/src/main/scala/org/apache/spark/sql/hive/SQLBuilder.scala -- diff --git

[2/5] spark git commit: [SPARK-13928] Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2016-03-19 Thread wenchen
http://git-wip-us.apache.org/repos/asf/spark/blob/8ef3399a/mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala -- diff --git

spark git commit: [SPARK-13764][SQL] Parse modes in JSON data source

2016-03-21 Thread wenchen
nch: refs/heads/master Commit: e474088144cdd2632cf2fef6b2cf10b3cd191c23 Parents: f58319a Author: hyukjinkwon <gurwls...@gmail.com> Authored: Mon Mar 21 15:42:35 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Mon Mar 2

spark git commit: [SPARK-14019][SQL] Remove noop SortOrder in Sort

2016-03-20 Thread wenchen
; Authored: Mon Mar 21 10:34:54 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Mon Mar 21 10:34:54 2016 +0800 -- .../sql/catalyst/optimizer/Optimizer.scala | 12 .../optimizer/Elim

spark git commit: [SPARK-13320][SQL] Support Star in CreateStruct/CreateArray and Error Handling when DataFrame/DataSet Functions using Star

2016-03-21 Thread wenchen
b7 Author: gatorsmile <gatorsm...@gmail.com> Authored: Tue Mar 22 08:21:02 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Tue Mar 22 08:21:02 2016 +0800 -- .../spark/sql/catalyst

spark git commit: [SPARK-13953][SQL] Specifying the field name for corrupted record via option at JSON datasource

2016-03-22 Thread wenchen
: f2e855f Author: hyukjinkwon <gurwls...@gmail.com> Authored: Tue Mar 22 20:30:48 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Tue Mar 22 20:30:48 2016 +0800 -- python/pyspark/sql/readwriter.p

spark git commit: [SPARK-14085][SQL] Star Expansion for Hash

2016-03-23 Thread wenchen
6f390f1f31ee60faa1662489b Parents: de4e48b Author: gatorsmile <gatorsm...@gmail.com> Authored: Thu Mar 24 11:13:36 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu Mar 24 11:13:36 2016 +0800 ---

spark git commit: [SPARK-13957][SQL] Support Group By Ordinal in SQL

2016-03-24 Thread wenchen
git-wip-us.apache.org/repos/asf/spark/tree/05f652d6 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/05f652d6 Branch: refs/heads/master Commit: 05f652d6c2bbd764a1dd5a45301811e14519486f Parents: 0874ff3 Author: gatorsmile <gatorsm...@gmail.com> Authored: Fri Mar 25 12:55:58 2016 +0800 C

spark git commit: [SPARK-12789][SQL] Support Order By Ordinal in SQL

2016-03-21 Thread wenchen
7 Parents: c35c60f Author: gatorsmile <gatorsm...@gmail.com> Authored: Mon Mar 21 18:08:41 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Mon Mar 21 18:08:41 2016 +0800 -- .../spark/sql/cataly

spark git commit: [SPARK-13976][SQL] do not remove sub-queries added by user when generate SQL

2016-03-19 Thread wenchen
ong term, we should find a way to add sub-queries correctly, so that arbitrary logical plans can be converted to SQL string. ## How was this patch tested? `LogicalPlanToSQLSuite` Author: Wenchen Fan <wenc...@databricks.com> Closes #11786 from cloud-fan/bug-fix. Project: http://git-wip-us.a

spark git commit: [SPARK-13637][SQL] use more information to simplify the code in Expand builder

2016-03-08 Thread wenchen
ion: * the `groupByExprs` parameter are all `Attribute`s * the `child` parameter is a `Project` that append aliased group by expressions to its child's output ## How was this patch tested? by existing tests. Author: Wenchen Fan <wenc...@databricks.com> Closes #11485 from cloud-fan/expand. Project: h

spark git commit: [SPARK-14158][SQL] implement buildReader for json data source

2016-03-29 Thread wenchen
ath. ## How was this patch tested? existing tests Author: Wenchen Fan <wenc...@databricks.com> Closes #11960 from cloud-fan/json. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/83775bc7 Tree: http://git-wip-us.apache.

spark git commit: [SPARK-14863][SQL] Cache TreeNode's hashCode by default

2016-04-22 Thread wenchen
010e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bdde010e Branch: refs/heads/master Commit: bdde010edbc79e506e183e2b9a2b9b19f7b226fb Parents: 39a77e1 Author: Josh Rosen <joshro...@databricks.com> Authored: Sat Apr 23 13:42:44 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.c

spark git commit: [SPARK-14609][SQL] Native support for LOAD DATA DDL command

2016-04-22 Thread wenchen
er Commit: e09ab5da8b02da98d7b2496d549c1d53cceb8728 Parents: 284b15d Author: Liang-Chi Hsieh <sim...@tw.ibm.com> Authored: Fri Apr 22 18:26:28 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Fri Apr 22 18:26:28 2016 +0800 --

spark git commit: [SPARK-12660][SPARK-14967][SQL] Implement Except Distinct by Left Anti Join

2016-04-29 Thread wenchen
7df33007d7a9780dafa2c740dbe6a3 Parents: e249e6f Author: gatorsmile <gatorsm...@gmail.com> Authored: Fri Apr 29 15:30:36 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Fri Apr 29 15:30:36 2016 +0800 -

spark git commit: [SPARK-15003] Use ConcurrentHashMap in place of HashMap for NewAccumulator.originals

2016-04-29 Thread wenchen
; Authored: Sat Apr 30 07:54:53 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Sat Apr 30 07:54:53 2016 +0800 -- .../scala/org/apache/spark/NewAccumulator.scala | 20 -

spark git commit: [SPARK-15321] Fix bug where Array[Timestamp] cannot be encoded/decoded correctly

2016-05-19 Thread wenchen
66ec249 Author: Sumedh Mungee <smun...@gmail.com> Authored: Fri May 20 12:30:04 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Fri May 20 12:30:04 2016 +0800 -- .../apache/spark/sql/catalyst/encod

spark git commit: [SPARK-11827][SQL] Adding java.math.BigInteger support in Java type inference for POJOs and Java collections

2016-05-19 Thread wenchen
(cherry picked from commit 17591d90e6873f30a042112f56a1686726ccbd60) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e6810e9c Tree: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-11827][SQL] Adding java.math.BigInteger support in Java type inference for POJOs and Java collections

2016-05-19 Thread wenchen
er Commit: 17591d90e6873f30a042112f56a1686726ccbd60 Parents: d5c47f8 Author: Kevin Yu <q...@us.ibm.com> Authored: Fri May 20 12:41:14 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Fri May 20 12:41:14 2016 +0800 ---

spark git commit: [SPARK-15367][SQL] Add refreshTable back

2016-05-20 Thread wenchen
ble in `hivecontext-compatibility` Author: gatorsmile <gatorsm...@gmail.com> Closes #13156 from gatorsmile/refreshTable. (cherry picked from commit 39fd469078271aa12f3163606000e06e382d35dc) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/sp

spark git commit: [SPARK-15367][SQL] Add refreshTable back

2016-05-20 Thread wenchen
/39fd4690 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/39fd4690 Branch: refs/heads/master Commit: 39fd469078271aa12f3163606000e06e382d35dc Parents: c94b34e Author: gatorsmile <gatorsm...@gmail.com> Authored: Fri May 20 14:38:25 2016 +0800 Committer: Wenchen Fan <wenc...@databri

spark git commit: [SPARK-15285][SQL] Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-23 Thread wenchen
(cherry picked from commit fa244e5a90690d6a31be50f2aa203ae1a2e9a1cf) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d0bcec15 Tree: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-15285][SQL] Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-23 Thread wenchen
er Commit: fa244e5a90690d6a31be50f2aa203ae1a2e9a1cf Parents: d207716 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Authored: Mon May 23 21:12:34 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Mon May 23 21:12:34 2016 -0700 ---

spark git commit: [SPARK-15315][SQL] Adding error check to the CSV datasource writer for unsupported complex data types.

2016-05-23 Thread wenchen
pes_SPARK-15315. (cherry picked from commit 03c7b7c4b9374f0cb6a29aeaf495bd21c2563de4) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4673b88b Tree: http://git-wip-us.a

spark git commit: [SPARK-15297][SQL] Fix Set -V Command

2016-05-18 Thread wenchen
ase. Author: gatorsmile <gatorsm...@gmail.com> Closes #13081 from gatorsmile/setVcommand. (cherry picked from commit 9c2a376e413b0701097b0784bd725e4ca87cd837) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-15297][SQL] Fix Set -V Command

2016-05-18 Thread wenchen
//git-wip-us.apache.org/repos/asf/spark/diff/9c2a376e Branch: refs/heads/master Commit: 9c2a376e413b0701097b0784bd725e4ca87cd837 Parents: ebfe3a1 Author: gatorsmile <gatorsm...@gmail.com> Authored: Thu May 19 10:05:53 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu

spark git commit: [SPARK-15379][SQL] check special invalid date

2016-05-22 Thread wenchen
nvolves UI changes, please attach a screenshot; otherwise, remove this) Author: wangyang <wangy...@haizhi.com> Closes #13169 from wangyang1992/invalid_date. (cherry picked from commit fc44b694bf5162b3a044768da4627b9969909829) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http

spark git commit: [SPARK-15379][SQL] check special invalid date

2016-05-22 Thread wenchen
-wip-us.apache.org/repos/asf/spark/tree/fc44b694 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fc44b694 Branch: refs/heads/master Commit: fc44b694bf5162b3a044768da4627b9969909829 Parents: 3eff65f Author: wangyang <wangy...@haizhi.com> Authored: Sun May 22 19:30:14 2016 -0700 C

spark git commit: [SPARK-11753][SQL][TEST-HADOOP2.2] Make allowNonNumericNumbers option work

2016-05-24 Thread wenchen
e`. Author: Liang-Chi Hsieh <sim...@tw.ibm.com> Author: Liang-Chi Hsieh <vii...@appier.com> Closes #9759 from viirya/fix-json-nonnumric. (cherry picked from commit c24b6b679c3efa053f7de19be73eb36dc70d9930) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-

spark git commit: [SPARK-15114][SQL] Column name generated by typed aggregate is super verbose

2016-05-21 Thread wenchen
t-wip-us.apache.org/repos/asf/spark/tree/5e1ee289 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5e1ee289 Branch: refs/heads/master Commit: 5e1ee28984b169eaab5d2f832921d32cf09de915 Parents: f39621c Author: Dilip Biswal <dbis...@us.ibm.com> Authored: Sat May 21 08:36:0

spark git commit: [SPARK-15114][SQL] Column name generated by typed aggregate is super verbose

2016-05-21 Thread wenchen
his patch tested? A new test was added in DataSetAggregatorSuite. Author: Dilip Biswal <dbis...@us.ibm.com> Closes #13045 from dilipbiswal/spark-15114. (cherry picked from commit 5e1ee28984b169eaab5d2f832921d32cf09de915) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Projec

spark git commit: [SPARK-15462][SQL][TEST] unresolved === false` is enough in testcases.

2016-05-21 Thread wenchen
9 Branch: refs/heads/master Commit: f39621c998a0fe91a5115f3f843c3ca8dd71c1ab Parents: 666bf2e Author: Dongjoon Hyun <dongj...@apache.org> Authored: Sat May 21 08:11:14 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.co

spark git commit: [SPARK-15462][SQL][TEST] unresolved === false` is enough in testcases.

2016-05-21 Thread wenchen
ses #13241 from dongjoon-hyun/SPARK-15462. (cherry picked from commit f39621c998a0fe91a5115f3f843c3ca8dd71c1ab) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/69f39552

spark git commit: [SPARK-15312][SQL] Detect Duplicate Key in Partition Spec and Table Properties

2016-05-22 Thread wenchen
park/tree/a11175ee Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a11175ee Branch: refs/heads/master Commit: a11175eecacd4a57325dab29fff9ebfad819f22f Parents: 6d0bfb9 Author: gatorsmile <gatorsm...@gmail.com> Authored: Sat May 21 23:56:10 2016 -0700 Committer: Wenchen Fan <wenc...@databri

spark git commit: [SPARK-15312][SQL] Detect Duplicate Key in Partition Spec and Table Properties

2016-05-22 Thread wenchen
ded test cases in DDLSuite Author: gatorsmile <gatorsm...@gmail.com> Closes #13095 from gatorsmile/detectDuplicate. (cherry picked from commit a11175eecacd4a57325dab29fff9ebfad819f22f) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/sp

spark git commit: [SPARK-15206][SQL] add testcases for distinct aggregate in having clause

2016-05-21 Thread wenchen
1 21:41:12 2016 -0700 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Sat May 21 21:41:12 2016 -0700 -- .../hive/execution/AggregationQuerySuite.scala | 31 1 file c

spark git commit: [SPARK-15206][SQL] add testcases for distinct aggregate in having clause

2016-05-21 Thread wenchen
nch. This is a followup PR for [#12974](https://github.com/apache/spark/pull/12974), which is for 1.6 branch. Author: xin Wu <xi...@us.ibm.com> Closes #12984 from xwu0226/SPARK-15206. (cherry picked from commit df9adb5ec994f054b2fa58e492867bbc5a60c234) Signed-off-by: Wenchen Fan <wenc...@da

spark git commit: [SPARK-14939][SQL] Add FoldablePropagation optimizer

2016-05-19 Thread wenchen
gj...@apache.org> Authored: Thu May 19 15:57:44 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Thu May 19 15:57:44 2016 +0800 -- .../catalyst/expressions/objects/objects.scala | 2 +-

spark git commit: [SPARK-14939][SQL] Add FoldablePropagation optimizer

2016-05-19 Thread wenchen
- INPUT +- Scan OneRowRelation[] ``` ## How was this patch tested? Pass the Jenkins tests including a new test case. Author: Dongjoon Hyun <dongj...@apache.org> Closes #12719 from dongjoon-hyun/SPARK-14939. (cherry picked from commit 5907ebfc11aac8029cfc6d5f8e91cd5f53af54fe) Signed-off-b

spark git commit: [SPARK-15185][SQL] InMemoryCatalog: Silent Removal of an Existent Table/Function/Partitions by Rename

2016-05-08 Thread wenchen
ify if HiveExternalCatalog also detects these errors. Author: gatorsmile <gatorsm...@gmail.com> Closes #12960 from gatorsmile/renameInMemoryCatalog. (cherry picked from commit e9131ec277731de4a73026f2fb4559182c236f84) Signed-off-by: Wenchen Fan <wenc...@databricks.com> Project: http://git-wi

spark git commit: [SPARK-15185][SQL] InMemoryCatalog: Silent Removal of an Existent Table/Function/Partitions by Rename

2016-05-08 Thread wenchen
pache.org/repos/asf/spark/tree/e9131ec2 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e9131ec2 Branch: refs/heads/master Commit: e9131ec277731de4a73026f2fb4559182c236f84 Parents: 454ba4d Author: gatorsmile <gatorsm...@gmail.com> Authored: Mon May 9 12:40:30 2016 +0800 Committer: We

spark git commit: [SPARK-15184][SQL] Fix Silent Removal of An Existent Temp Table by Rename Table

2016-05-08 Thread wenchen
5:18 2016 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Mon May 9 13:05:18 2016 +0800 -- .../sql/catalyst/catalog/SessionCatalog.scala | 9 +++ .../spark/sql/execution/command/DDLSuite.scala | 60 ++

  1   2   3   4   5   6   7   8   9   10   >