svn commit: r16352 - /dev/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz.md5 /release/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz.md5

2016-10-04 Thread rxin
Author: rxin Date: Tue Oct 4 17:24:16 2016 New Revision: 16352 Log: (empty) Added: release/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz.md5 - copied unchanged from r16351, dev/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz.md5 Removed: dev/spark/spark-2.0.1/spark

svn commit: r16354 - /dev/spark/spark-2.0.1/spark-2.0.1.tgz /release/spark/spark-2.0.1/spark-2.0.1.tgz

2016-10-04 Thread rxin
Author: rxin Date: Tue Oct 4 17:24:35 2016 New Revision: 16354 Log: (empty) Added: release/spark/spark-2.0.1/spark-2.0.1.tgz - copied unchanged from r16353, dev/spark/spark-2.0.1/spark-2.0.1.tgz Removed: dev/spark/spark-2.0.1/spark-2.0.1.tgz

svn commit: r16353 - /dev/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz.sha /release/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz.sha

2016-10-04 Thread rxin
Author: rxin Date: Tue Oct 4 17:24:25 2016 New Revision: 16353 Log: (empty) Added: release/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz.sha - copied unchanged from r16352, dev/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz.sha Removed: dev/spark/spark-2.0.1/spark

svn commit: r16350 - /dev/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz /release/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz

2016-10-04 Thread rxin
Author: rxin Date: Tue Oct 4 17:23:39 2016 New Revision: 16350 Log: (empty) Added: release/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz - copied unchanged from r16349, dev/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz Removed: dev/spark/spark-2.0.1/spark-2.0.1-bin

svn commit: r16349 - /dev/spark/spark-2.0.1/

2016-10-04 Thread rxin
Author: rxin Date: Tue Oct 4 17:18:24 2016 New Revision: 16349 Log: Add remaining 2.0.1 files Added: dev/spark/spark-2.0.1/ dev/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz (with props) dev/spark/spark-2.0.1/spark-2.0.1-bin-without-hadoop.tgz.asc dev/spark/spark-2.0.1

[2/3] spark-website git commit: Add Spark 2.0.1 release.

2016-10-04 Thread rxin
http://git-wip-us.apache.org/repos/asf/spark-website/blob/a8dce991/site/news/spark-2-0-1-released.html -- diff --git a/site/news/spark-2-0-1-released.html b/site/news/spark-2-0-1-released.html new file mode 100644 index 000..9

[3/3] spark-website git commit: Add Spark 2.0.1 release.

2016-10-04 Thread rxin
Add Spark 2.0.1 release. Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/a8dce991 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/a8dce991 Diff: http://git-wip-us.apache.org/repos/asf/spark-webs

[1/3] spark-website git commit: Add Spark 2.0.1 release.

2016-10-04 Thread rxin
Repository: spark-website Updated Branches: refs/heads/asf-site 7c96b646e -> a8dce9912 http://git-wip-us.apache.org/repos/asf/spark-website/blob/a8dce991/site/releases/spark-release-1-1-1.html -- diff --git a/site/releases/spar

spark git commit: [SPARK-17773] Input/Output] Add VoidObjectInspector

2016-10-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master b1b47274b -> d2dc8c4a1 [SPARK-17773] Input/Output] Add VoidObjectInspector ## What changes were proposed in this pull request? Added VoidObjectInspector to the list of PrimitiveObjectInspectors ## How was this patch tested? (Please explai

spark git commit: [SPARK-17702][SQL] Code generation including too many mutable states exceeds JVM size limit.

2016-10-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master c571cfb2d -> b1b47274b [SPARK-17702][SQL] Code generation including too many mutable states exceeds JVM size limit. ## What changes were proposed in this pull request? Code generation including too many mutable states exceeds JVM size lim

spark git commit: [SPARK-17112][SQL] "select null" via JDBC triggers IllegalArgumentException in Thriftserver

2016-10-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master 2bbecdec2 -> c571cfb2d [SPARK-17112][SQL] "select null" via JDBC triggers IllegalArgumentException in Thriftserver ## What changes were proposed in this pull request? Currently, Spark Thrift Server raises `IllegalArgumentException` for qu

spark git commit: [SPARK-17112][SQL] "select null" via JDBC triggers IllegalArgumentException in Thriftserver

2016-10-03 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 584393202 -> 7429199e5 [SPARK-17112][SQL] "select null" via JDBC triggers IllegalArgumentException in Thriftserver ## What changes were proposed in this pull request? Currently, Spark Thrift Server raises `IllegalArgumentException` fo

svn commit: r16342 - /dev/spark/spark-2.0.1/ /release/spark/spark-2.0.1/

2016-10-03 Thread rxin
Author: rxin Date: Mon Oct 3 21:45:15 2016 New Revision: 16342 Log: Release 2.0.1 Added: release/spark/spark-2.0.1/ - copied from r16341, dev/spark/spark-2.0.1/ Removed: dev/spark/spark-2.0.1/ - To unsubscribe

svn commit: r16341 - /dev/spark/spark-2.0.1/

2016-10-03 Thread rxin
Author: rxin Date: Mon Oct 3 19:25:38 2016 New Revision: 16341 Log: Add spark-2.0.1 Added: dev/spark/spark-2.0.1/ dev/spark/spark-2.0.1/spark-2.0.0-bin-hadoop2.7.tgz.sha dev/spark/spark-2.0.1/spark-2.0.0-bin-without-hadoop.tgz (with props) dev/spark/spark-2.0.1/spark-2.0.0-bin

spark git commit: [SPARK-17073][SQL] generate column-level statistics

2016-10-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master a27033c0b -> 7bf921276 [SPARK-17073][SQL] generate column-level statistics ## What changes were proposed in this pull request? Generate basic column statistics for all the atomic types: - numeric types: max, min, num of nulls, ndv (number

spark git commit: [SPARK-14914][CORE][SQL] Skip/fix some test cases on Windows due to limitation of Windows

2016-10-02 Thread rxin
Repository: spark Updated Branches: refs/heads/master f8d7fade4 -> 76dc2d907 [SPARK-14914][CORE][SQL] Skip/fix some test cases on Windows due to limitation of Windows ## What changes were proposed in this pull request? This PR proposes to fix/skip some tests failed on Windows. This PR takes

spark git commit: [SPARK-17509][SQL] When wrapping catalyst datatype to Hive data type avoid…

2016-10-02 Thread rxin
Repository: spark Updated Branches: refs/heads/master b88cb63da -> f8d7fade4 [SPARK-17509][SQL] When wrapping catalyst datatype to Hive data type avoid… ## What changes were proposed in this pull request? When wrapping catalyst datatypes to Hive data type, wrap function was doing an expens

spark git commit: [SPARK-17717][SQL] Add Exist/find methods to Catalog [FOLLOW-UP]

2016-10-01 Thread rxin
Repository: spark Updated Branches: refs/heads/master 4bcd9b728 -> af6ece33d [SPARK-17717][SQL] Add Exist/find methods to Catalog [FOLLOW-UP] ## What changes were proposed in this pull request? We added find and exists methods for Databases, Tables and Functions to the user facing Catalog in

spark git commit: [SPARK-17740] Spark tests should mock / interpose HDFS to ensure that streams are closed

2016-09-30 Thread rxin
Repository: spark Updated Branches: refs/heads/master 15e9bbb49 -> 4bcd9b728 [SPARK-17740] Spark tests should mock / interpose HDFS to ensure that streams are closed ## What changes were proposed in this pull request? As a followup to SPARK-17666, ensure filesystem connections are not leaked

spark git commit: [MINOR][DOC] Add an up-to-date description for default serialization during shuffling

2016-09-30 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 a99ea4c9e -> 744aac8e6 [MINOR][DOC] Add an up-to-date description for default serialization during shuffling ## What changes were proposed in this pull request? This PR aims to make the doc up-to-date. The documentation is generally

spark git commit: [MINOR][DOC] Add an up-to-date description for default serialization during shuffling

2016-09-30 Thread rxin
Repository: spark Updated Branches: refs/heads/master aef506e39 -> 15e9bbb49 [MINOR][DOC] Add an up-to-date description for default serialization during shuffling ## What changes were proposed in this pull request? This PR aims to make the doc up-to-date. The documentation is generally corr

spark git commit: [SPARK-15353][CORE] Making peer selection for block replication pluggable

2016-09-30 Thread rxin
Repository: spark Updated Branches: refs/heads/master 81455a9cd -> a26afd521 [SPARK-15353][CORE] Making peer selection for block replication pluggable ## What changes were proposed in this pull request? This PR makes block replication strategies pluggable. It provides two trait that can be i

spark git commit: [SPARK-17703][SQL] Add unnamed version of addReferenceObj for minor objects.

2016-09-30 Thread rxin
Repository: spark Updated Branches: refs/heads/master f327e1686 -> 81455a9cd [SPARK-17703][SQL] Add unnamed version of addReferenceObj for minor objects. ## What changes were proposed in this pull request? There are many minor objects in references, which are extracted to the generated class

spark git commit: [SPARK-17717][SQL] Add exist/find methods to Catalog.

2016-09-29 Thread rxin
Repository: spark Updated Branches: refs/heads/master 2f7395670 -> 74ac1c438 [SPARK-17717][SQL] Add exist/find methods to Catalog. ## What changes were proposed in this pull request? The current user facing catalog does not implement methods for checking object existence or finding objects. Y

spark git commit: [SPARK-17641][SQL] Collect_list/Collect_set should not collect null values.

2016-09-28 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 d358298f1 -> 0a69477a1 [SPARK-17641][SQL] Collect_list/Collect_set should not collect null values. ## What changes were proposed in this pull request? We added native versions of `collect_set` and `collect_list` in Spark 2.0. These cur

spark git commit: [SPARK-17641][SQL] Collect_list/Collect_set should not collect null values.

2016-09-28 Thread rxin
Repository: spark Updated Branches: refs/heads/master 557d6e322 -> 7d0923202 [SPARK-17641][SQL] Collect_list/Collect_set should not collect null values. ## What changes were proposed in this pull request? We added native versions of `collect_set` and `collect_list` in Spark 2.0. These current

spark git commit: [SPARK-17713][SQL] Move row-datasource related tests out of JDBCSuite

2016-09-28 Thread rxin
Repository: spark Updated Branches: refs/heads/master a6cfa3f38 -> 557d6e322 [SPARK-17713][SQL] Move row-datasource related tests out of JDBCSuite ## What changes were proposed in this pull request? As a followup for https://github.com/apache/spark/pull/15273 we should move non-JDBC specific

spark git commit: [SPARK-17673][SQL] Incorrect exchange reuse with RowDataSourceScan (backport)

2016-09-28 Thread rxin
ied. rxin Author: Eric Liang Closes #15282 from ericl/spark-17673-2. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d358298f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d358298f Diff: http://git-wip-us.apache.org/re

spark git commit: [SPARK-17673][SQL] Incorrect exchange reuse with RowDataSourceScan

2016-09-28 Thread rxin
Repository: spark Updated Branches: refs/heads/master 46d1203bf -> a6cfa3f38 [SPARK-17673][SQL] Incorrect exchange reuse with RowDataSourceScan ## What changes were proposed in this pull request? It seems the equality check for reuse of `RowDataSourceScanExec` nodes doesn't respect the outpu

spark git commit: [SPARK-17666] Ensure that RecordReaders are closed by data source file scans (backport)

2016-09-28 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 2cd327ef5 -> 1b02f8820 [SPARK-17666] Ensure that RecordReaders are closed by data source file scans (backport) This is a branch-2.0 backport of #15245. ## What changes were proposed in this pull request? This patch addresses a potent

spark git commit: [SPARK-17666] Ensure that RecordReaders are closed by data source file scans

2016-09-27 Thread rxin
Repository: spark Updated Branches: refs/heads/master e7bce9e18 -> b03b4adf6 [SPARK-17666] Ensure that RecordReaders are closed by data source file scans ## What changes were proposed in this pull request? This patch addresses a potential cause of resource leaks in data source file scans. As

spark git commit: Fix two comments since Actor is not used anymore.

2016-09-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master 93c743f1a -> 6ee28423a Fix two comments since Actor is not used anymore. ## What changes were proposed in this pull request? Fix two comments since Actor is not used anymore. Author: Ding Fei Closes #15251 from danix800/comment-fixing.

spark git commit: [SPARK-17651][SPARKR] Set R package version number along with mvn

2016-09-23 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 452e468f2 -> b111a81f2 [SPARK-17651][SPARKR] Set R package version number along with mvn This PR sets the R package version while tagging releases. Note that since R doesn't accept `-SNAPSHOT` in version number field, we remove that wh

spark git commit: [SPARK-17651][SPARKR] Set R package version number along with mvn

2016-09-23 Thread rxin
Repository: spark Updated Branches: refs/heads/master 90a30f463 -> 7c382524a [SPARK-17651][SPARKR] Set R package version number along with mvn ## What changes were proposed in this pull request? This PR sets the R package version while tagging releases. Note that since R doesn't accept `-SNA

spark git commit: [SPARK-17599][SPARK-17569] Backport and to Spark 2.0 branch

2016-09-22 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 0a593db36 -> c2cb84165 [SPARK-17599][SPARK-17569] Backport and to Spark 2.0 branch ## What changes were proposed in this pull request? This Backports PR #15153 and PR #15122 to Spark 2.0 branch for Structured Streaming. It is structur

spark git commit: [SPARK-17635][SQL] Remove hardcode "agg_plan" in HashAggregateExec

2016-09-22 Thread rxin
Repository: spark Updated Branches: refs/heads/master a16619683 -> 79159a1e8 [SPARK-17635][SQL] Remove hardcode "agg_plan" in HashAggregateExec ## What changes were proposed in this pull request? "agg_plan" are hardcoded in HashAggregateExec, which have potential issue, so removing them. ##

spark git commit: Skip building R vignettes if Spark is not built

2016-09-22 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 b25a8e6e1 -> f14f47f07 Skip building R vignettes if Spark is not built ## What changes were proposed in this pull request? When we build the docs separately we don't have the JAR files from the Spark build in the same tree. As the Spa

spark git commit: Skip building R vignettes if Spark is not built

2016-09-22 Thread rxin
Repository: spark Updated Branches: refs/heads/master 17b72d31e -> 9f24a17c5 Skip building R vignettes if Spark is not built ## What changes were proposed in this pull request? When we build the docs separately we don't have the JAR files from the Spark build in the same tree. As the SparkR

spark git commit: Bump doc version for release 2.0.1.

2016-09-21 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 ec377e773 -> 053b20a79 Bump doc version for release 2.0.1. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/053b20a7 Tree: http://git-wip-us.apache.org/repos/asf/spa

spark git commit: [SPARK-17494][SQL] changePrecision() on compact decimal should respect rounding mode

2016-09-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 3497ebe51 -> 8bde03bf9 [SPARK-17494][SQL] changePrecision() on compact decimal should respect rounding mode ## What changes were proposed in this pull request? Floor()/Ceil() of decimal is implemented using changePrecision() by passing a

spark git commit: [SPARK-17494][SQL] changePrecision() on compact decimal should respect rounding mode

2016-09-21 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 966abd6af -> ec377e773 [SPARK-17494][SQL] changePrecision() on compact decimal should respect rounding mode ## What changes were proposed in this pull request? Floor()/Ceil() of decimal is implemented using changePrecision() by passin

spark git commit: [SPARK-17627] Mark Streaming Providers Experimental

2016-09-21 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 59e6ab11a -> 966abd6af [SPARK-17627] Mark Streaming Providers Experimental All of structured streaming is experimental in its first release. We missed the annotation on two of the APIs. Author: Michael Armbrust Closes #15188 from m

spark git commit: [SPARK-17627] Mark Streaming Providers Experimental

2016-09-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 6902edab7 -> 3497ebe51 [SPARK-17627] Mark Streaming Providers Experimental All of structured streaming is experimental in its first release. We missed the annotation on two of the APIs. Author: Michael Armbrust Closes #15188 from marmb

spark git commit: [MINOR][BUILD] Fix CheckStyle Error

2016-09-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master 976f3b122 -> 1ea49916a [MINOR][BUILD] Fix CheckStyle Error ## What changes were proposed in this pull request? This PR is to fix the code style errors before 2.0.1 release. ## How was this patch tested? Manual. Before: ``` ./dev/lint-java

spark git commit: [SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata

2016-09-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master 7e418e99c -> 976f3b122 [SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata ## What changes were proposed in this pull request? This PR modifies StreamExecution such that it discards metadata for batches that have already b

spark git commit: [SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata

2016-09-20 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 8d8e2332c -> 726f05716 [SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata ## What changes were proposed in this pull request? This PR modifies StreamExecution such that it discards metadata for batches that have alrea

spark git commit: [SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata

2016-09-19 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 7026eb87e -> 5456a1b4f [SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata ## What changes were proposed in this pull request? This PR modifies StreamExecution such that it discards metadata for batches that have alrea

spark git commit: [SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata

2016-09-19 Thread rxin
Repository: spark Updated Branches: refs/heads/master 26145a5af -> be9d57fc9 [SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata ## What changes were proposed in this pull request? This PR modifies StreamExecution such that it discards metadata for batches that have already b

spark git commit: [SPARK-17571][SQL] AssertOnQuery.condition should always return Boolean value

2016-09-18 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 151f808a1 -> 27ce39cf2 [SPARK-17571][SQL] AssertOnQuery.condition should always return Boolean value ## What changes were proposed in this pull request? AssertOnQuery has two apply constructor: one that accepts a closure that returns b

spark git commit: [SPARK-17571][SQL] AssertOnQuery.condition should always return Boolean value

2016-09-18 Thread rxin
Repository: spark Updated Branches: refs/heads/master 1dbb725db -> 8f0c35a4d [SPARK-17571][SQL] AssertOnQuery.condition should always return Boolean value ## What changes were proposed in this pull request? AssertOnQuery has two apply constructor: one that accepts a closure that returns boole

spark git commit: [SPARK-17561][DOCS] DataFrameWriter documentation formatting problems

2016-09-16 Thread rxin
Repository: spark Updated Branches: refs/heads/master dca771bec -> b9323fc93 [SPARK-17561][DOCS] DataFrameWriter documentation formatting problems ## What changes were proposed in this pull request? Fix ` / ` problems in SQL scaladoc. ## How was this patch tested? Scaladoc build and manual

spark git commit: [SPARK-17558] Bump Hadoop 2.7 version from 2.7.2 to 2.7.3

2016-09-16 Thread rxin
tly released and contained a number of bug fixes. ## How was this patch tested? The change should be covered by existing tests. Author: Reynold Xin Closes #15115 from rxin/SPARK-17558. (cherry picked from commit dca771bec6edb1cd8fc75861d364e0ba9dccf7c3) Signed-off-by: Reynold Xin Project: h

spark git commit: [SPARK-17558] Bump Hadoop 2.7 version from 2.7.2 to 2.7.3

2016-09-16 Thread rxin
tly released and contained a number of bug fixes. ## How was this patch tested? The change should be covered by existing tests. Author: Reynold Xin Closes #15115 from rxin/SPARK-17558. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/com

spark git commit: [SPARK-17301][SQL] Remove unused classTag field from AtomicType base class

2016-08-29 Thread rxin
Repository: spark Updated Branches: refs/heads/master 736a7911c -> 48b459ddd [SPARK-17301][SQL] Remove unused classTag field from AtomicType base class There's an unused `classTag` val in the AtomicType base class which is causing unnecessary slowness in deserialization because it needs to gr

spark git commit: [SPARK-17301][SQL] Remove unused classTag field from AtomicType base class

2016-08-29 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 976a43dbf -> 59032570f [SPARK-17301][SQL] Remove unused classTag field from AtomicType base class There's an unused `classTag` val in the AtomicType base class which is causing unnecessary slowness in deserialization because it needs t

spark git commit: [SPARK-17274][SQL] Move join optimizer rules into a separate file

2016-08-27 Thread rxin
gle file. ## How was this patch tested? This should be covered by existing tests. Author: Reynold Xin Closes #14846 from rxin/SPARK-17274. (cherry picked from commit 718b6bad2d698b76be6906d51da13626e9f3890e) Signed-off-by: Reynold Xin Project: http://git-wip-us.apache.org/repos/asf/spark/r

spark git commit: [SPARK-17274][SQL] Move join optimizer rules into a separate file

2016-08-27 Thread rxin
ile. ## How was this patch tested? This should be covered by existing tests. Author: Reynold Xin Closes #14846 from rxin/SPARK-17274. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/718b6bad Tree: http://git-wip-us.apache.

spark git commit: [SPARK-17273][SQL] Move expression optimizer rules into a separate file

2016-08-27 Thread rxin
ion rules into a single file. ## How was this patch tested? This should be covered by existing tests. Author: Reynold Xin Closes #14845 from rxin/SPARK-17273. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5aad4509 Tree: h

spark git commit: [SPARK-17272][SQL] Move subquery optimizer rules into its own file

2016-08-27 Thread rxin
gle file. ## How was this patch tested? This should be covered by existing tests. Author: Reynold Xin Closes #14844 from rxin/SPARK-17272. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0243b328 Tree: http://git-

spark git commit: [SPARK-17270][SQL] Move object optimization rules into its own file (branch-2.0)

2016-08-27 Thread rxin
vered by existing tests. Author: Reynold Xin Closes #14843 from rxin/SPARK-17270-branch-2.0. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f91614f3 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f91614f3 Diff: http:/

spark git commit: [SPARK-17269][SQL] Move finish analysis optimization stage into its own file

2016-08-26 Thread rxin
I'm submitting separate pull requests so we can more easily merge this in branch-2.0 to simplify optimizer backports. This should be covered by existing tests. Author: Reynold Xin Closes #14838 from rxin/SPARK-17269. (cherry picked from commit dcefac438788c51d84641bfbc505efe095731a39) Si

spark git commit: [SPARK-17269][SQL] Move finish analysis optimization stage into its own file

2016-08-26 Thread rxin
sis optimization stage rules into a single file. I'm submitting separate pull requests so we can more easily merge this in branch-2.0 to simplify optimizer backports. ## How was this patch tested? This should be covered by existing tests. Author: Reynold Xin Closes #14838 from rxin/SPA

spark git commit: [SPARK-17235][SQL] Support purging of old logs in MetadataLog

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 52feb3fbf -> dfdfc3092 [SPARK-17235][SQL] Support purging of old logs in MetadataLog ## What changes were proposed in this pull request? This patch adds a purge interface to MetadataLog, and an implementation in HDFSMetadataLog. The pu

spark git commit: [SPARK-17235][SQL] Support purging of old logs in MetadataLog

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master a11d10f18 -> f64a1ddd0 [SPARK-17235][SQL] Support purging of old logs in MetadataLog ## What changes were proposed in this pull request? This patch adds a purge interface to MetadataLog, and an implementation in HDFSMetadataLog. The purge

spark git commit: [SPARK-17246][SQL] Add BigDecimal literal

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 deb6a54cf -> 52feb3fbf [SPARK-17246][SQL] Add BigDecimal literal ## What changes were proposed in this pull request? This PR adds parser support for `BigDecimal` literals. If you append the suffix `BD` to a valid number then this will

spark git commit: [SPARK-17246][SQL] Add BigDecimal literal

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master 8e5475be3 -> a11d10f18 [SPARK-17246][SQL] Add BigDecimal literal ## What changes were proposed in this pull request? This PR adds parser support for `BigDecimal` literals. If you append the suffix `BD` to a valid number then this will be i

spark git commit: [SPARK-17242][DOCUMENT] Update links of external dstream projects

2016-08-25 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 73014a2aa -> 27ed6d5dc [SPARK-17242][DOCUMENT] Update links of external dstream projects ## What changes were proposed in this pull request? Updated links of external dstream projects. ## How was this patch tested? Just document chan

spark git commit: [SPARK-17242][DOCUMENT] Update links of external dstream projects

2016-08-25 Thread rxin
Repository: spark Updated Branches: refs/heads/master b964a172a -> 341e0e778 [SPARK-17242][DOCUMENT] Update links of external dstream projects ## What changes were proposed in this pull request? Updated links of external dstream projects. ## How was this patch tested? Just document changes.

spark git commit: [SPARK-17215][SQL] Method `SQLContext.parseDataType(dataTypeString: String)` could be removed.

2016-08-24 Thread rxin
Repository: spark Updated Branches: refs/heads/master 4d0706d61 -> 5f02d2e5b [SPARK-17215][SQL] Method `SQLContext.parseDataType(dataTypeString: String)` could be removed. ## What changes were proposed in this pull request? Method `SQLContext.parseDataType(dataTypeString: String)` could be r

spark git commit: [SPARK-17228][SQL] Not infer/propagate non-deterministic constraints

2016-08-24 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 3258f27a8 -> aa57083af [SPARK-17228][SQL] Not infer/propagate non-deterministic constraints ## What changes were proposed in this pull request? Given that filters based on non-deterministic constraints shouldn't be pushed down in the

spark git commit: [SPARK-17228][SQL] Not infer/propagate non-deterministic constraints

2016-08-24 Thread rxin
Repository: spark Updated Branches: refs/heads/master 3a60be4b1 -> ac27557eb [SPARK-17228][SQL] Not infer/propagate non-deterministic constraints ## What changes were proposed in this pull request? Given that filters based on non-deterministic constraints shouldn't be pushed down in the quer

spark git commit: [SPARK-16216][SQL][BRANCH-2.0] Backport Read/write dateFormat/timestampFormat options for CSV and JSON

2016-08-24 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 9f363a690 -> 3258f27a8 [SPARK-16216][SQL][BRANCH-2.0] Backport Read/write dateFormat/timestampFormat options for CSV and JSON ## What changes were proposed in this pull request? This PR backports https://github.com/apache/spark/pull/1

spark git commit: [SPARK-17186][SQL] remove catalog table type INDEX

2016-08-23 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 a6e6a047b -> df87f161c [SPARK-17186][SQL] remove catalog table type INDEX ## What changes were proposed in this pull request? Actually Spark SQL doesn't support index, the catalog table type `INDEX` is from Hive. However, most operati

spark git commit: [SPARK-17186][SQL] remove catalog table type INDEX

2016-08-23 Thread rxin
Repository: spark Updated Branches: refs/heads/master b9994ad05 -> 52fa45d62 [SPARK-17186][SQL] remove catalog table type INDEX ## What changes were proposed in this pull request? Actually Spark SQL doesn't support index, the catalog table type `INDEX` is from Hive. However, most operations

spark git commit: [MINOR][SQL] Remove implemented functions from comments of 'HiveSessionCatalog.scala'

2016-08-23 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 a772b4b5d -> a6e6a047b [MINOR][SQL] Remove implemented functions from comments of 'HiveSessionCatalog.scala' ## What changes were proposed in this pull request? This PR removes implemented functions from comments of `HiveSessionCatalo

spark git commit: [MINOR][SQL] Remove implemented functions from comments of 'HiveSessionCatalog.scala'

2016-08-23 Thread rxin
Repository: spark Updated Branches: refs/heads/master c1937dd19 -> b9994ad05 [MINOR][SQL] Remove implemented functions from comments of 'HiveSessionCatalog.scala' ## What changes were proposed in this pull request? This PR removes implemented functions from comments of `HiveSessionCatalog.sc

spark git commit: [SPARK-16862] Configurable buffer size in `UnsafeSorterSpillReader`

2016-08-23 Thread rxin
Repository: spark Updated Branches: refs/heads/master bf8ff833e -> c1937dd19 [SPARK-16862] Configurable buffer size in `UnsafeSorterSpillReader` ## What changes were proposed in this pull request? Jira: https://issues.apache.org/jira/browse/SPARK-16862 `BufferedInputStream` used in `UnsafeSo

spark git commit: [SPARK-16550][SPARK-17042][CORE] Certain classes fail to deserialize in block manager replication

2016-08-22 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 b65b041af -> ff2f87380 [SPARK-16550][SPARK-17042][CORE] Certain classes fail to deserialize in block manager replication ## What changes were proposed in this pull request? This is a straightforward clone of JoshRosen 's original patc

spark git commit: [SPARK-16550][SPARK-17042][CORE] Certain classes fail to deserialize in block manager replication

2016-08-22 Thread rxin
Repository: spark Updated Branches: refs/heads/master 71afeeea4 -> 8e223ea67 [SPARK-16550][SPARK-17042][CORE] Certain classes fail to deserialize in block manager replication ## What changes were proposed in this pull request? This is a straightforward clone of JoshRosen 's original patch. I

spark git commit: [SPARK-17162] Range does not support SQL generation

2016-08-22 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 6dcc1a3f0 -> 01a4d69f3 [SPARK-17162] Range does not support SQL generation ## What changes were proposed in this pull request? The range operator previously didn't support SQL generation, which made it not possible to use in views. #

spark git commit: [SPARK-17162] Range does not support SQL generation

2016-08-22 Thread rxin
Repository: spark Updated Branches: refs/heads/master 929cb8bee -> 84770b59f [SPARK-17162] Range does not support SQL generation ## What changes were proposed in this pull request? The range operator previously didn't support SQL generation, which made it not possible to use in views. ## Ho

spark git commit: [SPARK-17158][SQL] Change error message for out of range numeric literals

2016-08-19 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 efe832200 -> 379b12729 [SPARK-17158][SQL] Change error message for out of range numeric literals ## What changes were proposed in this pull request? Modifies error message for numeric literals to Numeric literal does not fit in range

spark git commit: [SPARK-17158][SQL] Change error message for out of range numeric literals

2016-08-19 Thread rxin
Repository: spark Updated Branches: refs/heads/master a117afa7c -> ba1737c21 [SPARK-17158][SQL] Change error message for out of range numeric literals ## What changes were proposed in this pull request? Modifies error message for numeric literals to Numeric literal does not fit in range [min

spark git commit: [SPARK-17149][SQL] array.sql for testing array related functions

2016-08-19 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 ae89c8e17 -> efe832200 [SPARK-17149][SQL] array.sql for testing array related functions ## What changes were proposed in this pull request? This patch creates array.sql in SQLQueryTestSuite for testing array related functions, includin

spark git commit: [SPARK-17149][SQL] array.sql for testing array related functions

2016-08-19 Thread rxin
Repository: spark Updated Branches: refs/heads/master acac7a508 -> a117afa7c [SPARK-17149][SQL] array.sql for testing array related functions ## What changes were proposed in this pull request? This patch creates array.sql in SQLQueryTestSuite for testing array related functions, including:

spark git commit: [SPARK-16686][SQL] Remove PushProjectThroughSample since it is handled by ColumnPruning

2016-08-19 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 d0707c6ba -> 3276ccfac [SPARK-16686][SQL] Remove PushProjectThroughSample since it is handled by ColumnPruning We push down `Project` through `Sample` in `Optimizer` by the rule `PushProjectThroughSample`. However, if the projected co

spark git commit: HOTFIX: compilation broken due to protected ctor.

2016-08-18 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 c180d637a -> 05b180faa HOTFIX: compilation broken due to protected ctor. (cherry picked from commit b482c09fa22c5762a355f95820e4ba3e2517fb77) Signed-off-by: Reynold Xin Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commi

spark git commit: HOTFIX: compilation broken due to protected ctor.

2016-08-18 Thread rxin
Repository: spark Updated Branches: refs/heads/master f5472dda5 -> b482c09fa HOTFIX: compilation broken due to protected ctor. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b482c09f Tree: http://git-wip-us.apache.org/rep

spark git commit: [SPARK-16947][SQL] Support type coercion and foldable expression for inline tables

2016-08-18 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 ea684b69c -> c180d637a [SPARK-16947][SQL] Support type coercion and foldable expression for inline tables This patch improves inline table support with the following: 1. Support type coercion. 2. Support using foldable expressions. Pr

spark git commit: [SPARK-17069] Expose spark.range() as table-valued function in SQL

2016-08-18 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 176af17a7 -> ea684b69c [SPARK-17069] Expose spark.range() as table-valued function in SQL This adds analyzer rules for resolving table-valued functions, and adds one builtin implementation for range(). The arguments for range() are the

spark git commit: [MINOR][DOC] Fix the descriptions for `properties` argument in the documenation for jdbc APIs

2016-08-16 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 3e0163bee -> 68a24d3e7 [MINOR][DOC] Fix the descriptions for `properties` argument in the documenation for jdbc APIs ## What changes were proposed in this pull request? This should be credited to mvervuurt. The main purpose of this PR

spark git commit: [MINOR][DOC] Fix the descriptions for `properties` argument in the documenation for jdbc APIs

2016-08-16 Thread rxin
Repository: spark Updated Branches: refs/heads/master f7c9ff57c -> 0f6aa8afa [MINOR][DOC] Fix the descriptions for `properties` argument in the documenation for jdbc APIs ## What changes were proposed in this pull request? This should be credited to mvervuurt. The main purpose of this PR is

spark git commit: [SPARK-17068][SQL] Make view-usage visible during analysis

2016-08-16 Thread rxin
Repository: spark Updated Branches: refs/heads/master 4a2c375be -> f7c9ff57c [SPARK-17068][SQL] Make view-usage visible during analysis ## What changes were proposed in this pull request? This PR adds a field to subquery alias in order to make the usage of views in a resolved `LogicalPlan` mo

spark git commit: [SPARK-17084][SQL] Rename ParserUtils.assert to validate

2016-08-16 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 6cb3eab7c -> 3e0163bee [SPARK-17084][SQL] Rename ParserUtils.assert to validate ## What changes were proposed in this pull request? This PR renames `ParserUtils.assert` to `ParserUtils.validate`. This is done because this method is use

spark git commit: [SPARK-17084][SQL] Rename ParserUtils.assert to validate

2016-08-16 Thread rxin
Repository: spark Updated Branches: refs/heads/master e28a8c589 -> 4a2c375be [SPARK-17084][SQL] Rename ParserUtils.assert to validate ## What changes were proposed in this pull request? This PR renames `ParserUtils.assert` to `ParserUtils.validate`. This is done because this method is used to

spark git commit: [SPARK-17089][DOCS] Remove api doc link for mapReduceTriplets operator

2016-08-16 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 022230c20 -> 6cb3eab7c [SPARK-17089][DOCS] Remove api doc link for mapReduceTriplets operator ## What changes were proposed in this pull request? Remove the api doc link for mapReduceTriplets operator because in latest api they are re

spark git commit: [SPARK-17089][DOCS] Remove api doc link for mapReduceTriplets operator

2016-08-16 Thread rxin
Repository: spark Updated Branches: refs/heads/master c34b546d6 -> e28a8c589 [SPARK-17089][DOCS] Remove api doc link for mapReduceTriplets operator ## What changes were proposed in this pull request? Remove the api doc link for mapReduceTriplets operator because in latest api they are remove

[2/2] spark git commit: [SPARK-16964][SQL] Remove private[sql] and private[spark] from sql.execution package [Backport]

2016-08-16 Thread rxin
[SPARK-16964][SQL] Remove private[sql] and private[spark] from sql.execution package [Backport] ## What changes were proposed in this pull request? This PR backports https://github.com/apache/spark/pull/14554 to branch-2.0. I have also changed the visibility of a few similar Hive classes. ## Ho

[1/2] spark git commit: [SPARK-16964][SQL] Remove private[sql] and private[spark] from sql.execution package [Backport]

2016-08-16 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 237ae54c9 -> 1c5697116 http://git-wip-us.apache.org/repos/asf/spark/blob/1c569711/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala

spark git commit: Revert "[SPARK-16964][SQL] Remove private[hive] from sql.hive.execution package"

2016-08-16 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 2e2c787bf -> 237ae54c9 Revert "[SPARK-16964][SQL] Remove private[hive] from sql.hive.execution package" This reverts commit 2e2c787bf588e129eaaadc792737fd9d2892939c. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: h

<    3   4   5   6   7   8   9   10   11   12   >