spark git commit: [HOT FIX #6125] Do not wait for all stages to start rendering

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 d518c0369 - 2d4a961f8 [HOT FIX #6125] Do not wait for all stages to start rendering zsxwing Author: Andrew Or and...@databricks.com Closes #6138 from andrewor14/dag-viz-clean-properly and squashes the following commits: 19d4e98

spark git commit: [SPARK-7612] [MLLIB] update NB training to use mllib's BLAS

2015-05-13 Thread meng
Repository: spark Updated Branches: refs/heads/master 3113da9c7 - d5f18de16 [SPARK-7612] [MLLIB] update NB training to use mllib's BLAS This is similar to the changes to k-means, which gives us better control on the performance. dbtsai Author: Xiangrui Meng m...@databricks.com Closes #6128

spark git commit: [SPARK-7612] [MLLIB] update NB training to use mllib's BLAS

2015-05-13 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.4 2d4a961f8 - 82f387fe2 [SPARK-7612] [MLLIB] update NB training to use mllib's BLAS This is similar to the changes to k-means, which gives us better control on the performance. dbtsai Author: Xiangrui Meng m...@databricks.com Closes

spark git commit: [SPARK-7545] [MLLIB] Added check in Bernoulli Naive Bayes to make sure that both training and predict features have values of 0 or 1

2015-05-13 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 5db18ba6e - 61e05fc58 [SPARK-7545] [MLLIB] Added check in Bernoulli Naive Bayes to make sure that both training and predict features have values of 0 or 1 Author: leahmcguire lmcgu...@salesforce.com Closes #6073 from

spark git commit: [MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 …

2015-05-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master e676fc0c6 - 3cd9ad240 [MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 … …arch. - zSeries 64-bit Java reports its architecture as s390x, so enhance the 64-bit check to accommodate that value. - SizeEstimator

spark git commit: [SPARK-7593] [ML] Python Api for ml.feature.Bucketizer

2015-05-13 Thread meng
Repository: spark Updated Branches: refs/heads/master 51030b8a9 - 5db18ba6e [SPARK-7593] [ML] Python Api for ml.feature.Bucketizer Added `ml.feature.Bucketizer` to PySpark. cc mengxr Author: Burak Yavuz brk...@gmail.com Closes #6124 from brkyvz/ml-bucket and squashes the following commits:

spark git commit: [SPARK-7593] [ML] Python Api for ml.feature.Bucketizer

2015-05-13 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.4 3a60bcb80 - 11911b0ae [SPARK-7593] [ML] Python Api for ml.feature.Bucketizer Added `ml.feature.Bucketizer` to PySpark. cc mengxr Author: Burak Yavuz brk...@gmail.com Closes #6124 from brkyvz/ml-bucket and squashes the following

spark git commit: [STREAMING] [MINOR] Keep streaming.UIUtils private

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 895d46a24 - e499a1e61 [STREAMING] [MINOR] Keep streaming.UIUtils private zsxwing Author: Andrew Or and...@databricks.com Closes #6134 from andrewor14/private-streaming-uiutils and squashes the following commits: 225df94 [Andrew Or]

spark git commit: [STREAMING] [MINOR] Keep streaming.UIUtils private

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master aa1837875 - bb6dec3b1 [STREAMING] [MINOR] Keep streaming.UIUtils private zsxwing Author: Andrew Or and...@databricks.com Closes #6134 from andrewor14/private-streaming-uiutils and squashes the following commits: 225df94 [Andrew Or]

spark git commit: [SPARK-7356] [STREAMING] Fix flakey tests in FlumePollingStreamSuite using SparkSink's batch CountDownLatch.

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 e499a1e61 - 6c0644ae2 [SPARK-7356] [STREAMING] Fix flakey tests in FlumePollingStreamSuite using SparkSink's batch CountDownLatch. This is meant to make the FlumePollingStreamSuite deterministic. Now we basically count the number of

[2/2] spark git commit: [SPARK-7081] Faster sort-based shuffle path using binary processing cache-aware sort

2015-05-13 Thread rxin
[SPARK-7081] Faster sort-based shuffle path using binary processing cache-aware sort This patch introduces a new shuffle manager that enhances the existing sort-based shuffle with a new cache-friendly sort algorithm that operates directly on binary data. The goals of this patch are to lower

[2/2] spark git commit: [SPARK-7081] Faster sort-based shuffle path using binary processing cache-aware sort

2015-05-13 Thread rxin
[SPARK-7081] Faster sort-based shuffle path using binary processing cache-aware sort This patch introduces a new shuffle manager that enhances the existing sort-based shuffle with a new cache-friendly sort algorithm that operates directly on binary data. The goals of this patch are to lower

[1/2] spark git commit: [SPARK-7081] Faster sort-based shuffle path using binary processing cache-aware sort

2015-05-13 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.4 6c0644ae2 - c53ebea9d http://git-wip-us.apache.org/repos/asf/spark/blob/c53ebea9/core/src/main/scala/org/apache/spark/shuffle/unsafe/UnsafeShuffleManager.scala -- diff

[1/2] spark git commit: [SPARK-7081] Faster sort-based shuffle path using binary processing cache-aware sort

2015-05-13 Thread rxin
Repository: spark Updated Branches: refs/heads/master 61d1e87c0 - 73bed408f http://git-wip-us.apache.org/repos/asf/spark/blob/73bed408/core/src/main/scala/org/apache/spark/shuffle/unsafe/UnsafeShuffleManager.scala -- diff

spark git commit: [SQL] Move some classes into packages that are more appropriate.

2015-05-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.4 d5c52d9ac - acd872bbd [SQL] Move some classes into packages that are more appropriate. JavaTypeInference into catalyst types.DateUtils into catalyst CacheManager into execution DefaultParserDialect into catalyst Author: Reynold Xin

spark git commit: [SQL] Move some classes into packages that are more appropriate.

2015-05-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 59250fe51 - e683182c3 [SQL] Move some classes into packages that are more appropriate. JavaTypeInference into catalyst types.DateUtils into catalyst CacheManager into execution DefaultParserDialect into catalyst Author: Reynold Xin

spark git commit: [SPARK-7382] [MLLIB] Feature Parity in PySpark for ml.classification

2015-05-13 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.4 d9fb905be - 51230f2a9 [SPARK-7382] [MLLIB] Feature Parity in PySpark for ml.classification The missing pieces in ml.classification for Python! cc mengxr Author: Burak Yavuz brk...@gmail.com Closes #6106 from brkyvz/ml-class and

spark git commit: [SPARK-7382] [MLLIB] Feature Parity in PySpark for ml.classification

2015-05-13 Thread meng
Repository: spark Updated Branches: refs/heads/master 61e05fc58 - df2fb1305 [SPARK-7382] [MLLIB] Feature Parity in PySpark for ml.classification The missing pieces in ml.classification for Python! cc mengxr Author: Burak Yavuz brk...@gmail.com Closes #6106 from brkyvz/ml-class and squashes

spark git commit: [SPARK-7303] [SQL] push down project if possible when the child is sort

2015-05-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.4 51230f2a9 - d5c52d9ac [SPARK-7303] [SQL] push down project if possible when the child is sort Optimize the case of `project(_, sort)` , a example is: `select key from (select * from testData order by key) t` before this PR: ``` ==

spark git commit: [SPARK-7303] [SQL] push down project if possible when the child is sort

2015-05-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master df2fb1305 - 59250fe51 [SPARK-7303] [SQL] push down project if possible when the child is sort Optimize the case of `project(_, sort)` , a example is: `select key from (select * from testData order by key) t` before this PR: ``` == Parsed

spark git commit: [SPARK-7399] Spark compilation error for scala 2.11

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master f6e18388d - f88ac7015 [SPARK-7399] Spark compilation error for scala 2.11 Subsequent fix following #5966. I tried this out locally. Author: Andrew Or and...@databricks.com Closes #6129 from andrewor14/211-compilation and squashes the

spark git commit: [SPARK-7608] Clean up old state in RDDOperationGraphListener

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 acd872bbd - ec342308a [SPARK-7608] Clean up old state in RDDOperationGraphListener This is necessary for streaming and long-running Spark applications. zsxwing tdas Author: Andrew Or and...@databricks.com Closes #6125 from

[1/3] spark git commit: [SPARK-7464] DAG visualization: highlight the same RDDs on hover

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 e6b8cef51 - 4b4f10bc9 http://git-wip-us.apache.org/repos/asf/spark/blob/4b4f10bc/core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.css -- diff --git

[2/3] spark git commit: [SPARK-7464] DAG visualization: highlight the same RDDs on hover

2015-05-13 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/44403414/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js

[2/3] spark git commit: [SPARK-7464] DAG visualization: highlight the same RDDs on hover

2015-05-13 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/4b4f10bc/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js

spark git commit: [SPARK-7502] DAG visualization: gracefully handle removed stages

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 44403414d - aa1837875 [SPARK-7502] DAG visualization: gracefully handle removed stages Old stages are removed without much feedback to the user. This happens very often in streaming. See screenshots below for more detail. zsxwing

spark git commit: [SPARK-7399] Spark compilation error for scala 2.11

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 ec342308a - e6b8cef51 [SPARK-7399] Spark compilation error for scala 2.11 Subsequent fix following #5966. I tried this out locally. Author: Andrew Or and...@databricks.com Closes #6129 from andrewor14/211-compilation and squashes the

spark git commit: [MINOR] [SQL] Removes debugging println

2015-05-13 Thread lian
Repository: spark Updated Branches: refs/heads/branch-1.4 9ca28d9de - d78f0e1b4 [MINOR] [SQL] Removes debugging println Author: Cheng Lian l...@databricks.com Closes #6123 from liancheng/remove-println and squashes the following commits: 03356b6 [Cheng Lian] Removes debugging println

spark git commit: [SPARK-6734] [SQL] Add UDTF.close support in Generate

2015-05-13 Thread lian
Repository: spark Updated Branches: refs/heads/branch-1.4 d78f0e1b4 - 42cf4a2a5 [SPARK-6734] [SQL] Add UDTF.close support in Generate Some third-party UDTF extensions generate additional rows in the GenericUDTF.close() method, which is supported / documented by Hive.

spark git commit: [SPARK-6734] [SQL] Add UDTF.close support in Generate

2015-05-13 Thread lian
Repository: spark Updated Branches: refs/heads/master aa6ba3f21 - 0da254fb2 [SPARK-6734] [SQL] Add UDTF.close support in Generate Some third-party UDTF extensions generate additional rows in the GenericUDTF.close() method, which is supported / documented by Hive.

spark git commit: [HOTFIX] Bug in merge script

2015-05-13 Thread pwendell
Repository: spark Updated Branches: refs/heads/master bce00dac4 - 32e27df41 [HOTFIX] Bug in merge script Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/32e27df4 Tree:

spark git commit: [HOTFIX] Use 'new Job' in fsBasedParquet.scala

2015-05-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.4 aec83949a - d518c0369 [HOTFIX] Use 'new Job' in fsBasedParquet.scala Same issue as #6095 cc liancheng Author: zsxwing zsxw...@gmail.com Closes #6136 from zsxwing/hotfix and squashes the following commits: 4beea54 [zsxwing] Use 'new

[1/2] spark git commit: [SPARK-7567] [SQL] Migrating Parquet data source to FSBasedRelation

2015-05-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master bec938f77 - 7ff16e8ab http://git-wip-us.apache.org/repos/asf/spark/blob/7ff16e8a/sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetFilterSuite.scala -- diff --git