git commit: automatically set master according to `spark.master` in `spark-defaults....

2014-07-31 Thread pwendell
Repository: spark Updated Branches: refs/heads/master 5a110da25 - 669e3f058 automatically set master according to `spark.master` in `spark-defaults automatically set master according to `spark.master` in `spark-defaults.conf` Author: CrazyJvm crazy...@gmail.com Closes #1644 from

git commit: [SPARK-2762] SparkILoop leaks memory in multi-repl configurations

2014-07-31 Thread matei
Repository: spark Updated Branches: refs/heads/master 669e3f058 - 92ca910eb [SPARK-2762] SparkILoop leaks memory in multi-repl configurations This pull request is a small refactor so that a partial function (hence a closure) is not created. Instead, a regular function is used. The behavior

git commit: [SPARK-2743][SQL] Resolve original attributes in ParquetTableScan

2014-07-31 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 92ca910eb - 3072b9602 [SPARK-2743][SQL] Resolve original attributes in ParquetTableScan Author: Michael Armbrust mich...@databricks.com Closes #1647 from marmbrus/parquetCase and squashes the following commits: a1799b7 [Michael Armbrust]

git commit: [SPARK-2397][SQL] Deprecate LocalHiveContext

2014-07-31 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 3072b9602 - 72cfb1398 [SPARK-2397][SQL] Deprecate LocalHiveContext LocalHiveContext is redundant with HiveContext. The only difference is it creates `./metastore` instead of `./metastore_db`. Author: Michael Armbrust

git commit: SPARK-2028: Expose mapPartitionsWithInputSplit in HadoopRDD

2014-07-31 Thread matei
Repository: spark Updated Branches: refs/heads/master 72cfb1398 - f19331235 SPARK-2028: Expose mapPartitionsWithInputSplit in HadoopRDD This allows users to gain access to the InputSplit which backs each partition. An alternative solution would have been to have a .withInputSplit() method

git commit: SPARK-2749 [BUILD] Part 2. Fix a follow-on scalastyle error

2014-07-31 Thread pwendell
Repository: spark Updated Branches: refs/heads/master f68105df5 - 4dbabb39a SPARK-2749 [BUILD] Part 2. Fix a follow-on scalastyle error The test compile error is fixed, but the build still fails because of one scalastyle error.

git commit: SPARK-2646. log4j initialization not quite compatible with log4j 2.x

2014-07-31 Thread pwendell
Repository: spark Updated Branches: refs/heads/master 4dbabb39a - e5749a134 SPARK-2646. log4j initialization not quite compatible with log4j 2.x The logging code that handles log4j initialization leads to an stack overflow error when used with log4j 2.x, which has just been released. This

git commit: [SPARK-2511][MLLIB] add HashingTF and IDF

2014-07-31 Thread meng
Repository: spark Updated Branches: refs/heads/master e5749a134 - dc0865bc7 [SPARK-2511][MLLIB] add HashingTF and IDF This is roughly the TF-IDF implementation used in the Databricks Cloud Demo: http://databricks.com/cloud/ . Both `HashingTF` and `IDF` are implemented as transformers,

git commit: [SPARK-2523] [SQL] Hadoop table scan bug fixing (fix failing Jenkins maven test)

2014-07-31 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master dc0865bc7 - 49b361298 [SPARK-2523] [SQL] Hadoop table scan bug fixing (fix failing Jenkins maven test) This PR tries to resolve the broken Jenkins maven test issue introduced by #1439. Now, we create a single query test to run both the

git commit: Improvements to merge_spark_pr.py

2014-07-31 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master 49b361298 - e02136214 Improvements to merge_spark_pr.py This commit fixes a couple of issues in the merge_spark_pr.py developer script: - Allow recovery from failed cherry-picks. - Fix detection of pull requests that have already been

git commit: Docs: monitoring, streaming programming guide

2014-07-31 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master e02136214 - cc820502f Docs: monitoring, streaming programming guide Fix several awkward wordings and grammatical issues in the following documents: * docs/monitoring.md * docs/streaming-programming-guide.md Author: kballou

git commit: [SPARK-2777][MLLIB] change ALS factors storage level to MEMORY_AND_DISK

2014-07-31 Thread meng
Repository: spark Updated Branches: refs/heads/master 9998efab9 - b19008320 [SPARK-2777][MLLIB] change ALS factors storage level to MEMORY_AND_DISK Now the factors are persisted in memory only. If they get kicked off by later jobs, we might have to start the computation from very beginning.

git commit: [SPARK-2702][Core] Upgrade Tachyon dependency to 0.5.0

2014-07-31 Thread pwendell
Repository: spark Updated Branches: refs/heads/master c4755403e - 2cdc3e5c6 [SPARK-2702][Core] Upgrade Tachyon dependency to 0.5.0 Author: Haoyuan Li haoy...@cs.berkeley.edu Closes #1651 from haoyuan/upgrade-tachyon and squashes the following commits: 6f3f98f [Haoyuan Li] upgrade tachyon to

git commit: SPARK-2632, SPARK-2576. Fixed by only importing what is necessary during class definition.

2014-07-31 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 2cdc3e5c6 - 149910111 SPARK-2632, SPARK-2576. Fixed by only importing what is necessary during class definition. Without this patch, it imports everything available in the scope. ```scala scala val a = 10l val a = 10l a: Long = 10