spark git commit: [SPARK-10001] [CORE] Interrupt tasks in repl with Ctrl+C

2016-04-21 Thread davies
Repository: spark Updated Branches: refs/heads/master 3405cc775 -> 80127935d [SPARK-10001] [CORE] Interrupt tasks in repl with Ctrl+C ## What changes were proposed in this pull request? Improve signal handling to allow interrupting running tasks from the REPL (with Ctrl+C). If no tasks are

spark git commit: [SPARK-14835][SQL] Remove MetastoreRelation dependency from SQLBuilder

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 145433f1a -> 3405cc775 [SPARK-14835][SQL] Remove MetastoreRelation dependency from SQLBuilder ## What changes were proposed in this pull request? This patch removes SQLBuilder's dependency on MetastoreRelation. We should be able to move

spark git commit: [SPARK-14369] [SQL] Locality support for FileScanRDD

2016-04-21 Thread davies
Repository: spark Updated Branches: refs/heads/master b29bc3f51 -> 145433f1a [SPARK-14369] [SQL] Locality support for FileScanRDD (This PR is a rebased version of PR #12153.) ## What changes were proposed in this pull request? This PR adds preliminary locality support for `FileFormat` data

spark git commit: [SPARK-14793] [SQL] Code generation for large complex type exceeds JVM size limit.

2016-04-21 Thread davies
Repository: spark Updated Branches: refs/heads/master df1953f0d -> f1fdb2382 [SPARK-14793] [SQL] Code generation for large complex type exceeds JVM size limit. ## What changes were proposed in this pull request? Code generation for complex type, `CreateArray`, `CreateMap`, `CreateStruct`,

spark git commit: [SPARK-14824][SQL] Rename HiveContext object to HiveUtils

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 0bf8df250 -> df1953f0d [SPARK-14824][SQL] Rename HiveContext object to HiveUtils ## What changes were proposed in this pull request? Just a rename so we can get rid of `HiveContext.scala`. Note that this will conflict with #12585. ##

spark git commit: [HOTFIX] Fix Java 7 compilation break

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master f181aee07 -> 0bf8df250 [HOTFIX] Fix Java 7 compilation break Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0bf8df25 Tree:

spark git commit: [SPARK-14821][SQL] Implement AnalyzeTable in sql/core and remove HiveSqlAstBuilder

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 4e726227a -> f181aee07 [SPARK-14821][SQL] Implement AnalyzeTable in sql/core and remove HiveSqlAstBuilder ## What changes were proposed in this pull request? This patch moves analyze table parsing into SparkSqlAstBuilder and removes

spark git commit: [SPARK-14479][ML] GLM supports output link prediction

2016-04-21 Thread meng
Repository: spark Updated Branches: refs/heads/master f25a3ea8d -> 4e726227a [SPARK-14479][ML] GLM supports output link prediction ## What changes were proposed in this pull request? GLM supports output link prediction. ## How was this patch tested? unit test. Author: Yanbo Liang

spark git commit: [SPARK-14734][ML][MLLIB] Added asML, fromML methods for all spark.mllib Vector, Matrix types

2016-04-21 Thread dbtsai
Repository: spark Updated Branches: refs/heads/master e2b5647ab -> f25a3ea8d [SPARK-14734][ML][MLLIB] Added asML, fromML methods for all spark.mllib Vector, Matrix types ## What changes were proposed in this pull request? For maintaining wrappers around spark.mllib algorithms in spark.ml,

spark git commit: [SPARK-14569][ML] Log instrumentation in KMeans

2016-04-21 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 411454475 -> 6d1e4c4a6 [SPARK-14569][ML] Log instrumentation in KMeans ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-14569 Log instrumentation in KMeans: - featuresCol - predictionCol -

spark git commit: [SPARK-14780] [R] Add `setLogLevel` to SparkR

2016-04-21 Thread davies
Repository: spark Updated Branches: refs/heads/master f82aa8248 -> 411454475 [SPARK-14780] [R] Add `setLogLevel` to SparkR ## What changes were proposed in this pull request? This PR aims to add `setLogLevel` function to SparkR shell. **Spark Shell** ```scala scala> sc.setLogLevel("ERROR")

spark git commit: [MINOR] Comment whitespace changes in #12553

2016-04-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master a2e8d4fdd -> ef6be7bed [MINOR] Comment whitespace changes in #12553 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ef6be7be Tree:

spark git commit: [SPARK-13643][SQL] Implement SparkSession

2016-04-21 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 8e1bb0456 -> a2e8d4fdd [SPARK-13643][SQL] Implement SparkSession ## What changes were proposed in this pull request? After removing most of `HiveContext` in 8fc267ab3322e46db81e725a5cb1adb5a71b2b4d we can now move existing functionality

spark git commit: [SPARK-14801][SQL] Move MetastoreRelation to its own file

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master e4904d870 -> 8e1bb0456 [SPARK-14801][SQL] Move MetastoreRelation to its own file ## What changes were proposed in this pull request? This class is currently in HiveMetastoreCatalog.scala, which is a large file that makes refactoring and

spark git commit: [SPARK-14699][CORE] Stop endpoints before closing the connections and don't stop client in Outbox

2016-04-21 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 3a21e8d5e -> e4904d870 [SPARK-14699][CORE] Stop endpoints before closing the connections and don't stop client in Outbox ## What changes were proposed in this pull request? In general, `onDisconnected` is for dealing with unexpected

spark git commit: [SPARK-14795][SQL] Remove the use of Hive's variable substitution

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 79008e6cf -> 3a21e8d5e [SPARK-14795][SQL] Remove the use of Hive's variable substitution ## What changes were proposed in this pull request? This patch builds on #12556 and completely removes the use of Hive's variable substitution. ##

spark git commit: [SPARK-14799][SQL] Remove MetastoreRelation dependency from AnalyzeTable - part 1

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master a70d40314 -> 79008e6cf [SPARK-14799][SQL] Remove MetastoreRelation dependency from AnalyzeTable - part 1 ## What changes were proposed in this pull request? This patch isolates AnalyzeTable's dependency on MetastoreRelation into a single

spark git commit: [SPARK-14783] Preserve full exception stacktrace in IsolatedClientLoader

2016-04-21 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master 4f369176b -> a70d40314 [SPARK-14783] Preserve full exception stacktrace in IsolatedClientLoader In IsolatedClientLoader, we have a`catch` block which throws an exception without wrapping the original exception, causing the full exception

spark git commit: [SPARK-4452] [CORE] Shuffle data structures can starve others on the same thread for memory

2016-04-21 Thread davies
Repository: spark Updated Branches: refs/heads/master 649335d6c -> 4f369176b [SPARK-4452] [CORE] Shuffle data structures can starve others on the same thread for memory ## What changes were proposed in this pull request? In #9241 It implemented a mechanism to call spill() on those SQL

spark git commit: [SPARK-14797][BUILD] Spark SQL POM should not hardcode spark-sketch_2.11 dep.

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 6fdd0e32a -> 649335d6c [SPARK-14797][BUILD] Spark SQL POM should not hardcode spark-sketch_2.11 dep. Spark SQL's POM hardcodes a dependency on `spark-sketch_2.11`, which causes Scala 2.10 builds to include the `_2.11` dependency. This is

spark git commit: [SPARK-13988][CORE] Make replaying event logs multi threaded in Histo…ry server to ensure a single large log does not block other logs from being rendered.

2016-04-21 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 4ac6e75cd -> 6fdd0e32a [SPARK-13988][CORE] Make replaying event logs multi threaded in Histo…ry server to ensure a single large log does not block other logs from being rendered. ## What changes were proposed in this pull request? The

spark git commit: [HOTFIX] Remove wrong DDL tests

2016-04-21 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master d53a51c1e -> 4ac6e75cd [HOTFIX] Remove wrong DDL tests ## What changes were proposed in this pull request? As we moved most parsing rules to `SparkSqlParser`, some tests expected to throw exception are not correct anymore. ## How was

spark git commit: [SPARK-14779][CORE] Corrected log message in Worker case KillExecutor

2016-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master ec2a27602 -> d53a51c1e [SPARK-14779][CORE] Corrected log message in Worker case KillExecutor In o.a.s.deploy.worker.Worker.scala, when receiving a KillExecutor message from an invalid Master, fixed typo by changing the log message to read

spark git commit: [SPARK-14787][SQL] Upgrade Joda-Time library from 2.9 to 2.9.3

2016-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.6 1cda10b4c -> c3ed9504d [SPARK-14787][SQL] Upgrade Joda-Time library from 2.9 to 2.9.3 ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-14787 The possible problems are described in the

spark git commit: [SPARK-14739][PYSPARK] Fix Vectors parser bugs

2016-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.6 17b138443 -> 1cda10b4c [SPARK-14739][PYSPARK] Fix Vectors parser bugs ## What changes were proposed in this pull request? The PySpark deserialization has a bug that shows while deserializing all zero sparse vectors. This fix filters

spark git commit: [SPARK-14739][PYSPARK] Fix Vectors parser bugs

2016-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master 8bd05c9db -> 2b8906c43 [SPARK-14739][PYSPARK] Fix Vectors parser bugs ## What changes were proposed in this pull request? The PySpark deserialization has a bug that shows while deserializing all zero sparse vectors. This fix filters out

spark git commit: [SPARK-8393][STREAMING] JavaStreamingContext#awaitTermination() throws non-declared InterruptedException

2016-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master cb51680d2 -> 8bd05c9db [SPARK-8393][STREAMING] JavaStreamingContext#awaitTermination() throws non-declared InterruptedException ## What changes were proposed in this pull request? `JavaStreamingContext.awaitTermination` methods should be

spark git commit: [SPARK-14753][CORE] remove internal flag in Accumulable

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 228128ce2 -> cb51680d2 [SPARK-14753][CORE] remove internal flag in Accumulable ## What changes were proposed in this pull request? the `Accumulable.internal` flag is only used to avoid registering internal accumulators for 2 certain

spark git commit: [SPARK-14794][SQL] Don't pass analyze command into Hive

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 3b9fd5173 -> 228128ce2 [SPARK-14794][SQL] Don't pass analyze command into Hive ## What changes were proposed in this pull request? We shouldn't pass analyze command to Hive because some of those would require running MapReduce jobs. For

spark git commit: [HOTFIX] Disable flaky tests

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 77d847ddb -> 3b9fd5173 [HOTFIX] Disable flaky tests Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3b9fd517 Tree:

spark git commit: [SPARK-14792][SQL] Move as many parsing rules as possible into SQL parser

2016-04-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master cfe472a34 -> 77d847ddb [SPARK-14792][SQL] Move as many parsing rules as possible into SQL parser ## What changes were proposed in this pull request? This patch moves as many parsing rules as possible into SQL parser. There are only three