spark git commit: [SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None

2017-08-09 Thread lixiao
Repository: spark Updated Branches: refs/heads/master c06f3f5ac -> 84454d7d3 [SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None ## What changes were proposed in this pull request? Currently `df.na.replace("*", Map[String, String]("NULL" -> null))` will produce exception

spark git commit: [SPARK-21551][PYTHON] Increase timeout for PythonRDD.serveIterator

2017-08-09 Thread rxin
Repository: spark Updated Branches: refs/heads/master 0fb73253f -> c06f3f5ac [SPARK-21551][PYTHON] Increase timeout for PythonRDD.serveIterator ## What changes were proposed in this pull request? This modification increases the timeout for `serveIterator` (which is not dynamically configurab

spark git commit: [SPARK-21587][SS] Added filter pushdown through watermarks.

2017-08-09 Thread tdas
Repository: spark Updated Branches: refs/heads/master 2d799d080 -> 0fb73253f [SPARK-21587][SS] Added filter pushdown through watermarks. ## What changes were proposed in this pull request? Push filter predicates through EventTimeWatermark if they're deterministic and do not reference the wat

spark git commit: [SPARK-21504][SQL] Add spark version info into table metadata

2017-08-09 Thread lixiao
Repository: spark Updated Branches: refs/heads/master b78cf13bf -> 2d799d080 [SPARK-21504][SQL] Add spark version info into table metadata ## What changes were proposed in this pull request? This PR is to add the spark version info in the table metadata. When creating the table, this value is

spark git commit: [SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0)

2017-08-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master 83fe3b5e1 -> b78cf13bf [SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0) ## What changes were proposed in this pull request? This pr updated `lz4-java` to the latest (v1.4.0) and removed custom `LZ4BlockInputStream`. We currently

spark git commit: [SPARK-21665][CORE] Need to close resources after use

2017-08-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master 6426adffa -> 83fe3b5e1 [SPARK-21665][CORE] Need to close resources after use ## What changes were proposed in this pull request? Resources in Core - SparkSubmitArguments.scala, Spark-launcher - AbstractCommandBuilder.java, resource-manager

spark git commit: [SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite

2017-08-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master b35660dd0 -> 6426adffa [SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite Signed-off-by: 10087686 ## What changes were proposed in this pull request? After Unit

spark git commit: [SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite

2017-08-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 f6d56d2f1 -> 3ca55eaaf [SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite Signed-off-by: 10087686 ## What changes were proposed in this pull request? After U