spark git commit: [SPARK-8851] [YARN] In Client mode, make sure the client logs in and updates tokens

2015-07-17 Thread tgraves
Repository: spark Updated Branches: refs/heads/master ec8973d12 - c043a3e9d [SPARK-8851] [YARN] In Client mode, make sure the client logs in and updates tokens In client side, the flow is SparkSubmit - SparkContext - yarn/Client. Since the yarn client only gets a cloned config and the

spark git commit: [SPARK-9126] [MLLIB] do not assert on time taken by Thread.sleep()

2015-07-17 Thread meng
Repository: spark Updated Branches: refs/heads/master 322d286bb - 358e7bf65 [SPARK-9126] [MLLIB] do not assert on time taken by Thread.sleep() Measure lower and upper bounds for task time and use them for validation. This PR also implements `Stopwatch.toString`. This suite should finish in

spark git commit: [SPARK-9022] [SQL] Generated projections for UnsafeRow

2015-07-17 Thread rxin
Repository: spark Updated Branches: refs/heads/master 5a3c1ad08 - ec8973d12 [SPARK-9022] [SQL] Generated projections for UnsafeRow Added two projections: GenerateUnsafeProjection and FromUnsafeProjection, which could be used to convert UnsafeRow from/to GenericInternalRow. They will re-use

spark git commit: [SPARK-9102] [SQL] Improve project collapse with nondeterministic expressions

2015-07-17 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 111c05538 - 3f6d28a5c [SPARK-9102] [SQL] Improve project collapse with nondeterministic expressions Currently we will stop project collapse when the lower projection has nondeterministic expressions. However it's overkill sometimes, we

spark git commit: [SPARK-9093] [SPARKR] Fix single-quotes strings in SparkR

2015-07-17 Thread sarutak
Repository: spark Updated Branches: refs/heads/master 3f6d28a5c - 5a3c1ad08 [SPARK-9093] [SPARKR] Fix single-quotes strings in SparkR [[SPARK-9093] Fix single-quotes strings in SparkR - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-9093) This is the result of lintr at the

spark git commit: [MINOR] [ML] fix wrong annotation of RFormula.formula

2015-07-17 Thread meng
Repository: spark Updated Branches: refs/heads/master c043a3e9d - 441e072a2 [MINOR] [ML] fix wrong annotation of RFormula.formula fix wrong annotation of RFormula.formula Author: Yanbo Liang yblia...@gmail.com Closes #7470 from yanboliang/RFormula and squashes the following commits:

spark git commit: [SPARK-9130][SQL] throw exception when check equality between external and internal row

2015-07-17 Thread rxin
Repository: spark Updated Branches: refs/heads/master 441e072a2 - 59d24c226 [SPARK-9130][SQL] throw exception when check equality between external and internal row instead of return false, throw exception when check equality between external and internal row is better. Author: Wenchen Fan

spark git commit: [SPARK-8945][SQL] Add add and subtract expressions for IntervalType

2015-07-17 Thread rxin
Repository: spark Updated Branches: refs/heads/master 305e77cd8 - eba6a1af4 [SPARK-8945][SQL] Add add and subtract expressions for IntervalType JIRA: https://issues.apache.org/jira/browse/SPARK-8945 Add add and subtract expressions for IntervalType. Author: Liang-Chi Hsieh vii...@appier.com

spark git commit: [SPARK-9109] [GRAPHX] Keep the cached edge in the graph

2015-07-17 Thread ankurdave
Repository: spark Updated Branches: refs/heads/master eba6a1af4 - 587c315b2 [SPARK-9109] [GRAPHX] Keep the cached edge in the graph The change here is to keep the cached RDDs in the graph object so that when the graph.unpersist() is called these RDDs are correctly unpersisted. ```java

spark git commit: [SPARK-9109] [GRAPHX] Keep the cached edge in the graph

2015-07-17 Thread ankurdave
Repository: spark Updated Branches: refs/heads/branch-1.4 bb1401507 - f34f3d71f [SPARK-9109] [GRAPHX] Keep the cached edge in the graph The change here is to keep the cached RDDs in the graph object so that when the graph.unpersist() is called these RDDs are correctly unpersisted. ```java

spark git commit: [SPARK-9138] [MLLIB] fix Vectors.dense

2015-07-17 Thread meng
Repository: spark Updated Branches: refs/heads/master 587c315b2 - f9a82a884 [SPARK-9138] [MLLIB] fix Vectors.dense Vectors.dense() should accept numbers directly, like the one in Scala. We already use it in doctests, it worked by luck. cc mengxr jkbradley Author: Davies Liu

spark git commit: [SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math

2015-07-17 Thread rxin
Repository: spark Updated Branches: refs/heads/master 170723860 - 529a2c2d9 [SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math JIRA: https://issues.apache.org/jira/browse/SPARK-8280 https://issues.apache.org/jira/browse/SPARK-8281 Author: Yijie Shen henry.yijies...@gmail.com

spark git commit: [SPARK-9118] [ML] Implement IntArrayParam in mllib

2015-07-17 Thread meng
Repository: spark Updated Branches: refs/heads/master 34a889db8 - 101790820 [SPARK-9118] [ML] Implement IntArrayParam in mllib Implement IntArrayParam in mllib Author: Rekha Joshi rekhajo...@gmail.com Author: Joshi rekhajo...@gmail.com Closes #7481 from rekhajoshm/SPARK-9118 and squashes

spark git commit: [SPARK-8600] [ML] Naive Bayes API for spark.ml Pipelines

2015-07-17 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 806c579f4 - 997464287 [SPARK-8600] [ML] Naive Bayes API for spark.ml Pipelines Naive Bayes API for spark.ml Pipelines Author: Yanbo Liang yblia...@gmail.com Closes #7284 from yanboliang/spark-8600 and squashes the following commits:

spark git commit: [SPARK-9030] [STREAMING] Add Kinesis.createStream unit tests that actual sends data

2015-07-17 Thread tdas
Repository: spark Updated Branches: refs/heads/master bd903ee89 - b13ef7723 [SPARK-9030] [STREAMING] Add Kinesis.createStream unit tests that actual sends data Current Kinesis unit tests do not test createStream by sending data. This PR is to add such unit test. Note that this unit will not

spark git commit: [SPARK-8792] [ML] Add Python API for PCA transformer

2015-07-17 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 6da106969 - 830666f6f [SPARK-8792] [ML] Add Python API for PCA transformer Add Python API for PCA transformer Author: Yanbo Liang yblia...@gmail.com Closes #7190 from yanboliang/spark-8792 and squashes the following commits: 8f4ac31

spark git commit: [SPARK-9062] [ML] Change output type of Tokenizer to Array(String, true)

2015-07-17 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master f9a82a884 - 806c579f4 [SPARK-9062] [ML] Change output type of Tokenizer to Array(String, true) jira: https://issues.apache.org/jira/browse/SPARK-9062 Currently output type of Tokenizer is Array(String, false), which is not compatible

spark git commit: [SPARK-9113] [SQL] enable analysis check code for self join

2015-07-17 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 15fc2ffe5 - fd6b3101f [SPARK-9113] [SQL] enable analysis check code for self join The check was unreachable before, as `case operator: LogicalPlan` catches everything already. Author: Wenchen Fan cloud0...@outlook.com Closes #7449 from

spark git commit: [SPARK-9117] [SQL] fix BooleanSimplification in case-insensitive

2015-07-17 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master fd6b3101f - bd903ee89 [SPARK-9117] [SQL] fix BooleanSimplification in case-insensitive Author: Wenchen Fan cloud0...@outlook.com Closes #7452 from cloud-fan/boolean-simplify and squashes the following commits: 2a6e692 [Wenchen Fan] fix

spark git commit: [SPARK-9136] [SQL] fix several bugs in DateTimeUtils.stringToTimestamp

2015-07-17 Thread davies
Repository: spark Updated Branches: refs/heads/master 997464287 - 074085d67 [SPARK-9136] [SQL] fix several bugs in DateTimeUtils.stringToTimestamp a follow up of https://github.com/apache/spark/pull/7353 1. we should use `Calendar.HOUR_OF_DAY` instead of `Calendar.HOUR`(this is for AM, PM).

spark git commit: [SPARK-9142] [SQL] Removing unnecessary self types in Catalyst.

2015-07-17 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 42d8a012f - b2aa490bb [SPARK-9142] [SQL] Removing unnecessary self types in Catalyst. Just a small change to add Product type to the base expression/plan abstract classes, based on suggestions on #7434 and offline discussions. Author:

spark git commit: [SPARK-9080][SQL] add isNaN predicate expression

2015-07-17 Thread rxin
Repository: spark Updated Branches: refs/heads/master b2aa490bb - 15fc2ffe5 [SPARK-9080][SQL] add isNaN predicate expression JIRA: https://issues.apache.org/jira/browse/SPARK-9080 cc rxin Author: Yijie Shen henry.yijies...@gmail.com Closes #7464 from yijieshen/isNaN and squashes the

spark git commit: [SPARK-5681] [STREAMING] Move 'stopReceivers' to the event loop to resolve the race condition

2015-07-17 Thread tdas
Repository: spark Updated Branches: refs/heads/master 074085d67 - ad0954f6d [SPARK-5681] [STREAMING] Move 'stopReceivers' to the event loop to resolve the race condition This is an alternative way to fix `SPARK-5681`. It minimizes the changes. Closes #4467 Author: zsxwing zsxw...@gmail.com

spark git commit: [SPARK-9090] [ML] Fix definition of residual in LinearRegressionSummary, EnsembleTestHelper, and SquaredError

2015-07-17 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master ad0954f6d - 6da106969 [SPARK-9090] [ML] Fix definition of residual in LinearRegressionSummary, EnsembleTestHelper, and SquaredError Make the definition of residuals in Spark consistent with literature. We have been using `prediction -

spark git commit: [SPARK-8593] [CORE] Sort app attempts by start time.

2015-07-17 Thread srowen
Repository: spark Updated Branches: refs/heads/master 8b8be1f5d - 42d8a012f [SPARK-8593] [CORE] Sort app attempts by start time. This makes sure attempts are listed in the order they were executed, and that the app's state matches the state of the most current attempt. Author: Joshi

spark git commit: [SPARK-8593] [CORE] Sort app attempts by start time.

2015-07-17 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.4 f34f3d71f - 59838280f [SPARK-8593] [CORE] Sort app attempts by start time. This makes sure attempts are listed in the order they were executed, and that the app's state matches the state of the most current attempt. Author: Joshi