spark git commit: [SPARK-12327] Disable commented code lintr temporarily

2015-12-14 Thread shivaram 
Repository: spark
Updated Branches:
  refs/heads/master b51a4cdff -> fb3778de6


[SPARK-12327] Disable commented code lintr temporarily

cc yhuai felixcheung shaneknapp

Author: Shivaram Venkataraman 

Closes #10300 from shivaram/comment-lintr-disable.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fb3778de
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fb3778de
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fb3778de

Branch: refs/heads/master
Commit: fb3778de685881df66bf0222b520f94dca99e8c8
Parents: b51a4cd
Author: Shivaram Venkataraman 
Authored: Mon Dec 14 16:13:55 2015 -0800
Committer: Shivaram Venkataraman 
Committed: Mon Dec 14 16:13:55 2015 -0800

--
 R/pkg/.lintr | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/fb3778de/R/pkg/.lintr
--
diff --git a/R/pkg/.lintr b/R/pkg/.lintr
index 038236f..39c8726 100644
--- a/R/pkg/.lintr
+++ b/R/pkg/.lintr
@@ -1,2 +1,2 @@
-linters: with_defaults(line_length_linter(100), camel_case_linter = NULL, 
open_curly_linter(allow_single_line = TRUE), 
closed_curly_linter(allow_single_line = TRUE))
+linters: with_defaults(line_length_linter(100), camel_case_linter = NULL, 
open_curly_linter(allow_single_line = TRUE), 
closed_curly_linter(allow_single_line = TRUE), commented_code_linter = NULL)
 exclusions: list("inst/profile/general.R" = 1, "inst/profile/shell.R")


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-12327] Disable commented code lintr temporarily

2015-12-14 Thread shivaram 
Repository: spark
Updated Branches:
  refs/heads/branch-1.6 c0f0f6cb0 -> 352a0c80f


[SPARK-12327] Disable commented code lintr temporarily

cc yhuai felixcheung shaneknapp

Author: Shivaram Venkataraman 

Closes #10300 from shivaram/comment-lintr-disable.

(cherry picked from commit fb3778de685881df66bf0222b520f94dca99e8c8)
Signed-off-by: Shivaram Venkataraman 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/352a0c80
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/352a0c80
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/352a0c80

Branch: refs/heads/branch-1.6
Commit: 352a0c80f4833a97916a75388ef290067c2dbede
Parents: c0f0f6c
Author: Shivaram Venkataraman 
Authored: Mon Dec 14 16:13:55 2015 -0800
Committer: Shivaram Venkataraman 
Committed: Mon Dec 14 16:14:03 2015 -0800

--
 R/pkg/.lintr | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/352a0c80/R/pkg/.lintr
--
diff --git a/R/pkg/.lintr b/R/pkg/.lintr
index 038236f..39c8726 100644
--- a/R/pkg/.lintr
+++ b/R/pkg/.lintr
@@ -1,2 +1,2 @@
-linters: with_defaults(line_length_linter(100), camel_case_linter = NULL, 
open_curly_linter(allow_single_line = TRUE), 
closed_curly_linter(allow_single_line = TRUE))
+linters: with_defaults(line_length_linter(100), camel_case_linter = NULL, 
open_curly_linter(allow_single_line = TRUE), 
closed_curly_linter(allow_single_line = TRUE), commented_code_linter = NULL)
 exclusions: list("inst/profile/general.R" = 1, "inst/profile/shell.R")


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-12188][SQL][FOLLOW-UP] Code refactoring and comment correction in Dataset APIs

2015-12-14 Thread marmbrus 
Repository: spark
Updated Branches:
  refs/heads/master 9ea1a8efc -> d13ff82cb


[SPARK-12188][SQL][FOLLOW-UP] Code refactoring and comment correction in 
Dataset APIs

marmbrus This PR is to address your comment. Thanks for your review!

Author: gatorsmile 

Closes #10214 from gatorsmile/followup12188.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d13ff82c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d13ff82c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d13ff82c

Branch: refs/heads/master
Commit: d13ff82cba10a1ce9889c4b416f1e43c717e3f10
Parents: 9ea1a8e
Author: gatorsmile 
Authored: Mon Dec 14 18:33:45 2015 -0800
Committer: Michael Armbrust 
Committed: Mon Dec 14 18:33:45 2015 -0800

--
 sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/d13ff82c/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
index 3bd18a1..dc69822 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
@@ -79,7 +79,7 @@ class Dataset[T] private[sql](
 
   /**
* The encoder where the expressions used to construct an object from an 
input row have been
-   * bound to the ordinals of the given schema.
+   * bound to the ordinals of this [[Dataset]]'s output schema.
*/
   private[sql] val boundTEncoder = resolvedTEncoder.bind(logicalPlan.output)
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-12274][SQL] WrapOption should not have type constraint for child

2015-12-14 Thread marmbrus 
Repository: spark
Updated Branches:
  refs/heads/master fb3778de6 -> 9ea1a8efc


[SPARK-12274][SQL] WrapOption should not have type constraint for child

I think it was a mistake, and we have not catched it so far until 
https://github.com/apache/spark/pull/10260 which begin to check if the 
`fromRowExpression` is resolved.

Author: Wenchen Fan 

Closes #10263 from cloud-fan/encoder.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9ea1a8ef
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9ea1a8ef
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9ea1a8ef

Branch: refs/heads/master
Commit: 9ea1a8efca7869618c192546ef4e6d94c17689b5
Parents: fb3778d
Author: Wenchen Fan 
Authored: Mon Dec 14 16:48:11 2015 -0800
Committer: Michael Armbrust 
Committed: Mon Dec 14 16:48:11 2015 -0800

--
 .../org/apache/spark/sql/catalyst/expressions/objects.scala | 5 +
 1 file changed, 1 insertion(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/9ea1a8ef/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala
--
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala
index b2facfd..96bc4fe 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala
@@ -296,15 +296,12 @@ case class UnwrapOption(
  * (in the case of reference types) equality with null.
  * @param child The expression to evaluate and wrap.
  */
-case class WrapOption(child: Expression)
-  extends UnaryExpression with ExpectsInputTypes {
+case class WrapOption(child: Expression) extends UnaryExpression {
 
   override def dataType: DataType = ObjectType(classOf[Option[_]])
 
   override def nullable: Boolean = true
 
-  override def inputTypes: Seq[AbstractDataType] = ObjectType :: Nil
-
   override def eval(input: InternalRow): Any =
 throw new UnsupportedOperationException("Only code-generated evaluation is 
supported")
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-12288] [SQL] Support UnsafeRow in Coalesce/Except/Intersect.

2015-12-14 Thread davies 
Repository: spark
Updated Branches:
  refs/heads/master d13ff82cb -> 606f99b94


[SPARK-12288] [SQL] Support UnsafeRow in Coalesce/Except/Intersect.

Support UnsafeRow for the Coalesce/Except/Intersect.

Could you review if my code changes are ok? davies Thank you!

Author: gatorsmile 

Closes #10285 from gatorsmile/unsafeSupportCIE.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/606f99b9
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/606f99b9
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/606f99b9

Branch: refs/heads/master
Commit: 606f99b942a25dc3d86138b00026462ffe58cded
Parents: d13ff82
Author: gatorsmile 
Authored: Mon Dec 14 19:42:16 2015 -0800
Committer: Davies Liu 
Committed: Mon Dec 14 19:42:16 2015 -0800

--
 .../spark/sql/execution/basicOperators.scala| 12 ++-
 .../execution/RowFormatConvertersSuite.scala| 35 
 2 files changed, 46 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/606f99b9/sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala
--
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala
index a42aea0..b3e4688 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala
@@ -137,7 +137,7 @@ case class Union(children: Seq[SparkPlan]) extends 
SparkPlan {
   }
 }
   }
-  override def outputsUnsafeRows: Boolean = 
children.forall(_.outputsUnsafeRows)
+  override def outputsUnsafeRows: Boolean = 
children.exists(_.outputsUnsafeRows)
   override def canProcessUnsafeRows: Boolean = true
   override def canProcessSafeRows: Boolean = true
   protected override def doExecute(): RDD[InternalRow] =
@@ -250,7 +250,9 @@ case class Coalesce(numPartitions: Int, child: SparkPlan) 
extends UnaryNode {
 child.execute().coalesce(numPartitions, shuffle = false)
   }
 
+  override def outputsUnsafeRows: Boolean = child.outputsUnsafeRows
   override def canProcessUnsafeRows: Boolean = true
+  override def canProcessSafeRows: Boolean = true
 }
 
 /**
@@ -263,6 +265,10 @@ case class Except(left: SparkPlan, right: SparkPlan) 
extends BinaryNode {
   protected override def doExecute(): RDD[InternalRow] = {
 left.execute().map(_.copy()).subtract(right.execute().map(_.copy()))
   }
+
+  override def outputsUnsafeRows: Boolean = 
children.exists(_.outputsUnsafeRows)
+  override def canProcessUnsafeRows: Boolean = true
+  override def canProcessSafeRows: Boolean = true
 }
 
 /**
@@ -275,6 +281,10 @@ case class Intersect(left: SparkPlan, right: SparkPlan) 
extends BinaryNode {
   protected override def doExecute(): RDD[InternalRow] = {
 left.execute().map(_.copy()).intersection(right.execute().map(_.copy()))
   }
+
+  override def outputsUnsafeRows: Boolean = 
children.exists(_.outputsUnsafeRows)
+  override def canProcessUnsafeRows: Boolean = true
+  override def canProcessSafeRows: Boolean = true
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/spark/blob/606f99b9/sql/core/src/test/scala/org/apache/spark/sql/execution/RowFormatConvertersSuite.scala
--
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/RowFormatConvertersSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/RowFormatConvertersSuite.scala
index 13d68a1..2328899 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/RowFormatConvertersSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/RowFormatConvertersSuite.scala
@@ -58,6 +58,41 @@ class RowFormatConvertersSuite extends SparkPlanTest with 
SharedSQLContext {
 assert(!preparedPlan.outputsUnsafeRows)
   }
 
+  test("coalesce can process unsafe rows") {
+val plan = Coalesce(1, outputsUnsafe)
+val preparedPlan = sqlContext.prepareForExecution.execute(plan)
+assert(getConverters(preparedPlan).size === 1)
+assert(preparedPlan.outputsUnsafeRows)
+  }
+
+  test("except can process unsafe rows") {
+val plan = Except(outputsUnsafe, outputsUnsafe)
+val preparedPlan = sqlContext.prepareForExecution.execute(plan)
+assert(getConverters(preparedPlan).size === 2)
+assert(preparedPlan.outputsUnsafeRows)
+  }
+
+  test("except requires all of its input rows' formats to agree") {
+val plan = Except(outputsSafe, outputsUnsafe)
+assert(plan.canProcessSafeRows && plan.canProcessUnsafeRows)
+val preparedPlan =

spark git commit: [MINOR][DOC] Fix broken word2vec link

2015-12-14 Thread srowen 
Repository: spark
Updated Branches:
  refs/heads/master ed87f6d3b -> e25f1fe42


[MINOR][DOC] Fix broken word2vec link

Follow-up of [SPARK-12199](https://issues.apache.org/jira/browse/SPARK-12199) 
and #10193 where a broken link has been left as is.

Author: BenFradet 

Closes #10282 from BenFradet/SPARK-12199.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e25f1fe4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e25f1fe4
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e25f1fe4

Branch: refs/heads/master
Commit: e25f1fe42747be71c6b6e6357ca214f9544e3a46
Parents: ed87f6d
Author: BenFradet 
Authored: Mon Dec 14 13:50:30 2015 +
Committer: Sean Owen 
Committed: Mon Dec 14 13:50:30 2015 +

--
 docs/ml-features.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/e25f1fe4/docs/ml-features.md
--
diff --git a/docs/ml-features.md b/docs/ml-features.md
index 158f3f2..677e4bf 100644
--- a/docs/ml-features.md
+++ b/docs/ml-features.md
@@ -63,7 +63,7 @@ the [IDF Python 
docs](api/python/pyspark.ml.html#pyspark.ml.feature.IDF) for mor
 `Word2VecModel`. The model maps each word to a unique fixed-size vector. The 
`Word2VecModel`
 transforms each document into a vector using the average of all words in the 
document; this vector
 can then be used for as features for prediction, document similarity 
calculations, etc.
-Please refer to the [MLlib user guide on 
Word2Vec](mllib-feature-extraction.html#word2Vec) for more
+Please refer to the [MLlib user guide on 
Word2Vec](mllib-feature-extraction.html#word2vec) for more
 details.
 
 In the following code segment, we start with a set of documents, each of which 
is represented as a sequence of words. For each document, we transform it into 
a feature vector. This feature vector could then be passed to a learning 
algorithm.


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [MINOR][DOC] Fix broken word2vec link

2015-12-14 Thread srowen 
Repository: spark
Updated Branches:
  refs/heads/branch-1.6 94ce5025f -> c0f0f6cb0


[MINOR][DOC] Fix broken word2vec link

Follow-up of [SPARK-12199](https://issues.apache.org/jira/browse/SPARK-12199) 
and #10193 where a broken link has been left as is.

Author: BenFradet 

Closes #10282 from BenFradet/SPARK-12199.

(cherry picked from commit e25f1fe42747be71c6b6e6357ca214f9544e3a46)
Signed-off-by: Sean Owen 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c0f0f6cb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c0f0f6cb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c0f0f6cb

Branch: refs/heads/branch-1.6
Commit: c0f0f6cb0fef6e939744b60fdd4911c718f8fac5
Parents: 94ce502
Author: BenFradet 
Authored: Mon Dec 14 13:50:30 2015 +
Committer: Sean Owen 
Committed: Mon Dec 14 13:50:40 2015 +

--
 docs/ml-features.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/c0f0f6cb/docs/ml-features.md
--
diff --git a/docs/ml-features.md b/docs/ml-features.md
index 158f3f2..677e4bf 100644
--- a/docs/ml-features.md
+++ b/docs/ml-features.md
@@ -63,7 +63,7 @@ the [IDF Python 
docs](api/python/pyspark.ml.html#pyspark.ml.feature.IDF) for mor
 `Word2VecModel`. The model maps each word to a unique fixed-size vector. The 
`Word2VecModel`
 transforms each document into a vector using the average of all words in the 
document; this vector
 can then be used for as features for prediction, document similarity 
calculations, etc.
-Please refer to the [MLlib user guide on 
Word2Vec](mllib-feature-extraction.html#word2Vec) for more
+Please refer to the [MLlib user guide on 
Word2Vec](mllib-feature-extraction.html#word2vec) for more
 details.
 
 In the following code segment, we start with a set of documents, each of which 
is represented as a sequence of words. For each document, we transform it into 
a feature vector. This feature vector could then be passed to a learning 
algorithm.


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-12016] [MLLIB] [PYSPARK] Wrap Word2VecModel when loading it in pyspark

2015-12-14 Thread davies 
Repository: spark
Updated Branches:
  refs/heads/master e25f1fe42 -> b51a4cdff


[SPARK-12016] [MLLIB] [PYSPARK] Wrap Word2VecModel when loading it in pyspark

JIRA: https://issues.apache.org/jira/browse/SPARK-12016

We should not directly use Word2VecModel in pyspark. We need to wrap it in a 
Word2VecModelWrapper when loading it in pyspark.

Author: Liang-Chi Hsieh 

Closes #10100 from viirya/fix-load-py-wordvecmodel.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b51a4cdf
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b51a4cdf
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b51a4cdf

Branch: refs/heads/master
Commit: b51a4cdff3a7e640a8a66f7a9c17021f3056fd34
Parents: e25f1fe
Author: Liang-Chi Hsieh 
Authored: Mon Dec 14 09:59:42 2015 -0800
Committer: Davies Liu 
Committed: Mon Dec 14 09:59:42 2015 -0800

--
 .../spark/mllib/api/python/PythonMLLibAPI.scala | 33 ---
 .../mllib/api/python/Word2VecModelWrapper.scala | 62 
 python/pyspark/mllib/feature.py |  6 +-
 3 files changed, 67 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/b51a4cdf/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
--
diff --git 
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala 
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
index 8d546e3..29160a1 100644
--- 
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
+++ 
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
@@ -680,39 +680,6 @@ private[python] class PythonMLLibAPI extends Serializable {
 }
   }
 
-  private[python] class Word2VecModelWrapper(model: Word2VecModel) {
-def transform(word: String): Vector = {
-  model.transform(word)
-}
-
-/**
- * Transforms an RDD of words to its vector representation
- * @param rdd an RDD of words
- * @return an RDD of vector representations of words
- */
-def transform(rdd: JavaRDD[String]): JavaRDD[Vector] = {
-  rdd.rdd.map(model.transform)
-}
-
-def findSynonyms(word: String, num: Int): JList[Object] = {
-  val vec = transform(word)
-  findSynonyms(vec, num)
-}
-
-def findSynonyms(vector: Vector, num: Int): JList[Object] = {
-  val result = model.findSynonyms(vector, num)
-  val similarity = Vectors.dense(result.map(_._2))
-  val words = result.map(_._1)
-  List(words, similarity).map(_.asInstanceOf[Object]).asJava
-}
-
-def getVectors: JMap[String, JList[Float]] = {
-  model.getVectors.map({case (k, v) => (k, v.toList.asJava)}).asJava
-}
-
-def save(sc: SparkContext, path: String): Unit = model.save(sc, path)
-  }
-
   /**
* Java stub for Python mllib DecisionTree.train().
* This stub returns a handle to the Java object instead of the content of 
the Java object.

http://git-wip-us.apache.org/repos/asf/spark/blob/b51a4cdf/mllib/src/main/scala/org/apache/spark/mllib/api/python/Word2VecModelWrapper.scala
--
diff --git 
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/Word2VecModelWrapper.scala
 
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/Word2VecModelWrapper.scala
new file mode 100644
index 000..0f55980
--- /dev/null
+++ 
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/Word2VecModelWrapper.scala
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.api.python
+
+import java.util.{ArrayList => JArrayList, List => JList, Map => JMap}
+import scala.collection.JavaConverters._
+
+import org.apache.spark.SparkContext
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.mllib.feature.Word2VecModel
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+
+/**
+

spark git commit: [SPARK-12275] [SQL] No plan for BroadcastHint in some condition - 1.5 backport

2015-12-14 Thread andrewor14 
Repository: spark
Updated Branches:
  refs/heads/branch-1.5 e4cf12118 -> 0fdf5542b


[SPARK-12275] [SQL] No plan for BroadcastHint in some condition - 1.5 backport

backport #10265 to branch 1.5.

When SparkStrategies.BasicOperators's "case BroadcastHint(child) => 
apply(child)" is hit,
it only recursively invokes BasicOperators.apply with this "child".
It makes many strategies have no change to process this plan,
which probably leads to "No plan" issue, so we use planLater to go through all 
strategies.

https://issues.apache.org/jira/browse/SPARK-12275

Author: yucai 

Closes #10291 from yucai/backport_1.5_no_plan_for_broadcasthint and squashes 
the following commits:

b09715c [yucai] [SPARK-12275][SQL] No plan for BroadcastHint in some condition 
- 1.5 backport


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0fdf5542
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0fdf5542
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0fdf5542

Branch: refs/heads/branch-1.5
Commit: 0fdf5542bfa1627bf084735d6b9184ef12291941
Parents: e4cf121
Author: yucai 
Authored: Mon Dec 14 13:05:48 2015 -0800
Committer: Andrew Or 
Committed: Mon Dec 14 13:05:48 2015 -0800

--
 .../org/apache/spark/sql/execution/SparkStrategies.scala  | 2 +-
 .../test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala  | 7 +++
 2 files changed, 8 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/0fdf5542/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
--
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
index 4df5368..2b514e0 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
@@ -388,7 +388,7 @@ private[sql] abstract class SparkStrategies extends 
QueryPlanner[SparkPlan] {
   case e @ EvaluatePython(udf, child, _) =>
 BatchPythonEvaluation(udf, e.output, planLater(child)) :: Nil
   case LogicalRDD(output, rdd) => PhysicalRDD(output, rdd, "PhysicalRDD") 
:: Nil
-  case BroadcastHint(child) => apply(child)
+  case BroadcastHint(child) => planLater(child) :: Nil
   case _ => Nil
 }
   }

http://git-wip-us.apache.org/repos/asf/spark/blob/0fdf5542/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
--
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
index e2716d7..718e158 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
@@ -107,5 +107,12 @@ class DataFrameJoinSuite extends QueryTest with 
SharedSQLContext {
 
 // planner should not crash without a join
 broadcast(df1).queryExecution.executedPlan
+
+// SPARK-12275: no physical plan for BroadcastHint in some condition
+withTempPath { path =>
+  df1.write.parquet(path.getCanonicalPath)
+  val pf1 = sqlContext.read.parquet(path.getCanonicalPath)
+  assert(df1.join(broadcast(pf1)).count() === 4)
+}
   }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org