[GitHub] spark pull request: [SPARK-12503] [SQL] Pushing Limit Through Unio...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10451#issuecomment-167213191
  
**[Test build #48322 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48322/consoleFull)**
 for PR 10451 at commit 
[`3ccf3bd`](https://github.com/apache/spark/commit/3ccf3bd11f5ee89905c4a5cc4466fd4d332d0b42).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Limit(limitExpr: Expression, child: LogicalPlan, optimized: 
Boolean) extends UnaryNode `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12503] [SQL] Pushing Limit Through Unio...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10451#issuecomment-167213359
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48322/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12503] [SQL] Pushing Limit Through Unio...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10451#issuecomment-167213357
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167220952
  
**[Test build #48323 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48323/consoleFull)**
 for PR 10330 at commit 
[`04eb37e`](https://github.com/apache/spark/commit/04eb37edca49c0c5a344c17c898924e15e1680b9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12521][SQL][WIP] JDBCRelation does not ...

[hvanhovell]

Github user hvanhovell commented on the pull request:

https://github.com/apache/spark/pull/10473#issuecomment-167214192
  
@xguo27 see my comment on the JIRA: 
https://issues.apache.org/jira/browse/SPARK-12521

I think this is intended, and should not be altered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/10330#discussion_r48437827
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -200,6 +204,10 @@ case class TakeOrderedAndProject(
 projectOutput.getOrElse(child.output)
   }
 
+  override def outputsUnsafeRows: Boolean = true
--- End diff --

OK. I've updated it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/10330#discussion_r48441149
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -207,11 +220,15 @@ case class TakeOrderedAndProject(
   private val ord: InterpretedOrdering = new 
InterpretedOrdering(sortOrder, child.output)
 
   // TODO: remove @transient after figure out how to clean closure at 
InsertIntoHiveTable.
-  @transient private val projection = projectList.map(new 
InterpretedProjection(_, child.output))
+  @transient private val projection = 
projectList.map(UnsafeProjection.create(_, child.output))
 
   private def collectData(): Array[InternalRow] = {
 val data = child.execute().map(_.copy()).takeOrdered(limit)(ord)
-projection.map(data.map(_)).getOrElse(data)
+if (projection.isDefined) {
+  projection.map(p => 
data.map(p(_).copy().asInstanceOf[InternalRow])).get
--- End diff --

OK. I was thinking that it is needed to copy the returned row because it is 
the same object. But after I checked GenerateUnsafeProjection, looks like it 
will create new row every time. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167246248
  
**[Test build #48325 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48325/consoleFull)**
 for PR 10435 at commit 
[`04a7301`](https://github.com/apache/spark/commit/04a730154283a6125f05ed984115adf2e455ac60).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12413] Fix Mesos ZK persistence

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/10366#discussion_r48442537
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/rest/mesos/MesosRestServer.scala ---
@@ -99,7 +99,11 @@ private[mesos] class MesosSubmitRequestServlet(
 // cause spark-submit script to look for files in SPARK_HOME instead.
 // We only need the ability to specify where to find spark-submit 
script
 // which user can user spark.executor.home or spark.home 
configurations.
-val environmentVariables = 
request.environmentVariables.filterKeys(!_.equals("SPARK_HOME"))
+//
+// Do not use `filterKeys` here to avoid SI-6654, which breaks ZK 
persistence
--- End diff --

Should a scalastyle rule be added which prevents filterKeys to be used in 
future PRs ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167245693
  
**[Test build #48324 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48324/consoleFull)**
 for PR 10330 at commit 
[`6a519d8`](https://github.com/apache/spark/commit/6a519d85b58f25776be6e5819025b1bf485f7da8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/10330#discussion_r48440905
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -571,6 +571,12 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   mapData.collect().take(1).map(Row.fromTuple).toSeq)
   }
 
+  test("sort and limit") {
+checkAnswer(
+  sql("SELECT * FROM arrayData ORDER BY data[0] ASC LIMIT 1"),
--- End diff --

Will this test fail without your change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/10330#discussion_r48440889
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -207,11 +220,15 @@ case class TakeOrderedAndProject(
   private val ord: InterpretedOrdering = new 
InterpretedOrdering(sortOrder, child.output)
 
   // TODO: remove @transient after figure out how to clean closure at 
InsertIntoHiveTable.
-  @transient private val projection = projectList.map(new 
InterpretedProjection(_, child.output))
+  @transient private val projection = 
projectList.map(UnsafeProjection.create(_, child.output))
 
   private def collectData(): Array[InternalRow] = {
 val data = child.execute().map(_.copy()).takeOrdered(limit)(ord)
-projection.map(data.map(_)).getOrElse(data)
+if (projection.isDefined) {
+  projection.map(p => 
data.map(p(_).copy().asInstanceOf[InternalRow])).get
--- End diff --

Do we need `copy` here? We have already copied the rows when getting `data`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12385] [SQL] [WIP] Push Projection into...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10476#issuecomment-167250217
  
**[Test build #48326 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48326/consoleFull)**
 for PR 10476 at commit 
[`ae408e8`](https://github.com/apache/spark/commit/ae408e8085b5777ef69206f37d5ece563b200926).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167250340
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48324/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167250339
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167250332
  
**[Test build #48324 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48324/consoleFull)**
 for PR 10330 at commit 
[`6a519d8`](https://github.com/apache/spark/commit/6a519d85b58f25776be6e5819025b1bf485f7da8).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12385] [SQL] [WIP] Push Projection into...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10476#issuecomment-167251950
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48326/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167244841
  
overall LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/10330#discussion_r48441225
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/RowFormatConvertersSuite.scala
 ---
@@ -38,7 +38,7 @@ class RowFormatConvertersSuite extends SparkPlanTest with 
SharedSQLContext {
   private val outputsUnsafe = Sort(Nil, false, PhysicalRDD(Seq.empty, 
null, "name"))
   assert(outputsUnsafe.outputsUnsafeRows)
 
-  test("planner should insert unsafe->safe conversions when required") {
+  ignore("planner should insert unsafe->safe conversions when required") {
--- End diff --

cc @marmbrus @yhuai should we remove this test?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167251759
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48325/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167251754
  
**[Test build #48325 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48325/consoleFull)**
 for PR 10435 at commit 
[`04a7301`](https://github.com/apache/spark/commit/04a730154283a6125f05ed984115adf2e455ac60).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Hash(children: Seq[Expression]) extends Expression `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167251758
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167225803
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167225775
  
**[Test build #48323 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48323/consoleFull)**
 for PR 10330 at commit 
[`04eb37e`](https://github.com/apache/spark/commit/04eb37edca49c0c5a344c17c898924e15e1680b9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167225804
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48323/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12385] [SQL] [WIP] Push Projection into...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10476#issuecomment-167251942
  
**[Test build #48326 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48326/consoleFull)**
 for PR 10476 at commit 
[`ae408e8`](https://github.com/apache/spark/commit/ae408e8085b5777ef69206f37d5ece563b200926).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12385] [SQL] [WIP] Push Projection into...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10476#issuecomment-167251949
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/10333#discussion_r48442383
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ---
@@ -87,18 +87,21 @@ object Cast {
   private def resolvableNullability(from: Boolean, to: Boolean) = !from || 
to
 
   private def forceNullable(from: DataType, to: DataType) = (from, to) 
match {
-case (StringType, _: NumericType) => true
-case (StringType, TimestampType) => true
-case (DoubleType, TimestampType) => true
-case (FloatType, TimestampType) => true
-case (StringType, DateType) => true
-case (_: NumericType, DateType) => true
-case (BooleanType, DateType) => true
-case (DateType, _: NumericType) => true
-case (DateType, BooleanType) => true
-case (DoubleType, _: DecimalType) => true
-case (FloatType, _: DecimalType) => true
-case (_, DecimalType.Fixed(_, _)) => true // TODO: not all upcasts 
here can really give null
+case (NullType, _) => true
+case (_, _) if from == to => false
+
+case (StringType, BinaryType) => false
+case (StringType, _) => true
+
+case (FloatType | DoubleType, TimestampType) => true
+case (TimestampType, DateType) => false
+case (_, DateType) => true
+case (DateType, TimestampType) => false
+case (DateType, _) => true
--- End diff --

In the current ccde, there is no case for (DateType, StringType).
I can send a PR if this is to be added


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12521][SQL][WIP] JDBCRelation does not ...

[xguo27]

Github user xguo27 closed the pull request at:

https://github.com/apache/spark/pull/10473


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12521][SQL][WIP] JDBCRelation does not ...

[xguo27]

Github user xguo27 commented on the pull request:

https://github.com/apache/spark/pull/10473#issuecomment-167258976
  
Thanks @hvanhovell for clarifying it up. I will close this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/10330#discussion_r48441083
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -571,6 +571,12 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   mapData.collect().take(1).map(Row.fromTuple).toSeq)
   }
 
+  test("sort and limit") {
+checkAnswer(
+  sql("SELECT * FROM arrayData ORDER BY data[0] ASC LIMIT 1"),
--- End diff --

This should be removed. I just test it locally.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12503] [SQL] Pushing Limit Through Unio...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/10451#discussion_r48441937
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -153,6 +153,16 @@ object SetOperationPushDown extends Rule[LogicalPlan] 
with PredicateHelper {
 )
   )
 
+// Push down limit into union
+case Limit(exp, Union(left, right), optimized) if !optimized =>
--- End diff --

Thank you! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12385] [SQL] [WIP] Push Projection into...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/10476

[SPARK-12385] [SQL] [WIP] Push Projection into Join

```sql("SELECT * FROM testData JOIN testData2").explain(true)```

The current plan is like
```
== Analyzed Logical Plan ==
key: int, value: string, a: int, b: int
Project [key#2,value#3,a#4,b#5]
+- Join Inner, None
   :- Subquery testData
   :  +- LogicalRDD [key#2,value#3], MapPartitionsRDD[3] at beforeAll at 
BeforeAndAfterAll.scala:187
   +- Subquery testData2
  +- LogicalRDD [a#4,b#5], MapPartitionsRDD[5] at beforeAll at 
BeforeAndAfterAll.scala:187

== Optimized Logical Plan ==
Project [key#2,value#3,a#4,b#5]
+- Join Inner, None
   :- InMemoryRelation [key#2,value#3], true, 1, StorageLevel(true, 
true, false, true, 1), ConvertToUnsafe, Some(testData)
   +- LogicalRDD [a#4,b#5], MapPartitionsRDD[5] at beforeAll at 
BeforeAndAfterAll.scala:187
```

After this PR, the new plan does not have the unnecessary Project. 
```
== Optimized Logical Plan ==
Join Inner, None
:- InMemoryRelation [key#2,value#3], true, 1, StorageLevel(true, true, 
false, true, 1), ConvertToUnsafe, Some(testData)
+- LogicalRDD [a#4,b#5], MapPartitionsRDD[5] at beforeAll at 
BeforeAndAfterAll.scala:187
```

However, it loses the ordering and alias information of attributes in 
`Projection`. This is just a preliminary work. If their `outputSet`s do not 
match, we still can push it into the `Join`, since the runtime/execution of 
Join will do the Projection. 

**Question**: Is that OK to add another field `projectList: 
Seq[NamedExpression]` into `Join` of `LogicalPlan`? Is this the right 
direction? 

Even more aggressively, we can add it into the `LogicalPlan` nodes? In the 
long term, should we push Projection into the other node types?

Thank you!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark combineProjectJoin

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10476


commit ae408e8085b5777ef69206f37d5ece563b200926
Author: gatorsmile 
Date:   2015-12-25T06:17:53Z

combine Project and Join.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...

[radekg]

Github user radekg commented on a diff in the pull request:

https://github.com/apache/spark/pull/9608#discussion_r48442211
  
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala 
---
@@ -122,7 +122,8 @@ private[netty] class NettyRpcEnv(
 
   @Nullable
   override lazy val address: RpcAddress = {
-if (server != null) RpcAddress(host, server.getPort()) else null
+if (server != null)
+  RpcAddress(host, conf.getInt("spark.driver.advertisedPort", 
server.getPort())) else null
--- End diff --

My assumption was that this was the safest behaviour for the Netty based 
rpc. If the `spark.driver.advertisedPort` setting is `0`, it's an operator 
error, no different than setting the value to, say, `1`. Not sure if there is 
any value in testing for `0` here. Ideas welcome.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12503] [SQL] Pushing Limit Through Unio...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/10451#discussion_r48441867
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -153,6 +153,16 @@ object SetOperationPushDown extends Rule[LogicalPlan] 
with PredicateHelper {
 )
   )
 
+// Push down limit into union
+case Limit(exp, Union(left, right), optimized) if !optimized =>
--- End diff --

This works, but doesn't look good to me. Actually I don't have a good idea 
for it either, maybe we should just add comments to explain the assumption here.

cc @marmbrus @yhuai 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12520] [PySpark] Correct Descriptions a...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/10477

[SPARK-12520] [PySpark] Correct Descriptions and Add Use Cases in Equi-Join

After reading the JIRA https://issues.apache.org/jira/browse/SPARK-12520, I 
double checked the code. 

For example, users can do the Equi-Join like
  ```df.join(df2, 'name', 'outer').select('name', 'height').collect()```
- There exists a bug in 1.5 and 1.4. The code just ignores the third 
parameter (join type) users pass. However, the join type we called is `Inner`, 
even if the user-specified type is the other type (e.g., `Outer`). 
- After a PR: https://github.com/apache/spark/pull/8600, the 1.6 does not 
have such an issue, but the description has not been updated.  

Plan to submit another PR to fix 1.5 and issue an error message if users 
specify a non-inner join type when using Equi-Join. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark pyOuterJoin

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10477.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10477


commit 29231828d47ff74c190fae782ae08bfe89861958
Author: gatorsmile 
Date:   2015-12-25T18:35:19Z

equi-join with the other join type.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12520] [PySpark] Correct Descriptions a...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10477#issuecomment-167260644
  
**[Test build #48327 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48327/consoleFull)**
 for PR 10477 at commit 
[`2923182`](https://github.com/apache/spark/commit/29231828d47ff74c190fae782ae08bfe89861958).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12477][HOTIFX] Fix test compilation in ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10478#issuecomment-16726
  
  [Test build #48329 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48329/consoleFull)
 for   PR 10478 at commit 
[`34e0b8a`](https://github.com/apache/spark/commit/34e0b8aeafbcee1adcdf868b02a01d1df7d627f8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167268191
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167268192
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48330/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167268190
  
**[Test build #48330 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48330/consoleFull)**
 for PR 10479 at commit 
[`0b79c39`](https://github.com/apache/spark/commit/0b79c39a3cb8db0447a7dab0e92857d0d5c79665).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167273547
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48331/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12224][SPARKR] R support for JDBC sourc...

[felixcheung]

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/10480

[SPARK-12224][SPARKR] R support for JDBC source

Add R API for `read.jdbc`, `write.jdbc`.

Tested this quite a bit manually with different combinations of parameters. 
It's not clear if we could have automated tests in R for this - Scala 
`JDBCSuite` depends on Java H2 in-memory database.

Refactored some code into util so they could be tested.

Core's R SerDe code needs to be updated to allow access to 
java.util.Properties as `jobj` handle which is required by 
DataFrameReader/Writer's `jdbc` method. It would be more code to add a 
`sql/r/SQLUtils` helper function.

Tested:
```
# with postgresql
../bin/sparkR --driver-class-path 
/usr/share/java/postgresql-9.4.1207.jre7.jar

# read.jdbc
df <- read.jdbc(sqlContext, "jdbc:postgresql://localhost/db", "films2", 
user = "user", password = "12345")
df <- read.jdbc(sqlContext, "jdbc:postgresql://localhost/db", "films2", 
user = "user", password = 12345)

# partitionColumn and numPartitions test
df <- read.jdbc(sqlContext, "jdbc:postgresql://localhost/db", "films2", 
partitionColumn = "did", lowerBound = 0, upperBound = 200, numPartitions = 4, 
user = "user", password = 12345)
a <- SparkR:::toRDD(df)
SparkR:::getNumPartitions(a)
[1] 4
SparkR:::collectPartition(a, 2L)

# defaultParallelism test
df <- read.jdbc(sqlContext, "jdbc:postgresql://localhost/db", "films2", 
partitionColumn = "did", lowerBound = 0, upperBound = 200, user = "user", 
password = 12345)
SparkR:::getNumPartitions(a)
[1] 2

# predicates test
df <- read.jdbc(sqlContext, "jdbc:postgresql://localhost/db", "films2", 
predicates = list("did<=105"), user = "user", password = 12345)
count(df) == 1

# write.jdbc, default save mode "error"
irisDf <- as.DataFrame(sqlContext, iris)
write.jdbc(irisDf, "jdbc:postgresql://localhost/db", "films2", user = 
"user", password = "12345")
"error, already exists"

write.jdbc(irisDf, "jdbc:postgresql://localhost/db", "iris", user = "user", 
password = "12345")
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rreadjdbc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10480.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10480


commit b0f28523f17d7733d4e1fe2b7e040db127b7188b
Author: felixcheung 
Date:   2015-12-24T03:51:10Z

read.jdbc support

commit c3e7bec0de7658c954539b421168e966bea17843
Author: felixcheung 
Date:   2015-12-24T03:52:56Z

update comment

commit 5a0f6d2a3be14931126ac420b31f94a276eb5c02
Author: felixcheung 
Date:   2015-12-24T04:38:14Z

write.jdbc, doc update

commit 5b15f38d151b5c90bc9e41ef02ad25712eec31f4
Author: felixcheung 
Date:   2015-12-24T04:48:45Z

more doc update

commit 98ec05e8b9ec79ea8ea424dfbfc22f15ab0f7429
Author: felixcheung 
Date:   2015-12-24T16:02:42Z

update doc

commit 12b36130fe1bd6c526c6e7ba89deb77b0cfb0ee3
Author: felixcheung 
Date:   2015-12-24T16:12:44Z

code fix

commit 3f90db6686daea182161de3b4c668b6901be89c9
Author: felixcheung 
Date:   2015-12-26T03:50:51Z

fix serialization of java.util.Properties, add tests for util functions, 
add generic, fix bugs

commit de635b18147e42bb931e8d81e51522330873498e
Author: felixcheung 
Date:   2015-12-26T03:53:22Z

update doc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12224][SPARKR] R support for JDBC sourc...

[felixcheung]

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/10480#issuecomment-167276041
  
@shivaram @sun-rui 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167277279
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48334/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167277256
  
**[Test build #48334 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48334/consoleFull)**
 for PR 10479 at commit 
[`0a3f671`](https://github.com/apache/spark/commit/0a3f671883923b67d872a025092e07b3d7ca358c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12526][SPARKR]`ifelse`, `when`, `otherw...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10481#issuecomment-167281017
  
**[Test build #48336 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48336/consoleFull)**
 for PR 10481 at commit 
[`449b0f6`](https://github.com/apache/spark/commit/449b0f6074d08309ad1e3fa6a6611b2fb6a33a5e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167284629
  
**[Test build #48338 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48338/consoleFull)**
 for PR 10482 at commit 
[`2e1e58e`](https://github.com/apache/spark/commit/2e1e58e34ccb42c693822431f53b6d4e292fbf8d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291377
  
**[Test build #48341 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48341/consoleFull)**
 for PR 10482 at commit 
[`17572b8`](https://github.com/apache/spark/commit/17572b87aec97b6e35a05dd7f2314a6d5853af94).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291379
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48341/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291378
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[ted-yu]

Github user ted-yu commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291348
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOT-FIX] bypass hive test when parse logical ...

[tedyu]

Github user tedyu commented on the pull request:

https://github.com/apache/spark/pull/10430#issuecomment-167264161
  
Can this PR be merged so that maven build becomes green again ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12508][PROJECT-INFRA] Fix minor bugs in...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10455#issuecomment-167267735
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48328/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12508][PROJECT-INFRA] Fix minor bugs in...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10455#issuecomment-167267734
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12224][SPARKR] R support for JDBC sourc...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10480#issuecomment-167276610
  
**[Test build #48335 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48335/consoleFull)**
 for PR 10480 at commit 
[`de635b1`](https://github.com/apache/spark/commit/de635b18147e42bb931e8d81e51522330873498e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167277278
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167284876
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167284823
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167284824
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48338/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167284822
  
**[Test build #48338 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48338/consoleFull)**
 for PR 10482 at commit 
[`2e1e58e`](https://github.com/apache/spark/commit/2e1e58e34ccb42c693822431f53b6d4e292fbf8d).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291310
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48340/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291307
  
**[Test build #48340 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48340/consoleFull)**
 for PR 10482 at commit 
[`17572b8`](https://github.com/apache/spark/commit/17572b87aec97b6e35a05dd7f2314a6d5853af94).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291309
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167292150
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167292083
  
**[Test build #48337 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48337/consoleFull)**
 for PR 10435 at commit 
[`f408f1f`](https://github.com/apache/spark/commit/f408f1ff0e9b4839a48dea56a17ca4755ad8f717).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Hash(children: Seq[Expression]) extends Expression `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167292151
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48337/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12477][HOTIFX] Fix test compilation in ...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/10478#issuecomment-167270327
  
Merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12396][Core] Modify the function schedu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10447


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12526][SPARKR]`ifelse`, `when`, `otherw...

[saurfang]

GitHub user saurfang opened a pull request:

https://github.com/apache/spark/pull/10481

[SPARK-12526][SPARKR]`ifelse`, `when`, `otherwise` unable to take Column as 
value

`ifelse`, `when`, `otherwise` is unable to take `Column` typed S4 object as 
values.

For example:
```r
ifelse(lit(1) == lit(1), lit(2), lit(3))
ifelse(df$mpg > 0, df$mpg, 0)
```
will both fail with
```r
attempt to replicate an object of type 'environment'
```

The PR replaces `ifelse` calls with `if ... else ...` inside the function 
implementations to avoid attempt to vectorize(i.e. `rep()`). It remains to be 
discussed whether we should instead support vectorization in these functions 
for consistency because `ifelse` in base R is vectorized but I cannot foresee 
any scenarios these functions will want to be vectorized in SparkR.

For reference, added test cases which trigger failures:
```r
. Error: when(), otherwise() and ifelse() with column on a DataFrame 
--
error in evaluating the argument 'x' in selecting a method for function 
'collect': 
  error in evaluating the argument 'col' in selecting a method for function 
'select': 
  attempt to replicate an object of type 'environment'
Calls: when -> when -> ifelse -> ifelse

1: withCallingHandlers(eval(code, new_test_environment), error = 
capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: expect_equal(collect(select(df, when(df$a > 1 & df$b > 2, lit(1[, 
1], c(NA, 1)) at test_sparkSQL.R:1126
5: expect_that(object, equals(expected, label = expected.label, ...), info 
= info, label = label)
6: condition(object)
7: compare(actual, expected, ...)
8: collect(select(df, when(df$a > 1 & df$b > 2, lit(1
Error: Test failures
Execution halted
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/saurfang/spark spark-12526

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10481.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10481


commit 449b0f6074d08309ad1e3fa6a6611b2fb6a33a5e
Author: Forest Fang 
Date:   2015-12-26T05:21:18Z

replace ifelse with if...else... to avoid vectorization




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12526][SPARKR]`ifelse`, `when`, `otherw...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10481#issuecomment-167281714
  
**[Test build #48336 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48336/consoleFull)**
 for PR 10481 at commit 
[`449b0f6`](https://github.com/apache/spark/commit/449b0f6074d08309ad1e3fa6a6611b2fb6a33a5e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12526][SPARKR]`ifelse`, `when`, `otherw...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10481#issuecomment-167281934
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12224][SPARKR] R support for JDBC sourc...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10480#issuecomment-167283619
  
**[Test build #48335 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48335/consoleFull)**
 for PR 10480 at commit 
[`de635b1`](https://github.com/apache/spark/commit/de635b18147e42bb931e8d81e51522330873498e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12224][SPARKR] R support for JDBC sourc...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10480#issuecomment-167283792
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48335/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12224][SPARKR] R support for JDBC sourc...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10480#issuecomment-167283791
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167283801
  
**[Test build #48337 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48337/consoleFull)**
 for PR 10435 at commit 
[`f408f1f`](https://github.com/apache/spark/commit/f408f1ff0e9b4839a48dea56a17ca4755ad8f717).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291114
  
**[Test build #48341 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48341/consoleFull)**
 for PR 10482 at commit 
[`17572b8`](https://github.com/apache/spark/commit/17572b87aec97b6e35a05dd7f2314a6d5853af94).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167289961
  
**[Test build #48340 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48340/consoleFull)**
 for PR 10482 at commit 
[`17572b8`](https://github.com/apache/spark/commit/17572b87aec97b6e35a05dd7f2314a6d5853af94).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167293794
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167293796
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48343/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167293612
  
**[Test build #48343 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48343/consoleFull)**
 for PR 10482 at commit 
[`a410099`](https://github.com/apache/spark/commit/a4100991984be851ca0ce520f896008fac86ada1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167293884
  
**[Test build #48342 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48342/consoleFull)**
 for PR 10482 at commit 
[`a410099`](https://github.com/apache/spark/commit/a4100991984be851ca0ce520f896008fac86ada1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[maver1ck]

Github user maver1ck commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-167260638
  
I agree.
In YARN mode we have configuration per node
```
YARN: The --num-executors option to the Spark YARN client controls how many 
executors it will allocate on the cluster, while --executor-memory and 
--executor-cores control the resources per executor.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12508][PROJECT-INFRA] Fix minor bugs in...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10455#issuecomment-167264690
  
**[Test build #2255 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2255/consoleFull)**
 for PR 10455 at commit 
[`dc8d552`](https://github.com/apache/spark/commit/dc8d552881a8c0c4ac5a6c7fdac8fd4408911f77).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[JoshRosen]

GitHub user JoshRosen opened a pull request:

https://github.com/apache/spark/pull/10479

[SPARK-12525] Fix fatal compiler warnings in Kinesis ASL due to @transient 
annotations

The Scala 2.11 SBT build currently fails for Spark 1.6.0 and master due to 
warnings about the `@transient` annotation:

```
[error] [warn] 
/Users/joshrosen/Documents/spark/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala:73:
 no valid targets for annotation on value sc - it is discarded unused. You may 
specify targets with meta-annotations, e.g. @(transient @param)
[error] [warn] @transient sc: SparkContext,
```

This fix implemented here is the same as what we did in #8433: remove the 
`@transient` annotations when they are not necessary and replace use  
`@transient private val` in the remaining cases.

/cc @srowen and @skyluc for review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JoshRosen/spark fix-sbt-2.11

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10479.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10479


commit 0b79c39a3cb8db0447a7dab0e92857d0d5c79665
Author: Josh Rosen 
Date:   2015-12-25T23:19:02Z

Fix compiler warnings due to @transient annotations.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167268026
  
**[Test build #48330 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48330/consoleFull)**
 for PR 10479 at commit 
[`0b79c39`](https://github.com/apache/spark/commit/0b79c39a3cb8db0447a7dab0e92857d0d5c79665).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12477][HOTIFX] Fix test compilation in ...

[JoshRosen]

Github user JoshRosen closed the pull request at:

https://github.com/apache/spark/pull/10478


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12396][Core] Modify the function schedu...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/10447#issuecomment-167270292
  
Merging to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12396][Core] Modify the function schedu...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/10447#issuecomment-167270325
  
@echoTomei Thanks a lot!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167273539
  
**[Test build #48331 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48331/consoleFull)**
 for PR 10330 at commit 
[`f3054d6`](https://github.com/apache/spark/commit/f3054d63a32a9cb71ff0ed16b828a5af71cc).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12289][SQL] Support UnsafeRow in TakeOr...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10330#issuecomment-167273546
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167274817
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167274815
  
**[Test build #48333 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48333/consoleFull)**
 for PR 10479 at commit 
[`afe11f2`](https://github.com/apache/spark/commit/afe11f225e60d054a4f72681fba53cf220ecadeb).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12525] Fix fatal compiler warnings in K...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10479#issuecomment-167274818
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48333/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167284877
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48339/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167284872
  
**[Test build #48339 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48339/consoleFull)**
 for PR 10482 at commit 
[`2e1e58e`](https://github.com/apache/spark/commit/2e1e58e34ccb42c693822431f53b6d4e292fbf8d).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[ted-yu]

Github user ted-yu commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167287706
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291989
  
**[Test build #48343 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48343/consoleFull)**
 for PR 10482 at commit 
[`a410099`](https://github.com/apache/spark/commit/a4100991984be851ca0ce520f896008fac86ada1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10482#issuecomment-167291954
  
**[Test build #48342 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48342/consoleFull)**
 for PR 10482 at commit 
[`a410099`](https://github.com/apache/spark/commit/a4100991984be851ca0ce520f896008fac86ada1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12520] [PySpark] Correct Descriptions a...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10477#issuecomment-167261159
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org