[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11044#issuecomment-179081856
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11044#issuecomment-179081857
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50647/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11046#issuecomment-179117933
  
**[Test build #50656 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50656/consoleFull)**
 for PR 11046 at commit 
[`45baa11`](https://github.com/apache/spark/commit/45baa112d053bd49feefe7003599afbad2272634).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095] [Mesos] Support launching multipl...

[dragos]

Github user dragos commented on a diff in the pull request:

https://github.com/apache/spark/pull/10993#discussion_r51704761
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
 ---
@@ -373,40 +451,25 @@ private[spark] class CoarseMesosSchedulerBackend(
   override def frameworkMessage(d: SchedulerDriver, e: ExecutorID, s: 
SlaveID, b: Array[Byte]) {}
 
   /**
-   * Called when a slave is lost or a Mesos task finished. Update local 
view on
-   * what tasks are running and remove the terminated slave from the list 
of pending
-   * slave IDs that we might have asked to be killed. It also notifies the 
driver
-   * that an executor was removed.
+   * Called when a slave is lost or a Mesos task finished. Updates local 
view on
+   * what tasks are running. It also notifies the driver that an executor 
was removed.
*/
-  private def executorTerminated(d: SchedulerDriver, slaveId: String, 
reason: String): Unit = {
+  private def executorTerminated(d: SchedulerDriver,
+ slaveId: String,
+ taskId: String,
+ reason: String): Unit = {
 stateLock.synchronized {
-  if (slaveIdsWithExecutors.contains(slaveId)) {
-val slaveIdToTaskId = taskIdToSlaveId.inverse()
-if (slaveIdToTaskId.containsKey(slaveId)) {
-  val taskId: Int = slaveIdToTaskId.get(slaveId)
-  taskIdToSlaveId.remove(taskId)
-  removeExecutor(sparkExecutorId(slaveId, taskId.toString), 
SlaveLost(reason))
-}
-// TODO: This assumes one Spark executor per Mesos slave,
-// which may no longer be true after SPARK-5095
-pendingRemovedSlaveIds -= slaveId
-slaveIdsWithExecutors -= slaveId
-  }
+  removeExecutor(taskId, SlaveLost(reason))
+  slaves(slaveId).taskIDs.remove(taskId)
--- End diff --

You're right that memory-wise it's not a big loss. But I prefer clean code. 
For instance, what's the meaning of having a Slave record for a host that 
doesn't run any tasks? Can this become a source of confusion down the road? 
Since it seems that it's not that complex to clean up, I'd go for it now.

```
def removeTask(slave: Slave, taskId: String) = {
slave.taskIDs.remove(taskId)
if (slave.taskIDs.isEmpty) {
  slaves.remove(slaveId)
}
  }
```

In fact, this method may go inside `Slave`, who could properly encapsulate 
`taskIDs`. It may even be an inner class, so it can update `slaves`. Unless I'm 
missing something, we're talking about two lines of code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-179080593
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-179080358
  
**[Test build #50646 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50646/consoleFull)**
 for PR 8880 at commit 
[`71cb276`](https://github.com/apache/spark/commit/71cb2765661ff12bb2ea7d1c35a0759497e5c6c1).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-179080596
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50646/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[yanboliang]

Github user yanboliang commented on a diff in the pull request:

https://github.com/apache/spark/pull/11044#discussion_r51690764
  
--- Diff: python/pyspark/ml/recommendation.py ---
@@ -81,6 +82,23 @@ class ALS(JavaEstimator, HasCheckpointInterval, 
HasMaxIter, HasPredictionCol, Ha
 Row(user=1, item=0, prediction=2.6258413791656494)
 >>> predictions[2]
 Row(user=2, item=0, prediction=-1.5018409490585327)
+>>> import os, tempfile
+>>> path = tempfile.mkdtemp()
+>>> ALS_path = path + "/als"
+>>> als.save(ALS_path)
+>>> als2 = ALS.load(ALS_path)
+>>> als.getMaxIter()
+5
+>>> model_path = path + "/als_model"
+>>> model.save(model_path)
+>>> model2 = ALSModel.load(model_path)
+>>> model.rank == model2.rank
+True
--- End diff --

Can we also add test for ```userFactors``` or ```itemFactors```?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[maropu]

GitHub user maropu opened a pull request:

https://github.com/apache/spark/pull/11046

[SPARK-13158][Web UI] Show the information of broadcast blocks in WebUI

This pr enables a function to show the information of broadcast blocks, # 
of blocks total size in mem/disk in a cluster. This is derived from the 
question in spark-user mailing list.
https://www.mail-archive.com/user@spark.apache.org/msg45068.html


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maropu/spark BroadcastInfoInStoragePage

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11046.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11046


commit 45baa112d053bd49feefe7003599afbad2272634
Author: Takeshi YAMAMURO 
Date:   2016-02-03T08:46:48Z

Show the information of broadcast blocks in WebUI




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12506][SPARK-12126][SQL]use CatalystSca...

[maropu]

Github user maropu commented on the pull request:

https://github.com/apache/spark/pull/11005#issuecomment-179135955
  
I agree that `CatalystScan` is used to support arithmetic operations in 
datasources though, this current `CatalystScan` trait only processes filter 
expressions. If we make this kind of interface changes, it'd be better to 
consider other requirements discussed in jira; otherwise, we get stuck with the 
same issue every time we add new features in datasources.

Anyway, the idea to share codes in `SQLBuilder` and `JDBCRDD` is good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10841#issuecomment-179144758
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50654/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10841#issuecomment-179144751
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL][WIP] Create native DDL comm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11048#issuecomment-179150064
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11046#issuecomment-179177083
  
**[Test build #50659 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50659/consoleFull)**
 for PR 11046 at commit 
[`45baa11`](https://github.com/apache/spark/commit/45baa112d053bd49feefe7003599afbad2272634).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12888][SQL][follow-up] benchmark the ne...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10917#issuecomment-179094519
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50653/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12888][SQL][follow-up] benchmark the ne...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10917#issuecomment-179094513
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12113] [SQL] Add some timing metrics fo...

[maropu]

Github user maropu commented on the pull request:

https://github.com/apache/spark/pull/10116#issuecomment-179112669
  
@nongli okay and I'll do it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8321][SQL][WIP] Authorization Support(o...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11045#issuecomment-179077963
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[vectorijk]

Github user vectorijk commented on a diff in the pull request:

https://github.com/apache/spark/pull/11044#discussion_r51691090
  
--- Diff: python/pyspark/ml/recommendation.py ---
@@ -81,6 +82,23 @@ class ALS(JavaEstimator, HasCheckpointInterval, 
HasMaxIter, HasPredictionCol, Ha
 Row(user=1, item=0, prediction=2.6258413791656494)
 >>> predictions[2]
 Row(user=2, item=0, prediction=-1.5018409490585327)
+>>> import os, tempfile
+>>> path = tempfile.mkdtemp()
+>>> ALS_path = path + "/als"
+>>> als.save(ALS_path)
+>>> als2 = ALS.load(ALS_path)
+>>> als.getMaxIter()
+5
+>>> model_path = path + "/als_model"
+>>> model.save(model_path)
+>>> model2 = ALSModel.load(model_path)
+>>> model.rank == model2.rank
+True
--- End diff --

right, I also realized this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...

[dragos]

Github user dragos commented on a diff in the pull request:

https://github.com/apache/spark/pull/10319#discussion_r51693326
  
--- Diff: docs/running-on-mesos.md ---
@@ -387,6 +387,13 @@ See the [configuration page](configuration.html) for 
information on Spark config
 
   
 
+
+  spark.mesos.coarse.shutdown.ms
+  1 (10 seconds)
+   
+Time (in ms) to wait for executors to report that they have exited. 
Setting this too low has the risk of shutting down the Mesos driver (and 
thereby killing the spark executors) while the executor is still in the process 
of exiting cleanly.
--- End diff --

No, I wasn't really thinking too hard about it. I agree it's not worth 
making this a user-facing property.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11044#issuecomment-179119703
  
**[Test build #50655 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50655/consoleFull)**
 for PR 11044 at commit 
[`a294918`](https://github.com/apache/spark/commit/a294918f5d82c4aaa296ac35a9ba85d330612e0a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12978][SQL] Skip unnecessary final grou...

[maropu]

Github user maropu commented on the pull request:

https://github.com/apache/spark/pull/10896#issuecomment-179139867
  
Yes, it is. The input query is;
```
val fields = Seq(StringType, DoubleType, DoubleType)
  .zipWithIndex.map { case (dataType, index) =>
StructField(s"col$index", dataType, true)
  }
 
val df = sqlContext.createDataFrame(rdd, StructType(fields))
val df2 = df.repartition($"col0").cache
val df3 = df2.groupBy($"col0").agg(Map("col1"->"sum", "col2"->"avg"))
df3.explain(true)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[maropu]

Github user maropu commented on the pull request:

https://github.com/apache/spark/pull/11046#issuecomment-179175443
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13101][SQL][branch-1.6] nullability of ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11042#issuecomment-179078362
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50648/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13101][SQL][branch-1.6] nullability of ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11042#issuecomment-179078181
  
**[Test build #50648 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50648/consoleFull)**
 for PR 11042 at commit 
[`6b5a100`](https://github.com/apache/spark/commit/6b5a1001109f4b94b1a43518c8755de2acfaf719).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13101][SQL][branch-1.6] nullability of ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11042#issuecomment-179078359
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[yanboliang]

Github user yanboliang commented on a diff in the pull request:

https://github.com/apache/spark/pull/11044#discussion_r51690527
  
--- Diff: python/pyspark/ml/recommendation.py ---
@@ -81,6 +82,23 @@ class ALS(JavaEstimator, HasCheckpointInterval, 
HasMaxIter, HasPredictionCol, Ha
 Row(user=1, item=0, prediction=2.6258413791656494)
 >>> predictions[2]
 Row(user=2, item=0, prediction=-1.5018409490585327)
+>>> import os, tempfile
+>>> path = tempfile.mkdtemp()
+>>> ALS_path = path + "/als"
--- End diff --

nit: ```ALS_path``` -> ```als_path```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10841#issuecomment-179092907
  
**[Test build #50654 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50654/consoleFull)**
 for PR 10841 at commit 
[`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13002][Mesos] Send initial request of e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11047#issuecomment-179147515
  
**[Test build #50657 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50657/consoleFull)**
 for PR 11047 at commit 
[`1c75940`](https://github.com/apache/spark/commit/1c7594073267c8e0d4a58d7d4f6bd55df73d0316).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11044#issuecomment-179120009
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11044#issuecomment-179120015
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50655/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13002][Mesos] Send initial request of e...

[skyluc]

GitHub user skyluc opened a pull request:

https://github.com/apache/spark/pull/11047

[SPARK-13002][Mesos] Send initial request of executors for dyn allocation

Fix for [SPARK-13002](https://issues.apache.org/jira/browse/SPARK-13002) 
about the initial number of executors when running with dynamic allocation on 
Mesos.
Instead of fixing it just for the Mesos case, made the change in 
`ExecutorAllocationManager`. It is already driving the number of executors 
running on Mesos, only no the initial value.

The `None` and `Some(0)` are internal details on the computation of 
resources to reserved, in the Mesos backend scheduler. `executorLimitOption` 
has to be initialized correctly, otherwise the Mesos backend scheduler will, 
either, create to many executors at launch, or not create any executors and not 
be able to recover from this state.

Removed the 'special case' description in the doc. It was not totally 
accurate, and is not needed anymore.

This doesn't fix the same problem visible with Spark standalone. There is 
no straightforward way to send the initial value in standalone mode.

Somebody knowing this part of the yarn support should review this change.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/skyluc/spark issue/initial-dyn-alloc-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11047.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11047


commit 1c7594073267c8e0d4a58d7d4f6bd55df73d0316
Author: Luc Bourlier 
Date:   2016-01-22T14:42:21Z

Send initial request of executors for dyn allocation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL][WIP] Create native DDL comm...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11048#issuecomment-179149432
  
**[Test build #50658 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50658/consoleFull)**
 for PR 11048 at commit 
[`77252af`](https://github.com/apache/spark/commit/77252af5b83d026030c5dd1270ecd4d0788af779).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095] [Mesos] Support launching multipl...

[dragos]

Github user dragos commented on a diff in the pull request:

https://github.com/apache/spark/pull/10993#discussion_r51703515
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
 ---
@@ -245,99 +239,182 @@ private[spark] class CoarseMesosSchedulerBackend(
*/
   override def resourceOffers(d: SchedulerDriver, offers: JList[Offer]) {
 stateLock.synchronized {
-  val filters = Filters.newBuilder().setRefuseSeconds(5).build()
-  for (offer <- offers.asScala) {
+  logDebug(s"Received ${offers.size} resource offers.")
+
+  val (matchedOffers, unmatchedOffers) = offers.asScala.partition { 
offer =>
 val offerAttributes = toAttributeMap(offer.getAttributesList)
-val meetsConstraints = 
matchesAttributeRequirements(slaveOfferConstraints, offerAttributes)
+matchesAttributeRequirements(slaveOfferConstraints, 
offerAttributes)
+  }
+
+  declineUnmatchedOffers(d, unmatchedOffers)
+  handleMatchedOffers(d, matchedOffers)
+}
+  }
+
+  private def declineUnmatchedOffers(d: SchedulerDriver, offers: 
Buffer[Offer]) {
+for (offer <- offers) {
+  val id = offer.getId.getValue
+  val offerAttributes = toAttributeMap(offer.getAttributesList)
+  val mem = getResource(offer.getResourcesList, "mem")
+  val cpus = getResource(offer.getResourcesList, "cpus")
+  val filters = Filters.newBuilder()
+.setRefuseSeconds(rejectOfferDurationForUnmetConstraints).build()
+
+  logDebug(s"Declining offer: $id with attributes: $offerAttributes 
mem: $mem cpu: $cpus"
++ s" for $rejectOfferDurationForUnmetConstraints seconds")
+
+  d.declineOffer(offer.getId, filters)
+}
+  }
+
+  private def handleMatchedOffers(d: SchedulerDriver, offers: 
Buffer[Offer]) {
+val tasks = getTasks(offers)
+for (offer <- offers) {
+  val offerAttributes = toAttributeMap(offer.getAttributesList)
+  val offerMem = getResource(offer.getResourcesList, "mem")
+  val offerCpus = getResource(offer.getResourcesList, "cpus")
+  val id = offer.getId.getValue
+
+  if (tasks.contains(offer.getId)) { // accept
+val filters = Filters.newBuilder().setRefuseSeconds(5).build()
+val offerTasks = tasks(offer.getId)
+
+logDebug(s"Accepting offer: $id with attributes: $offerAttributes 
" +
+  s"mem: $offerMem cpu: $offerCpus.  Launching ${offerTasks.size} 
Mesos tasks.")
+
+for (task <- offerTasks) {
+  val taskId = task.getTaskId
+  val mem = getResource(task.getResourcesList, "mem")
+  val cpus = getResource(task.getResourcesList, "cpus")
+
+  logDebug(s"Launching Mesos task: ${taskId.getValue} with mem: 
$mem cpu: $cpus.")
+}
+
+d.launchTasks(
+  Collections.singleton(offer.getId),
+  offerTasks.asJava,
+  filters)
+  } else { // decline
+logDebug(s"Declining offer: $id with attributes: $offerAttributes 
" +
+  s"mem: $offerMem cpu: $offerCpus")
+
+d.declineOffer(offer.getId)
+  }
+}
+  }
+
+  private def getTasks(offers: Buffer[Offer]): mutable.Map[OfferID, 
List[MesosTaskInfo]] = {
+// offerID -> tasks
+val tasks = new HashMap[OfferID, 
List[MesosTaskInfo]].withDefaultValue(Nil)
+
+// offerID -> resources
+val remainingResources = HashMap[String, 
JList[Resource]](offers.map(offer =>
+  (offer.getId.getValue, offer.getResourcesList)): _*)
+
+var launchTasks = true
+
+// TODO(mgummelt): combine offers for a single slave
+// round-robin create executors on the available offers
+while (launchTasks) {
+  launchTasks = false
+
+  for (offer <- offers) {
 val slaveId = offer.getSlaveId.getValue
-val mem = getResource(offer.getResourcesList, "mem")
-val cpus = getResource(offer.getResourcesList, "cpus").toInt
-val id = offer.getId.getValue
-if (meetsConstraints) {
-  if (taskIdToSlaveId.size < executorLimit &&
-  totalCoresAcquired < maxCores &&
-  mem >= calculateTotalMemory(sc) &&
-  cpus >= 1 &&
-  failuresBySlaveId.getOrElse(slaveId, 0) < MAX_SLAVE_FAILURES 
&&
-  !slaveIdsWithExecutors.contains(slaveId)) {
-// Launch an executor on the slave
-val cpusToUse = math.min(cpus, maxCores - totalCoresAcquired)
-totalCoresAcquired += cpusToUse
-val taskId = newMesosTaskId()
-taskIdToSlaveId.put(taskId, slaveId)
-

[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11044#issuecomment-179081466
  
**[Test build #50647 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50647/consoleFull)**
 for PR 11044 at commit 
[`1ab4fd2`](https://github.com/apache/spark/commit/1ab4fd255815edeb13a18a04a6b4e34f65436281).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class ALS(JavaEstimator, HasCheckpointInterval, HasMaxIter, 
HasPredictionCol, HasRegParam, HasSeed,`
  * `class ALSModel(JavaModel, MLWritable, MLReadable):`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/10841#issuecomment-179084732
  
looks like an internal jenkins issue, jenkins retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12888][SQL][follow-up] benchmark the ne...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10917#issuecomment-179094018
  
**[Test build #50653 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50653/consoleFull)**
 for PR 10917 at commit 
[`4104d80`](https://github.com/apache/spark/commit/4104d80bb4a815fa816b5d4ee2123d13e7e53bdf).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13037][ML][PySpark] PySpark ml.recommen...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11044#issuecomment-179112303
  
**[Test build #50655 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50655/consoleFull)**
 for PR 11044 at commit 
[`a294918`](https://github.com/apache/spark/commit/a294918f5d82c4aaa296ac35a9ba85d330612e0a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Changes to support KMeans with large feature s...

[levin-royl]

Github user levin-royl commented on the pull request:

https://github.com/apache/spark/pull/10739#issuecomment-179119121
  
I wanted to know if you took a look at the code and the proposed solution 
in general. Are there any comments?

Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Changes to support KMeans with large feature s...

[levin-royl]

Github user levin-royl commented on the pull request:

https://github.com/apache/spark/pull/10739#issuecomment-179118808
  
Hi, there are indeed some similar issues I found, e.g.:

https://issues.apache.org/jira/browse/SPARK-4039
https://issues.apache.org/jira/browse/SPARK-1212
https://github.com/mesos/spark/pull/736

But the difference is that in the problem I describe reducing the 
dimensions of the problem (i.e., the feature space) to allow using dense 
vectors is not suitable. Also, the solution I implemented supports this while 
allowing full flexibility to the user --- i.e., using the default dense vector 
implementation or selecting an alternative (only when the default it is not 
desired). 

I will update the JIRA issue on this as well.

Please advise if there are any additional steps I need to do at this point.

Thanks in advance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL][WIP] Create native DDL comm...

[viirya]

GitHub user viirya opened a pull request:

https://github.com/apache/spark/pull/11048

[SPARK-13139][SQL][WIP] Create native DDL commands

JIRA: https://issues.apache.org/jira/browse/SPARK-13139

From JIRA: We currently delegate most DDLs directly to Hive, through 
NativePlaceholder in HiveQl.scala. In Spark 2.0, we want to provide native 
implementations for DDLs for both SQLContext and HiveContext.

This PR will do the first step to parse DDL commands and create logical 
commands that encapsulate them. Actual implementations still delegate to 
HiveNativeCommand now.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/viirya/spark-1 native-ddl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11048.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11048


commit 77252af5b83d026030c5dd1270ecd4d0788af779
Author: Liang-Chi Hsieh 
Date:   2016-02-03T10:00:02Z

init import: create database and create function.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL][WIP] Create native DDL comm...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11048#issuecomment-179150053
  
**[Test build #50658 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50658/consoleFull)**
 for PR 11048 at commit 
[`77252af`](https://github.com/apache/spark/commit/77252af5b83d026030c5dd1270ecd4d0788af779).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class CreateDataBase(`
  * `case class CreateFunction(`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL][WIP] Create native DDL comm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11048#issuecomment-179150071
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50658/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12798] [SQL] generated BroadcastHashJoi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10989#issuecomment-179077222
  
**[Test build #2509 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2509/consoleFull)**
 for PR 10989 at commit 
[`e0c8c65`](https://github.com/apache/spark/commit/e0c8c652b86ce9d17bcb5d629e6b55563b5c382b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13002][Mesos] Send initial request of e...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/11047#issuecomment-179150956
  
LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11046#issuecomment-179174939
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50656/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11046#issuecomment-179174625
  
**[Test build #50656 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50656/consoleFull)**
 for PR 11046 at commit 
[`45baa11`](https://github.com/apache/spark/commit/45baa112d053bd49feefe7003599afbad2272634).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11046#issuecomment-179174938
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13002][Mesos] Send initial request of e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11047#issuecomment-179190008
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50657/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13002][Mesos] Send initial request of e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11047#issuecomment-179190004
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13148] [YARN] [WIP] zero-keytab Oozie a...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11033#issuecomment-179232563
  
This now works: I've tested it by creating tokens, saving them to a file. 
pointing to in the env var then using spark-submit to bring up a cluster.

For the curious, you need 2-3 tokens: HDFS, RM, and, if Yarn timeline 
server is enabled, a timeline server delegation token.

```
Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.56.101:8020, Ident: 
(HDFS_DELEGATION_TOKEN token 263 for stevel); HDFS_DELEGATION_TOKEN token 263 
for stevel Renewer: yarn Issued: 03/02/16 13:26 Max Date: 10/02/16 13:26
Kind: RM_DELEGATION_TOKEN, Service: 192.168.56.101:8050, Ident: 
(owner=stevel@COTHAM, renewer=yarn, realUser=, issueDate=1454505993370, 
maxDate=1455110793370, sequenceNumber=5, masterKeyId=172); owner=stevel@COTHAM, 
renewer=yarn, realUser=, issueDate=1454505993370, maxDate=1455110793370, 
sequenceNumber=5, masterKeyId=172 Renewer: yarn Issued: 03/02/16 13:26 Max 
Date: 10/02/16 13:26
Kind: TIMELINE_DELEGATION_TOKEN, Service: 192.168.56.101:8188, Ident: 
(owner=stevel, renewer=yarn, realUser=, issueDate=1454505993925, 
maxDate=1455110793925, sequenceNumber=4, masterKeyId=2); owner=stevel, 
renewer=yarn, realUser=, issueDate=1454505993925, maxDate=1455110793925, 
sequenceNumber=4, masterKeyId=2 Renewer: yarn Issued: 03/02/16 13:26 Max Date: 
10/02/16 13:26
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11046#issuecomment-179225312
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11046#issuecomment-179225315
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50659/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13158][Web UI] Show the information of ...

[maropu]

Github user maropu commented on the pull request:

https://github.com/apache/spark/pull/11046#issuecomment-179110353
  
This is shown like;
https://cloud.githubusercontent.com/assets/692303/12777614/5399ed04-caa1-11e5-93cb-2012d8769cd5.png;>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10841#issuecomment-179144260
  
**[Test build #50654 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50654/consoleFull)**
 for PR 10841 at commit 
[`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13157][SQL] Support any kind of input f...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/11052#issuecomment-179447159
  
Merging this into master, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13163] [Web UI] Column width on new His...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11057#issuecomment-179461211
  
**[Test build #50679 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50679/consoleFull)**
 for PR 11057 at commit 
[`acb7b9f`](https://github.com/apache/spark/commit/acb7b9f647eb5b0f07f102b252254b05d82402ed).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3611] [Web UI] Show number of cores for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11039#issuecomment-179467856
  
**[Test build #50681 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50681/consoleFull)**
 for PR 11039 at commit 
[`ea8538c`](https://github.com/apache/spark/commit/ea8538c80a3cfbfd0ae5d4e353086dc22ace47ca).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13149][SQL]Add FileStreamSource and a s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11034#issuecomment-179474898
  
**[Test build #50684 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50684/consoleFull)**
 for PR 11034 at commit 
[`6a90c55`](https://github.com/apache/spark/commit/6a90c55beece97af10af5aa01f004ac85fda02a4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13162] Standalone mode does not respect...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11054#issuecomment-179479368
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50674/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13162] Standalone mode does not respect...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11054#issuecomment-179479365
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13146][SQL] Management API for continuo...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/11030#discussion_r51790124
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/ContinuousQueryManager.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import scala.collection.mutable
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.sql.execution.streaming.{Sink, StreamExecution}
+
+/**
+ * :: Experimental ::
+ * A class to manage all the [[org.apache.spark.sql.ContinuousQuery 
ContinuousQueries]] active
+ * on a [[SQLContext]].
+ * 
+ * @since 2.0.0
+ */
+@Experimental
+class ContinuousQueryManager {
+
+  private val activeQueries = new mutable.HashMap[String, ContinuousQuery]
+  private val activeQueriesLock = new Object
+  private val awaitTerminationLock = new Object
+
+  @volatile
+  private var lastTerminatedQuery: ContinuousQuery = null
+
+  /** Returns a list of active queries associated with this SQLContext */
+  def active: Array[ContinuousQuery] = activeQueriesLock.synchronized {
+activeQueries.values.toArray
+  }
+
+  /** Returns an active query from this SQLContext or throws exception if 
bad name */
+  def get(name: String): ContinuousQuery = activeQueriesLock.synchronized {
+activeQueries.get(name).getOrElse {
+  throw new IllegalArgumentException(s"There is no active query with 
name $name")
+}
+  }
+
+  /**
+   * Wait until any of the queries on this SQLContext is terminated, with 
or without
+   * exceptions. Returns the query that has been terminated.
+   */
+  def awaitAnyTermination(): ContinuousQuery = {
+awaitTerminationLock.synchronized {
+  lastTerminatedQuery = null
+  while (lastTerminatedQuery == null) {
+awaitTerminationLock.wait(10)
+  }
+  lastTerminatedQuery
+}
+  }
+
+  /**
+   * Wait until any of the queries on this SQLContext is terminated.
+   * Returns the stopped query if any query was terminated.
+   */
+  def awaitAnyTermination(timeoutMs: Long): Option[ContinuousQuery] = {
+val endTime = System.currentTimeMillis + timeoutMs
+def timeLeft = math.max(endTime - System.currentTimeMillis, 0)
+
+awaitTerminationLock.synchronized {
+  lastTerminatedQuery = null
+  while (timeLeft > 0 && lastTerminatedQuery == null) {
+awaitTerminationLock.wait(10)
--- End diff --

> So that if there is a race condition between one thread calling 
awaitTermination, and the sqlcontext being terminated, the waiting thread does 
not get stuff indefinitely waiting for the notifyAll to come

But `awaitTerminationLock` can guarantee no race condition like this. Right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13116][SQL] TungstenAggregate though it...

[ahshahid]

Github user ahshahid commented on the pull request:

https://github.com/apache/spark/pull/11058#issuecomment-179479583
  
Looks like code which we had contained TungstenAggregate.canProcessSafeRows 
as true. But in the latest codebase, these methods are not there.  
I have test which showed up this issue. I will check if I get the problem 
with the latest codebase of spark.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13148] [YARN] [WIP] zero-keytab Oozie a...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11033#issuecomment-179485268
  
**[Test build #50685 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50685/consoleFull)**
 for PR 11033 at commit 
[`7f20dfb`](https://github.com/apache/spark/commit/7f20dfbc952bda03603db3d5af7972fad5016e45).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13148] [YARN] [WIP] zero-keytab Oozie a...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11033#issuecomment-179485478
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50685/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13162] Standalone mode does not respect...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/11054#issuecomment-179490984
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13126] fix the right margin of history ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11029#issuecomment-179493775
  
**[Test build #50673 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50673/consoleFull)**
 for PR 11029 at commit 
[`564966d`](https://github.com/apache/spark/commit/564966d25d3749c75eea8fb6d882678b0552ff45).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13126] fix the right margin of history ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11029#issuecomment-179494059
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50673/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13149][SQL]Add FileStreamSource and a s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11034#issuecomment-179448544
  
**[Test build #50678 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50678/consoleFull)**
 for PR 11034 at commit 
[`71e6312`](https://github.com/apache/spark/commit/71e63129b675899f566d47b3367180db128b5472).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13116][SQL] TungstenAggregate though it...

[ahshahid]

Github user ahshahid commented on the pull request:

https://github.com/apache/spark/pull/11058#issuecomment-179462715
  
@davies 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12987][SQL]Fixing the name resolution i...

[kevinyu98]

Github user kevinyu98 commented on the pull request:

https://github.com/apache/spark/pull/11009#issuecomment-179478292
  
@marmbrus @cloud-fan @dilipbiswal @yzhou2001 : I have change the code based 
on Michael's comments, can you help review it again? 

Not sure why the first test failed, I run the sql test locally, it passed.

[info] Run completed in 3 minutes, 23 seconds.
[info] Total number of tests run: 1553
[info] Suites: completed 110, aborted 0
[info] Tests: succeeded 1553, failed 0, canceled 0, ignored 10, pending 0
[info] All tests passed.
[info] Passed: Total 1553, Failed 0, Errors 0, Passed 1553, Ignored 10




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13149][SQL]Add FileStreamSource

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11034#issuecomment-179481679
  
**[Test build #50678 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50678/consoleFull)**
 for PR 11034 at commit 
[`71e6312`](https://github.com/apache/spark/commit/71e63129b675899f566d47b3367180db128b5472).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13146][SQL] Management API for continuo...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/11030#discussion_r51792155
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/DataStreamWriter.scala ---
@@ -84,6 +84,17 @@ final class DataStreamWriter private[sql](df: DataFrame) 
{
   }
 
   /**
+   * Specifies a name to the [[ContinuousQuery]] to be started. This name 
must be unique among
--- End diff --

> Then its easy to write the docs of def start(name: String) "@param name 
Name of the query".

There is already a `start(path: String)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Wrap HiveClientImpl#conf with withHiveState

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11061#issuecomment-179489398
  
**[Test build #50688 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50688/consoleFull)**
 for PR 11061 at commit 
[`80f9b3f`](https://github.com/apache/spark/commit/80f9b3fdf00ec6dc2dbbe2210c124667d6b7331a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13126] fix the right margin of history ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11029#issuecomment-179494055
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12807] [YARN] Spark External Shuffle no...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10780#issuecomment-179502001
  
**[Test build #50675 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50675/consoleFull)**
 for PR 10780 at commit 
[`0ae0955`](https://github.com/apache/spark/commit/0ae0955464d2124fa3e3e634508750dfdedc8856).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13152][CORE] Fix task metrics deprecati...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11056#issuecomment-179504120
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13116][SQL] TungstenAggregate though it...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11058#issuecomment-179459249
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13152][CORE] Fix task metrics deprecati...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/11056#issuecomment-179466754
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13152][CORE] Fix task metrics deprecati...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/11056#discussion_r51786857
  
--- Diff: core/src/main/scala/org/apache/spark/executor/InputMetrics.scala 
---
@@ -81,10 +81,14 @@ class InputMetrics private (
*/
   def readMethod: DataReadMethod.Value = 
DataReadMethod.withName(_readMethod.localValue)
 
+  // Once incBytesRead & intRecordsRead is ready to be removed from the 
public API
+  // we can remove the internal versions and make the previous public API 
private.
--- End diff --

can you add that this is to suppress warnings?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12850] [SQL] Support Bucket Pruning (Pr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10942#issuecomment-179480225
  
**[Test build #50672 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50672/consoleFull)**
 for PR 10942 at commit 
[`a66e73f`](https://github.com/apache/spark/commit/a66e73f7c9ff1ce386392ce442840c1f8980ea12).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11316] setupGroups in coalescedRDD caus...

[zhuoliu]

GitHub user zhuoliu opened a pull request:

https://github.com/apache/spark/pull/11060

[SPARK-11316] setupGroups in coalescedRDD causes super long delay.

In coalescedRDD, the setupGroups causes super long delay due to the O(n^2) 
loop in the second while. That while is used to make sure that each 
PartitionGroup contains at least one partition. In some cases, this while will 
take O(n^2) to complete. If number of partitions is very large, this would take 
tens of minutes or even hours to complete.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhuoliu/spark 11316

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11060.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11060


commit e202f579c994be49b255edd4014c0e7cfebaad78
Author: zhuol 
Date:   2016-02-03T21:21:50Z

[SPARK-11316] setupGroups in coalesceRDD causes super long delay due to the 
O(n^2) loop.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13149][SQL]Add FileStreamSource

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11034#issuecomment-179488523
  
**[Test build #50687 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50687/consoleFull)**
 for PR 11034 at commit 
[`6a90c55`](https://github.com/apache/spark/commit/6a90c55beece97af10af5aa01f004ac85fda02a4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13162] Standalone mode does not respect...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11054#issuecomment-179493562
  
**[Test build #50689 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50689/consoleFull)**
 for PR 11054 at commit 
[`2a3a4f4`](https://github.com/apache/spark/commit/2a3a4f40ee869faecbbbd30c5e6a23848a15be1f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8964] [SQL] [WIP] Use Exchange to perfo...

[JoshRosen]

GitHub user JoshRosen reopened a pull request:

https://github.com/apache/spark/pull/7334

[SPARK-8964] [SQL] [WIP] Use Exchange to perform shuffle in Limit

Spark SQL's physical Limit operator currently performs its own shuffle 
rather than using Exchange to perform the shuffling. This is less efficient 
since this non-exchange shuffle path won't be able to benefit from SQL-specific 
shuffling optimizations, such as SQLSerializer2. It also involves additional 
unnecessary row copying.

Instead, I think that we should rewrite Limit to expand into three physical 
operators:

PerParititonLimit -> Exchange to one partition -> PerPartitionLimit

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JoshRosen/spark remove-copy-in-limit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/7334.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #7334


commit dfe6ff100ae2efcb1778d93bf828f9e2e8c46e18
Author: Josh Rosen 
Date:   2015-07-09T22:27:05Z

Use Exchange in Limit (SPARK-8964)

commit 089f9f5e3153009c35761cde7057af9abcea518c
Author: Josh Rosen 
Date:   2015-07-13T23:13:12Z

Merge remote-tracking branch 'origin/master' into remove-copy-in-limit

commit c02324ca36ad7a9cf8df00246d27f6fbfa61b7c0
Author: Josh Rosen 
Date:   2015-07-13T23:31:12Z

Use requiredChildDistribution in Limit

commit 70f69b63cfa561e84370fe2553d3f5570ceb488f
Author: Josh Rosen 
Date:   2015-07-13T23:58:21Z

Fix test compilation

commit 272c349394f6c53a42c621dce68fc840bcdfac16
Author: Josh Rosen 
Date:   2015-07-16T00:02:57Z

CollectLimit

commit 964838f2b5eb58da06afa18f77a056753e71defc
Author: Josh Rosen 
Date:   2015-07-16T00:55:07Z

Fix test

commit cc634566bdaed630c5bfff887f74defd003b4a58
Author: Josh Rosen 
Date:   2015-07-16T01:05:45Z

Insert ReturnAnswer before calling physical planner

commit 7dbb28e8fd49d670a3dba02b9470a6b10f8589c3
Author: Josh Rosen 
Date:   2015-07-16T01:14:43Z

Fix style

commit 9668c26d1d1ca0f56754353a90f56decd3d8e626
Author: Josh Rosen 
Date:   2015-07-16T01:50:12Z

Merge remote-tracking branch 'origin/master' into remove-copy-in-limit

commit 7202e89c7729ea200bb2d1320985e60c93cc968f
Author: Josh Rosen 
Date:   2015-07-21T01:38:38Z

Merge remote-tracking branch 'origin/master' into remove-copy-in-limit

commit fc7fe5680d81e964c7fc5a22c30aaa0dc837a854
Author: Josh Rosen 
Date:   2016-02-03T19:51:37Z

Merge remote-tracking branch 'origin/master' into remove-copy-in-limit

commit e3caa767517e6df89a058b492d8cfd9fa820a013
Author: Josh Rosen 
Date:   2016-02-03T20:04:58Z

Minor test cleanup.

commit cffe4daf218fd5c2a48b84ae831a6b40436a0f62
Author: Josh Rosen 
Date:   2016-02-03T22:03:02Z

Refactoring; split classes into separate file; fix tests.

commit 37f66887b35af510c93021e20e30d5d1c0ac696f
Author: Josh Rosen 
Date:   2016-02-03T22:06:35Z

Style fix.

commit b4de46737e393f67bac114b6343f968df6bba733
Author: Josh Rosen 
Date:   2016-02-03T22:23:27Z

Comments; order-preservation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13152][CORE] Fix task metrics deprecati...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11056#issuecomment-179506024
  
**[Test build #50692 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50692/consoleFull)**
 for PR 11056 at commit 
[`1076946`](https://github.com/apache/spark/commit/10769469acf446a42a328f6f19136eb937923ac1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13131] [SQL] Use best and average time ...

[nongli]

Github user nongli commented on the pull request:

https://github.com/apache/spark/pull/11018#issuecomment-179506296
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13163] [Web UI] Column width on new His...

[ajbozarth]

Github user ajbozarth commented on the pull request:

https://github.com/apache/spark/pull/11057#issuecomment-179455816
  
Currently the table sets each column to fit the largest data (such as App 
Name) for the whole table not just the current page:

![page_width_fixed](https://cloud.githubusercontent.com/assets/13952758/12796453/a62bc854-ca73-11e5-957c-1e76812f8a4c.png)
The culprit:

![width_long_name](https://cloud.githubusercontent.com/assets/13952758/12796473/c48cd78e-ca73-11e5-97ca-996d56bb7c6d.png)
Fix:

![column_width_fixed](https://cloud.githubusercontent.com/assets/13952758/12796486/db9af690-ca73-11e5-8112-1495f6403329.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13116][SQL] TungstenAggregate though it...

[ahshahid]

GitHub user ahshahid opened a pull request:

https://github.com/apache/spark/pull/11058

[SPARK-13116][SQL] TungstenAggregate though it is supposedly capable …

…of all processing unsafe & safe rows, fails if the input is safe rows.

Allowing targeted mutable row to be set with field values if the target row 
is Unsafe.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SnappyDataInc/spark SPARK-13116

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11058.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11058


commit 05f0626e8aeeec07776d92644eb2e8078320e436
Author: Asif Shahid 
Date:   2016-02-03T20:43:31Z

[SPARK-13116][SQL] TungstenAggregate though it is supposedly capable of all 
processing unsafe & safe rows, fails if the input is safe rows.
Allowing targeted mutable row to be set with field values if the target row 
is Unsafe.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13164][CORE][WIP] Replace deprecated sy...

[holdenk]

GitHub user holdenk opened a pull request:

https://github.com/apache/spark/pull/11059

[SPARK-13164][CORE][WIP] Replace deprecated synchronized buffer in core

Building with scala 2.11 results in the warning trait SynchronizedBuffer in 
package mutable is deprecated: Synchronization via traits is deprecated as it 
is inherently unreliable. Consider java.util.concurrent.ConcurrentLinkedQueue 
as an alternative. Investigation shows we are already using 
ConcurrentLinkedQueue in other locations so switch our uses of 
SynchronizedBuffer to ConcurrentLinkedQueue.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/spark 
SPARK-13164-replace-deprecated-synchronized-buffer-in-core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11059.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11059


commit 8d5b5b88dcecca993f2630794d29b058622f8d16
Author: Holden Karau 
Date:   2016-02-03T20:25:30Z

Move ContextCleaner away from deprecated syncrhonized buffer API

commit 2904e4552bc5c2e1347e05c9c1b724aa02dbfd0c
Author: Holden Karau 
Date:   2016-02-03T20:28:31Z

Move EventLoopSuite to ConcurrentLinkedQueue

commit 2ed2227092e4fff11b2f151cc6d880cf9d988436
Author: Holden Karau 
Date:   2016-02-03T20:48:25Z

Convert more tests in core to use concurrentlinkedqueue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13166][SQL] Remove DataStreamReader/Wri...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/11062

[SPARK-13166][SQL] Remove DataStreamReader/Writer

They seem redundant and we can simply use DataFrameReader/Writer.

The usage looks like:
```scala
val df = sqlContext.read.stream("...")
val handle = df.write.stream("...")
handle.stop()
```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark SPARK-13166

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11062.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11062


commit 6a397dcb2428fe4bff29c9e052591bce8d49f805
Author: Reynold Xin 
Date:   2016-02-03T22:16:34Z

[SPARK-13166][SQL] Remove DataStreamReader/Writer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13152][CORE] Fix task metrics deprecati...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11056#issuecomment-179503927
  
**[Test build #50677 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50677/consoleFull)**
 for PR 11056 at commit 
[`e2d518b`](https://github.com/apache/spark/commit/e2d518b03c692bb6da474a9f21a5b5065d41ddb1).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13166][SQL] Remove DataStreamReader/Wri...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/11062#discussion_r51797839
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -184,6 +185,34 @@ final class DataFrameWriter private[sql](df: 
DataFrame) {
   }
 
   /**
+   * Starts the execution of the streaming query, which will continually 
output results to the given
+   * path as new data arrives.  The returned [[ContinuousQuery]] object 
can be used to interact with
--- End diff --

extra space


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13166][SQL] Remove DataStreamReader/Wri...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/11062#discussion_r51797850
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -184,6 +185,34 @@ final class DataFrameWriter private[sql](df: 
DataFrame) {
   }
 
   /**
+   * Starts the execution of the streaming query, which will continually 
output results to the given
+   * path as new data arrives.  The returned [[ContinuousQuery]] object 
can be used to interact with
+   * the stream.
+   *
+   * @since 2.0.0
+   */
+  def stream(path: String): ContinuousQuery = {
+option("path", path).stream()
+  }
+
+  /**
+   * Starts the execution of the streaming query, which will continually 
output results to the given
+   * path as new data arrives.  The returned [[ContinuousQuery]] object 
can be used to interact with
--- End diff --

extra space.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13152][CORE] Fix task metrics deprecati...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11056#issuecomment-179504122
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50677/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12986][Doc] Fix pydoc warnings in mllib...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11025#issuecomment-179445411
  
**[Test build #50676 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50676/consoleFull)**
 for PR 11025 at commit 
[`812470d`](https://github.com/apache/spark/commit/812470d7551bb2a5f8c35a7ee86360fcdcb18222).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13163] [Web UI] Column width on new His...

[ajbozarth]

GitHub user ajbozarth opened a pull request:

https://github.com/apache/spark/pull/11057

[SPARK-13163] [Web UI] Column width on new History Server DataTables not 
getting set correctly

The column width for the new DataTables now adjusts for the current page 
rather than being hard-coded for the entire table's data.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajbozarth/spark spark13163

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11057.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11057


commit acb7b9f647eb5b0f07f102b252254b05d82402ed
Author: Alex Bozarth 
Date:   2016-02-03T20:39:01Z

Fixed column width on history server




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3611] [Web UI] Show number of cores for...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/11039#issuecomment-179462519
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13164][CORE][WIP] Replace deprecated sy...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11059#issuecomment-179468930
  
**[Test build #50682 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50682/consoleFull)**
 for PR 11059 at commit 
[`2ed2227`](https://github.com/apache/spark/commit/2ed2227092e4fff11b2f151cc6d880cf9d988436).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13152][CORE] Fix task metrics deprecati...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11056#issuecomment-179473068
  
**[Test build #50683 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50683/consoleFull)**
 for PR 11056 at commit 
[`1076946`](https://github.com/apache/spark/commit/10769469acf446a42a328f6f19136eb937923ac1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org