[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/11134#issuecomment-181999375
  
@srowen


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12807] [YARN] Spark External Shuffle no...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/10780#issuecomment-182005307
  
Merging this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12921] Fix another non-reflective TaskA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11131#issuecomment-182005376
  
**[Test build #50986 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50986/consoleFull)**
 for PR 11131 at commit 
[`8948409`](https://github.com/apache/spark/commit/89484095af50464c6f50617e7750068a4847afc7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13142] [Web UI] Problem accessing Web U...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/11135#issuecomment-182011226
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12177] [STREAMING] Update KafkaDStreams...

[markgrover]

Github user markgrover commented on a diff in the pull request:

https://github.com/apache/spark/pull/10953#discussion_r52360194
  
--- Diff: 
external/kafka/src/main/scala/org/apache/spark/streaming/kafka/NewDirectKafkaInputDStream.scala
 ---
@@ -0,0 +1,134 @@
+/*
--- End diff --

Hi Mario, I am following what Hadoop did when it was doing something 
similar. See NewHadoopRDD, for example. I am actually pretty flexible about the 
name so if folks disagree and have a better name to suggest, I am totally down 
for that. I just don't want to rename over and over again, so I'd suggest 
keeping the name the way it is right now, and get a consensus on what, if any, 
the new name should be. Once we have it here, I can change it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13237] [SQL] generated broadcast outer ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11130#issuecomment-182025801
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50982/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13237] [SQL] generated broadcast outer ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11130#issuecomment-182025607
  
**[Test build #50982 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50982/consoleFull)**
 for PR 11130 at commit 
[`9a1f532`](https://github.com/apache/spark/commit/9a1f5325e954d8464d28ebf415c9dca665e15d35).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/11134#issuecomment-182019134
  
LGTM :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13245][Core]Call shuffleMetrics methods...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11138#issuecomment-182037075
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13245][Core]Call shuffleMetrics methods...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11138#issuecomment-182037078
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50991/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13245][Core]Call shuffleMetrics methods...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11138#issuecomment-182045643
  
**[Test build #50993 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50993/consoleFull)**
 for PR 11138 at commit 
[`db0e1d2`](https://github.com/apache/spark/commit/db0e1d23ea3b045096e6dab0004d01797a0f0270).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13146][SQL] Management API for continuo...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/11030#discussion_r52369006
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/ContinuousQueryException.scala ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.sql.execution.streaming.Offset
+
+/**
+ * :: Experimental ::
+ * Exception that stopped a [[ContinuousQuery]].
+ * @paaram query  Query that caused the exception
--- End diff --

nit: p**a**aram -> param


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13146][SQL] Management API for continuo...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/11030#discussion_r52369032
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/ContinuousQueryManager.scala ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import scala.collection.mutable
+
+import org.apache.spark.annotation.Experimental
+import 
org.apache.spark.sql.execution.streaming.{ContinuousQueryListenerBus, Sink, 
StreamExecution}
+import org.apache.spark.sql.util.ContinuousQueryListener
+
+/**
+ * :: Experimental ::
+ * A class to manage all the [[org.apache.spark.sql.ContinuousQuery 
ContinuousQueries]] active
+ * on a [[SQLContext]].
+ *
+ * @since 2.0.0
+ */
+@Experimental
+class ContinuousQueryManager(sqlContext: SQLContext) {
+
+  private val listenerBus = new 
ContinuousQueryListenerBus(sqlContext.sparkContext.listenerBus)
+  private val activeQueries = new mutable.HashMap[String, ContinuousQuery]
+  private val activeQueriesLock = new Object
+  private val awaitTerminationLock = new Object
+
+  private var lastTerminatedQuery: ContinuousQuery = null
+
+  /**
+   * Returns a list of active queries associated with this SQLContext
+   *
+   * @since 2.0.0
+   */
+  def active: Array[ContinuousQuery] = activeQueriesLock.synchronized {
+activeQueries.values.toArray
+  }
+
+  /**
+   * Returns an active query from this SQLContext or throws exception if 
bad name
+   *
+   * @since 2.0.0
+   */
+  def get(name: String): ContinuousQuery = activeQueriesLock.synchronized {
+activeQueries.get(name).getOrElse {
+  throw new IllegalArgumentException(s"There is no active query with 
name $name")
+}
+  }
+
+  /**
+   * Wait until any of the queries on the associated SQLContext has 
terminated since the
+   * creation of the context, or since `clearTermination()` was called. If 
any query was terminated
+   * with an exception, then the exception will be thrown.
+   *
+   * If a query has terminated, then subsequent calls to 
`awaitAnyTermination()` will either
+   * return immediately (if the query was terminated by `query.stop()`),
+   * or throw the exception immediately (if the query was terminated with 
exception). Use
+   * `resetTerminated()` to clear past terminations and wait for new 
terminations.
+   *
+   * Note that if multiple queries have terminated
+   * @throws ContinuousQueryException, if any query has terminated with an 
exception without
+   * `timeoutMs` milliseconds.
+   *
+   * @since 2.0.0
+   */
+  def awaitAnyTermination(): Unit = {
+awaitTerminationLock.synchronized {
+  while (lastTerminatedQuery == null) {
+awaitTerminationLock.wait(10)
+  }
+  if (lastTerminatedQuery != null && 
lastTerminatedQuery.exception.nonEmpty) {
+throw lastTerminatedQuery.exception.get
+  }
+}
+  }
+
+  /**
+   * Wait until any of the queries on the associated SQLContext has 
terminated since the
+   * creation of the context, or since `clearTermination()` was called. 
Returns whether the query
+   * has terminated or not. If the query has terminated with an exception,
+   * then the exception will be thrown.
+   *
+   * If a query has terminated, then subsequent calls to 
`awaitAnyTermination()` will either
+   * return `true` immediately (if the query was terminated by 
`query.stop()`),
+   * or throw the exception immediately (if the query was terminated with 
exception). Use
+   * `resetTerminated()` to clear past terminations and wait for new 
terminations.
+   *
+   * @throws ContinuousQueryException, if any query has terminated with an 
exception
+   *
+   * @since 2.0.0
+   */
+  def awaitAnyTermination(timeoutMs: Long): Boolean = {
+val endTime = System.currentTimeMillis + timeoutMs
 

[GitHub] spark pull request: [SPARK-5095] [Mesos] Support launching multipl...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/10993#discussion_r52352874
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
 ---
@@ -260,113 +257,208 @@ private[spark] class CoarseMesosSchedulerBackend(
 offers.asScala.map(_.getId).foreach(d.declineOffer)
 return
   }
-  val filters = Filters.newBuilder().setRefuseSeconds(5).build()
-  for (offer <- offers.asScala) {
+
+  logDebug(s"Received ${offers.size} resource offers.")
+
+  val (matchedOffers, unmatchedOffers) = offers.asScala.partition { 
offer =>
 val offerAttributes = toAttributeMap(offer.getAttributesList)
-val meetsConstraints = 
matchesAttributeRequirements(slaveOfferConstraints, offerAttributes)
+matchesAttributeRequirements(slaveOfferConstraints, 
offerAttributes)
+  }
+
+  declineUnmatchedOffers(d, unmatchedOffers)
+  handleMatchedOffers(d, matchedOffers)
+}
+  }
+
+  private def declineUnmatchedOffers(d: SchedulerDriver, offers: 
Buffer[Offer]): Unit = {
+for (offer <- offers) {
+  val id = offer.getId.getValue
+  val offerAttributes = toAttributeMap(offer.getAttributesList)
+  val mem = getResource(offer.getResourcesList, "mem")
+  val cpus = getResource(offer.getResourcesList, "cpus")
+  val filters = Filters.newBuilder()
+.setRefuseSeconds(rejectOfferDurationForUnmetConstraints).build()
+
+  logDebug(s"Declining offer: $id with attributes: $offerAttributes 
mem: $mem cpu: $cpus"
++ s" for $rejectOfferDurationForUnmetConstraints seconds")
+
+  d.declineOffer(offer.getId, filters)
+}
+  }
+
+  /**
+* Launches executors on accepted offers, and declines unused offers. 
Executors are launched
+* round-robin on offers.
+*
+* @param d SchedulerDriver
+* @param offers Mesos offers that match attribute constraints
+*/
+  private def handleMatchedOffers(d: SchedulerDriver, offers: 
Buffer[Offer]): Unit = {
+val tasks = buildMesosTasks(offers)
+for (offer <- offers) {
+  val offerAttributes = toAttributeMap(offer.getAttributesList)
+  val offerMem = getResource(offer.getResourcesList, "mem")
+  val offerCpus = getResource(offer.getResourcesList, "cpus")
+  val id = offer.getId.getValue
+
+  if (tasks.contains(offer.getId)) { // accept
+val offerTasks = tasks(offer.getId)
+
+logDebug(s"Accepting offer: $id with attributes: $offerAttributes 
" +
+  s"mem: $offerMem cpu: $offerCpus.  Launching ${offerTasks.size} 
Mesos tasks.")
+
+for (task <- offerTasks) {
+  val taskId = task.getTaskId
+  val mem = getResource(task.getResourcesList, "mem")
+  val cpus = getResource(task.getResourcesList, "cpus")
+
+  logDebug(s"Launching Mesos task: ${taskId.getValue} with mem: 
$mem cpu: $cpus.")
+}
+
+d.launchTasks(
+  Collections.singleton(offer.getId),
+  offerTasks.asJava)
+  } else { // decline
+logDebug(s"Declining offer: $id with attributes: $offerAttributes 
" +
+  s"mem: $offerMem cpu: $offerCpus")
+
+d.declineOffer(offer.getId)
+  }
+}
+  }
+
+  /**
+* Returns a map from OfferIDs to the tasks to launch on those offers.  
In order to maximize
+* per-task memory and IO, tasks are round-robin assigned to offers.
+*
+* @param offers Mesos offers that match attribute constraints
+* @return A map from OfferID to a list of Mesos tasks to launch on 
that offer
+*/
+  private def buildMesosTasks(offers: Buffer[Offer]): Map[OfferID, 
List[MesosTaskInfo]] = {
+// offerID -> tasks
+val tasks = new HashMap[OfferID, 
List[MesosTaskInfo]].withDefaultValue(Nil)
+
+// offerID -> resources
+val remainingResources = mutable.Map(offers.map(offer =>
+  (offer.getId.getValue, offer.getResourcesList)): _*)
+
+var launchTasks = true
+
+// TODO(mgummelt): combine offers for a single slave
+//
+// round-robin create executors on the available offers
+while (launchTasks) {
+  launchTasks = false
+
+  for (offer <- offers) {
 val slaveId = offer.getSlaveId.getValue
-val mem = getResource(offer.getResourcesList, "mem")
-val cpus = getResource(offer.getResourcesList, "cpus").toInt
-val id = offer.getId.getValue
-if (meetsConstraints) {
-  if (taskIdToSlaveId.size < executorLimit &&
-  totalCoresAcquired < 

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10089#issuecomment-181997614
  
**[Test build #50983 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50983/consoleFull)**
 for PR 10089 at commit 
[`b58d517`](https://github.com/apache/spark/commit/b58d51767f0370c65fa65dfb15416b75fa914d05).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13176] [Core] Use native file linking i...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/11098#discussion_r52353770
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -516,7 +517,7 @@ private[spark] object Utils extends Logging {
 
 // The file does not exist in the target directory. Copy or move it 
there.
 if (removeSourceFile) {
-  Files.move(sourceFile, destFile)
+  Files.move(sourceFile.toPath, destFile.toPath)
--- End diff --

Should GFiles be used here ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11134#issuecomment-182011762
  
Yes that's fine if it passes tests and it does


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13248][STREAMING] Remove deprecated Str...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11139#issuecomment-182022752
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12921] Fix another non-reflective TaskA...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11131#issuecomment-182051807
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50986/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12921] Fix another non-reflective TaskA...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11131#issuecomment-182051799
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12921] Fix another non-reflective TaskA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11131#issuecomment-182050878
  
**[Test build #50986 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50986/consoleFull)**
 for PR 11131 at commit 
[`8948409`](https://github.com/apache/spark/commit/89484095af50464c6f50617e7750068a4847afc7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13176] [Core] Use native file linking i...

[jodersky]

Github user jodersky commented on a diff in the pull request:

https://github.com/apache/spark/pull/11098#discussion_r52355246
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -516,7 +517,7 @@ private[spark] object Utils extends Logging {
 
 // The file does not exist in the target directory. Copy or move it 
there.
 if (removeSourceFile) {
-  Files.move(sourceFile, destFile)
+  Files.move(sourceFile.toPath, destFile.toPath)
--- End diff --

I did two things in this pr
1) rename guava.common.io.Files to GFiles anf import Files
2) call Files (the native jvm interface) when possible

Since files offers the move method it can be used here. In fact the only
place that gfiles is needed is when comparing file content equality.
On Feb 9, 2016 10:47 AM, "tedyu"  wrote:

> In core/src/main/scala/org/apache/spark/util/Utils.scala
> :
>
> > @@ -516,7 +517,7 @@ private[spark] object Utils extends Logging {
> >
> >  // The file does not exist in the target directory. Copy or move 
it there.
> >  if (removeSourceFile) {
> > -  Files.move(sourceFile, destFile)
> > +  Files.move(sourceFile.toPath, destFile.toPath)
>
> Should GFiles be used here ?
>
> —
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095] [Mesos] Support launching multipl...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10993#issuecomment-182003212
  
**[Test build #50985 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50985/consoleFull)**
 for PR 10993 at commit 
[`ecad77a`](https://github.com/apache/spark/commit/ecad77a6ac85892f1155f596e84729342e484088).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Call shuffleMetrics methods only in one thread...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/11138#discussion_r52359944
  
--- Diff: 
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala 
---
@@ -106,8 +106,10 @@ final class ShuffleBlockFetcherIterator(
   /**
* Whether the iterator is still active. If isZombie is true, the 
callback interface will no
* longer place fetched blocks into [[results]].
+   *
+   * Guarded by `this`.
--- End diff --

you can use the `@GuardedBy` annotation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Call shuffleMetrics methods only in one thread...

[zsxwing]

GitHub user zsxwing opened a pull request:

https://github.com/apache/spark/pull/11138

Call shuffleMetrics methods only in one thread for 
ShuffleBlockFetcherIterator

Call shuffleMetrics's incRemoteBytesRead and incRemoteBlocksFetched when 
polling FetchResult from `results` so as to always use shuffleMetrics in one 
thread. 

Also fix a race condition that could cause memory leak.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zsxwing/spark SPARK-13245

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11138.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11138


commit 66976c26a90292cb58caa6bc440dea2998afb310
Author: Shixiong Zhu 
Date:   2016-02-09T19:10:40Z

Call shuffleMetrics methods only in one thread for 
ShuffleBlockFetcherIterator




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Call shuffleMetrics methods only in one thread...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/11138#discussion_r52360044
  
--- Diff: 
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala 
---
@@ -154,13 +164,13 @@ final class ShuffleBlockFetcherIterator(
 override def onBlockFetchSuccess(blockId: String, buf: 
ManagedBuffer): Unit = {
   // Only add the buffer to results queue if the iterator is not 
zombie,
   // i.e. cleanup() has not been called yet.
-  if (!isZombie) {
-// Increment the ref count because we need to pass this to a 
different thread.
-// This needs to be released after use.
-buf.retain()
-results.put(new SuccessFetchResult(BlockId(blockId), address, 
sizeMap(blockId), buf))
-shuffleMetrics.incRemoteBytesRead(buf.size)
-shuffleMetrics.incRemoteBlocksFetched(1)
+  ShuffleBlockFetcherIterator.this.synchronized {
--- End diff --

If we don't use a lock here, `isZombie` may become to `true` after `if 
(!isZombie) {`. Then we may put `SuccessFetchResult` into results after 
cleaning up it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13245][Core]Call shuffleMetrics methods...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11138#issuecomment-182028947
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50989/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13245][Core]Call shuffleMetrics methods...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11138#issuecomment-182028938
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...

[lresende]

Github user lresende closed the pull request at:

https://github.com/apache/spark/pull/9893


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11826][MLlib] Refactor add() and subtra...

[ehsanmok]

Github user ehsanmok commented on the pull request:

https://github.com/apache/spark/pull/9916#issuecomment-182008479
  
@jkbradley Again why it's taking too long?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13237] [SQL] generated broadcast outer ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11130#issuecomment-182025797
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13245][Core]Call shuffleMetrics methods...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11138#issuecomment-182026169
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50988/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13245][Core]Call shuffleMetrics methods...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11138#issuecomment-182026165
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13189] Cleanup build references to Scal...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11092#issuecomment-182034675
  
I agree with Sean that the bulk of these changes are not necessary, but 
this doesn't seem harmful so I'm going to merge it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12567][SQL] Add aes_{encrypt,decrypt} U...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10527#issuecomment-182034604
  
**[Test build #50990 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50990/consoleFull)**
 for PR 10527 at commit 
[`1de0453`](https://github.com/apache/spark/commit/1de0453ab64a94af97b1c65528762fcece198624).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13237] [SQL] generated broadcast outer ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11130#issuecomment-182039001
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095] [Mesos] Support launching multipl...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/10993#issuecomment-181999051
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12921] Fix another non-reflective TaskA...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11131#issuecomment-182002790
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10089#issuecomment-182002520
  
**[Test build #50983 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50983/consoleFull)**
 for PR 10089 at commit 
[`b58d517`](https://github.com/apache/spark/commit/b58d51767f0370c65fa65dfb15416b75fa914d05).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10089#issuecomment-182002605
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12177] [STREAMING] Update KafkaDStreams...

[markgrover]

Github user markgrover commented on a diff in the pull request:

https://github.com/apache/spark/pull/10953#discussion_r52360764
  
--- Diff: 
external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala 
---
@@ -610,6 +619,417 @@ object KafkaUtils {
   Set(topics.asScala.toSeq: _*)
 )
   }
+
+  // Start - Kafka functions using the new Consumer API
+
+  def addSSLOptions(
+kafkaParams: Map[String, String],
+sc: SparkContext): Map[String, String] = {
+val sparkConf = sc.getConf
+val defaultSSLOptions = SSLOptions.parse(sparkConf, "spark.ssl", None)
+val kafkaSSLOptions = SSLOptions.parse(sparkConf, "spark.ssl.kafka", 
Some(defaultSSLOptions))
+
+if (kafkaSSLOptions.enabled) {
+  val sslParams = Map[String, Option[_]](
+CommonClientConfigs.SECURITY_PROTOCOL_CONFIG -> Some("SSL"),
+SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG -> 
kafkaSSLOptions.trustStore,
+SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG -> 
kafkaSSLOptions.trustStorePassword,
+SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG -> 
kafkaSSLOptions.keyStore,
+SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG -> 
kafkaSSLOptions.keyStorePassword,
+SslConfigs.SSL_KEY_PASSWORD_CONFIG -> kafkaSSLOptions.keyPassword)
+  kafkaParams ++ 
sslParams.filter(_._2.isDefined).mapValues(_.get.toString)
+} else {
+  kafkaParams
+}
+
+  }
+
+  /** Make sure offsets are available in kafka, or throw an exception */
+  private def newCheckOffsets(
+kafkaParams: Map[String, String],
+offsetRanges: Array[OffsetRange]): Array[OffsetRange] = {
+val kc = new NewKafkaCluster(kafkaParams)
+try {
+  val topics = offsetRanges.map(_.topicPartition).toSet
+  val low = kc.getEarliestOffsets(topics)
+  val high = kc.getLatestOffsetsWithLeaders(topics)
+
+  val result = offsetRanges.filterNot { o =>
+low(o.topicPartition()) <= o.fromOffset &&
+  o.untilOffset <= high(o.topicPartition()).offset
+  }
+
+  if (!result.isEmpty) {
+throw new SparkException("Offsets not available in Kafka: " + 
result.mkString(","))
+  }
+
+  offsetRanges.map { o =>
+OffsetRange(o.topic, o.partition, o.fromOffset, o.untilOffset,
+  high(o.topicPartition()).host)
+  }
+} finally {
+  kc.close()
+}
+  }
+
+  /**
+* Create a RDD from Kafka using offset ranges for each topic and 
partition.
+*
+* @param sc SparkContext object
+* @param kafkaParams Kafka http://kafka.apache.org/documentation.html#configuration;>
+*configuration parameters. Requires 
"bootstrap.servers"
+*to be set with Kafka broker(s) (NOT zookeeper 
servers) specified in
+*host1:port1,host2:port2 form.
+* @param offsetRanges Each OffsetRange in the batch corresponds to a
+* range of offsets for a given Kafka 
topic/partition
+* @tparam K type of Kafka message key
+* @tparam V type of Kafka message value
+* @return RDD of (Kafka message key, Kafka message value)
+*/
+  def createNewRDD[K: ClassTag, V: ClassTag](
--- End diff --

Mario, see my comment below about new. As far as marking the previous ones 
as deprecated, I don't quite feel the same way. The new consumer API is brand 
new, and may take a bit to bake in (given issues like 
[KAFKA-3029](issues.apache.org/jira/browse/KAFKA-3029) and 
[KAFKA-3159](https://issues.apache.org/jira/browse/KAFKA-3159) are being 
found). It's also currently marked as beta. So, my recommendation would not be 
to deprecate the old API. It's only for those who are eager and curious to use 
the new API, and especially the associated encryption and authentication 
capabilities it offers, for now, in my opinion. Let me know if you differ 
though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13248][STREAMING] Remove deprecated Str...

[lresende]

GitHub user lresende opened a pull request:

https://github.com/apache/spark/pull/11139

[SPARK-13248][STREAMING] Remove deprecated Streaming APIs.

Remove deprecated Streaming APIs and adjust sample applications.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/spark streaming-deprecated-apis

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11139.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11139


commit 9902d064a888603159e93f6fe097100513725e85
Author: Luciano Resende 
Date:   2016-02-09T19:27:36Z

[SPARK-13248][STREAMING] Remove deprecated Streaming APIs.

Remove deprecated Streaming APIs and adjust sample applications.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12177] [STREAMING] Update KafkaDStreams...

[markgrover]

Github user markgrover commented on a diff in the pull request:

https://github.com/apache/spark/pull/10953#discussion_r52361163
  
--- Diff: 
external/kafka/src/main/scala/org/apache/spark/streaming/kafka/NewKafkaCluster.scala
 ---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.kafka
+
+import java.util
+import java.util.Collections
+
+import scala.collection.JavaConverters._
+import scala.reflect._
+
+import org.apache.kafka.clients.consumer.{KafkaConsumer, 
OffsetAndMetadata, OffsetResetStrategy}
+import org.apache.kafka.common.{PartitionInfo, TopicPartition}
+
+import org.apache.spark.SparkException
+
+/**
+ * @param kafkaParams Kafka http://kafka.apache.org/documentation.html#configuration;>
+ *configuration parameters.
+ *Requires "bootstrap.servers" to be set with Kafka 
broker(s),
+ *NOT zookeeper servers, specified in 
host1:port1,host2:port2 form
+ */
+private[spark]
+class NewKafkaCluster[K: ClassTag, V: ClassTag](val kafkaParams: 
Map[String, String])
--- End diff --

Thanks, it did conflict with this PR but I have merged it. I will make the 
NewKafkaCluster class public as well, similar to this change now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13237] [SQL] generated broadcast outer ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11130#issuecomment-182039006
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50992/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13146][SQL] Management API for continuo...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/11030#discussion_r52369023
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/ContinuousQueryManager.scala ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import scala.collection.mutable
+
+import org.apache.spark.annotation.Experimental
+import 
org.apache.spark.sql.execution.streaming.{ContinuousQueryListenerBus, Sink, 
StreamExecution}
+import org.apache.spark.sql.util.ContinuousQueryListener
+
+/**
+ * :: Experimental ::
+ * A class to manage all the [[org.apache.spark.sql.ContinuousQuery 
ContinuousQueries]] active
+ * on a [[SQLContext]].
+ *
+ * @since 2.0.0
+ */
+@Experimental
+class ContinuousQueryManager(sqlContext: SQLContext) {
+
+  private val listenerBus = new 
ContinuousQueryListenerBus(sqlContext.sparkContext.listenerBus)
+  private val activeQueries = new mutable.HashMap[String, ContinuousQuery]
+  private val activeQueriesLock = new Object
+  private val awaitTerminationLock = new Object
+
+  private var lastTerminatedQuery: ContinuousQuery = null
+
+  /**
+   * Returns a list of active queries associated with this SQLContext
+   *
+   * @since 2.0.0
+   */
+  def active: Array[ContinuousQuery] = activeQueriesLock.synchronized {
+activeQueries.values.toArray
+  }
+
+  /**
+   * Returns an active query from this SQLContext or throws exception if 
bad name
+   *
+   * @since 2.0.0
+   */
+  def get(name: String): ContinuousQuery = activeQueriesLock.synchronized {
+activeQueries.get(name).getOrElse {
+  throw new IllegalArgumentException(s"There is no active query with 
name $name")
+}
+  }
+
+  /**
+   * Wait until any of the queries on the associated SQLContext has 
terminated since the
+   * creation of the context, or since `clearTermination()` was called. If 
any query was terminated
+   * with an exception, then the exception will be thrown.
+   *
+   * If a query has terminated, then subsequent calls to 
`awaitAnyTermination()` will either
+   * return immediately (if the query was terminated by `query.stop()`),
+   * or throw the exception immediately (if the query was terminated with 
exception). Use
+   * `resetTerminated()` to clear past terminations and wait for new 
terminations.
+   *
+   * Note that if multiple queries have terminated
--- End diff --

nit: missing more doc


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13146][SQL] Management API for continuo...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/11030#discussion_r52368999
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/ContinuousQuery.scala ---
@@ -17,14 +17,84 @@
 
 package org.apache.spark.sql
 
+import org.apache.spark.annotation.Experimental
+
 /**
+ * :: Experimental ::
  * A handle to a query that is executing continuously in the background as 
new data arrives.
+ * All these methods are thread-safe.
+ * @since 2.0.0
  */
+@Experimental
 trait ContinuousQuery {
 
   /**
-   * Stops the execution of this query if it is running.  This method 
blocks until the threads
+   * Returns the name of the query.
+   * @since 2.0.0
+   */
+  def name: String
+
+  /**
+   * Returns the SQLContext associated with `this` query
+   * @since 2.0.0
+   */
+  def sqlContext: SQLContext
+
+  /**
+   * Whether the query is currently active or not
+   * @since 2.0.0
+   */
+  def isActive: Boolean
+
+  /**
+   * Returns the [[ContinuousQueryException]] if the query was terminated 
by an exception.
+   * @since 2.0.0
+   */
+  def exception: Option[ContinuousQueryException]
+
+  /**
+   * Returns current status of all the sources.
+   * @since 2.0.0
+   */
+   def sourceStatuses: Array[SourceStatus]
+
+  /** Returns current status of the sink. */
+  def sinkStatus: SinkStatus
+
+  /**
+   * Waits for the termination of `this` query, either by `query.stop()` 
or by an exception.
+   * If the query has terminated with an exception, then the exception 
will be thrown.
+   *
+   * If the query has terminated, then all subsequent calls to this method 
will either return
+   * `true` immediately (if the query was terminated by `stop()`), or 
throw the exception
--- End diff --

nit: this method doesn't return a Boolean.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13237] [SQL] generated broadcast outer ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11130#issuecomment-181752864
  
**[Test build #50961 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50961/consoleFull)**
 for PR 11130 at commit 
[`52efe91`](https://github.com/apache/spark/commit/52efe91168a4be7ce721d2f56e2b1e7aab9379db).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13154][PYTHON] Add linting for pydocs

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/11109#issuecomment-181753116
  
Also cc @davies & @MechCoder who have done some work on this file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13012] [Documentation] Replace example ...

[devaraj-kavali]

Github user devaraj-kavali commented on the pull request:

https://github.com/apache/spark/pull/11053#issuecomment-181752685
  
I am sorry @yinxusen for making you to give the same comment gain, I have 
fixed the indent issue and moved the files from mllib to ml package.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13237] [SQL] generated broadcast outer ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11130#issuecomment-181752684
  
**[Test build #50961 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50961/consoleFull)**
 for PR 11130 at commit 
[`52efe91`](https://github.com/apache/spark/commit/52efe91168a4be7ce721d2f56e2b1e7aab9379db).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13237] [SQL] generated broadcast outer ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11130#issuecomment-181752866
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13237] [SQL] generated broadcast outer ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11130#issuecomment-181752868
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50961/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12976][SQL] Add LazilyGenerateOrdering ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10894#issuecomment-181753360
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...

[hvanhovell]

Github user hvanhovell commented on the pull request:

https://github.com/apache/spark/pull/11051#issuecomment-181756011
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13040][Docs] Update JDBC deprecated SPA...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10948


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL][WIP] Create native DDL comm...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11048#issuecomment-181774944
  
**[Test build #50965 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50965/consoleFull)**
 for PR 11048 at commit 
[`3db2e1d`](https://github.com/apache/spark/commit/3db2e1d867b5750eb157c751a303c542bdad110e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13165][STREAMING] Replace deprecated sy...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11067


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13086][Shell] Use the Scala REPL settin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10984


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13170] [STREAMING] Investigate replacin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-181774605
  
**[Test build #50964 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50964/consoleFull)**
 for PR 1 at commit 
[`75bafe6`](https://github.com/apache/spark/commit/75bafe6e8937cdd95ac943649c120a7c483e028e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9307] [CORE] Logging: Make it either st...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11103#issuecomment-181774486
  
**[Test build #50963 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50963/consoleFull)**
 for PR 11103 at commit 
[`a489a0b`](https://github.com/apache/spark/commit/a489a0b23959c611c94b2653046670160f2c4429).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10427#issuecomment-181786940
  
**[Test build #50968 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50968/consoleFull)**
 for PR 10427 at commit 
[`7038bc0`](https://github.com/apache/spark/commit/7038bc05b9887eac7d5d61eac179124e6349abc2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11691][SQL] Allow to specify compressio...

[maropu]

Github user maropu commented on the pull request:

https://github.com/apache/spark/pull/9657#issuecomment-181786960
  
@zjffdu ping


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[dbtsai]

Github user dbtsai commented on the pull request:

https://github.com/apache/spark/pull/9#issuecomment-181766454
  
Agree, for code-gen, if we want to do it in this way, we would rather put 
them in a separate place. But will be nice to extend the code-gen framework so 
it can use one codebase to handle generic type. 

+@jkbradley @mengxr  BTW, we still need to run the separate `sbt` code to 
do code-gen, and why don't we do it in the compile time using quasiquote? This 
will not hurt the performance since it's compile time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13177][Examples]Update ActorWordCount e...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/3#issuecomment-181765818
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13086][Shell] Use the Scala REPL settin...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10984#issuecomment-181769822
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/9#discussion_r52282168
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala 
---
@@ -248,6 +269,11 @@ class KMeans @Since("1.5.0") (
   .setMaxIterations($(maxIter))
   .setSeed($(seed))
   .setEpsilon($(tol))
+
+if (isSet(initialModel)) {
+  algo.setInitialModel($(initialModel).parentModel)
--- End diff --

does MLlibKMeans check if the dimensionalities match? Also, why do you set 
it to parentModel?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/9#discussion_r52282317
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala 
---
@@ -248,6 +269,11 @@ class KMeans @Since("1.5.0") (
   .setMaxIterations($(maxIter))
   .setSeed($(seed))
   .setEpsilon($(tol))
+
+if (isSet(initialModel)) {
+  algo.setInitialModel($(initialModel).parentModel)
+}
+
 val parentModel = algo.run(rdd)
 val model = new KMeansModel(uid, parentModel)
 copyValues(model)
--- End diff --

The final returned model will have param of initialModel. Can this model be 
saved and loaded without any issue?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6761][SQL] Approximate quantile for Dat...

[viirya]

GitHub user viirya reopened a pull request:

https://github.com/apache/spark/pull/6042

[SPARK-6761][SQL] Approximate quantile for DataFrame

JIRA: https://issues.apache.org/jira/browse/SPARK-6761

Compute approximate quantile based on the paper Greenwald, Michael and 
Khanna, Sanjeev, "Space-efficient Online Computation of Quantile Summaries," 
SIGMOD '01.
  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/viirya/spark-1 approximate_quantile

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/6042.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6042


commit 10865378c3aba5e639c352bded61a616933a5f1c
Author: Liang-Chi Hsieh 
Date:   2015-05-10T11:19:20Z

Add support for calculating approximate quantile.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6761][SQL] Aggregation function for app...

[viirya]

Github user viirya closed the pull request at:

https://github.com/apache/spark/pull/8459


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6761][SQL] Aggregation function for app...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/8459#issuecomment-181772901
  
Reopen #6042 and close this one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13074][Core] Add JavaSparkContext. getP...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10978#issuecomment-181776497
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50959/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12995][GraphX] Remove deprecate APIs fr...

[maropu]

Github user maropu commented on the pull request:

https://github.com/apache/spark/pull/10918#issuecomment-181782798
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12995][GraphX] Remove deprecate APIs fr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10918#issuecomment-181784627
  
**[Test build #50967 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50967/consoleFull)**
 for PR 10918 at commit 
[`12d368a`](https://github.com/apache/spark/commit/12d368a71dbbeb62d3b55a8d04272d7309b4531c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13057][SQL] Add benchmark codes and the...

[maropu]

Github user maropu commented on the pull request:

https://github.com/apache/spark/pull/10965#issuecomment-181784411
  
@nongli Okay, I'll let you know the plan first. plz give me some time to 
look around similar codes in `Parquet` and `Orc`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13170] [STREAMING] Investigate replacin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13176] [Core] Use native file linking i...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11098#issuecomment-181762754
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13165][STREAMING] Replace deprecated sy...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11067#issuecomment-181763572
  
merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13176] [Core] Use native file linking i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11098


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13154][PYTHON] Add linting for pydocs

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/11109#issuecomment-181771884
  
cc @JoshRosen 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6761][SQL] Approximate quantile for Dat...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/6042#issuecomment-181776143
  
ping @thunterdb 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10427#issuecomment-181786010
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50966/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10427#issuecomment-181786009
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13170] [STREAMING] Investigate replacin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1#issuecomment-181800060
  
**[Test build #50964 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50964/consoleFull)**
 for PR 1 at commit 
[`75bafe6`](https://github.com/apache/spark/commit/75bafe6e8937cdd95ac943649c120a7c483e028e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10427#issuecomment-181817385
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50968/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10427#issuecomment-181817384
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10427#issuecomment-181817281
  
**[Test build #50968 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50968/consoleFull)**
 for PR 10427 at commit 
[`7038bc0`](https://github.com/apache/spark/commit/7038bc05b9887eac7d5d61eac179124e6349abc2).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL][WIP] Create native DDL comm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11048#issuecomment-181819018
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL][WIP] Create native DDL comm...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11048#issuecomment-181818773
  
**[Test build #50965 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50965/consoleFull)**
 for PR 11048 at commit 
[`3db2e1d`](https://github.com/apache/spark/commit/3db2e1d867b5750eb157c751a303c542bdad110e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13139][SQL][WIP] Create native DDL comm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11048#issuecomment-181819020
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50965/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12921] Fix another non-reflective TaskA...

[JoshRosen]

GitHub user JoshRosen opened a pull request:

https://github.com/apache/spark/pull/11131

[SPARK-12921] Fix another non-reflective TaskAttemptContext access in 
SpecificParquetRecordReaderBase

This is a minor followup to #10843 to fix one remaining place where we 
forgot to use reflective access of TaskAttemptContext methods.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JoshRosen/spark SPARK-12921-take-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11131.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11131


commit 89484095af50464c6f50617e7750068a4847afc7
Author: Josh Rosen 
Date:   2016-02-09T07:40:30Z

[SPARK-12921] Fix another non-reflective TaskAttemptContext access in 
SpecificParquetRecordReaderBase




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12921] Fix another non-reflective TaskA...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11131#issuecomment-181758152
  
**[Test build #50962 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50962/consoleFull)**
 for PR 11131 at commit 
[`8948409`](https://github.com/apache/spark/commit/89484095af50464c6f50617e7750068a4847afc7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13201][SPARK-13200] Deprecation warning...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/2


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13040][Docs] Update JDBC deprecated SPA...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10948#issuecomment-181764836
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10780][ML][WIP] Add initial model to km...

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/9#discussion_r52281022
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala 
---
@@ -237,6 +237,27 @@ class KMeans @Since("1.5.0") (
   @Since("1.5.0")
   def setSeed(value: Long): this.type = set(seed, value)
 
+  /** @group setParam */
+  @Since("2.0.0")
+  def setInitialModel(value: KMeansModel): this.type = set(initialModel, 
value)
+
+  /** @group setParam */
+  @Since("2.0.0")
+  def setInitialModel(value: Model[_]): this.type = {
+value match {
+  case m: KMeansModel => set(initialModel, m)
+  case other =>
+logInfo(s"KMeansModel required but ${other.getClass.getSimpleName} 
found.")
--- End diff --

Let's do warning or error. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13074][Core] Add JavaSparkContext. getP...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10978#issuecomment-181776124
  
**[Test build #50959 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50959/consoleFull)**
 for PR 10978 at commit 
[`a4ae59f`](https://github.com/apache/spark/commit/a4ae59fb3939c5c79288264f077d1020ba98aa13).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13074][Core] Add JavaSparkContext. getP...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10978#issuecomment-181780428
  
OK, I'm convinced. I'll leave it open for a bit to see if anyone wants to 
comment on the change, but seems like a straightforward 'port' to the Java API.

There are still tiny changes like the colon spacing and 
`mapValues(JavaRDD.fromRDD)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13016] [Documentation] Replace example ...

[devaraj-kavali]

GitHub user devaraj-kavali opened a pull request:

https://github.com/apache/spark/pull/11132

[SPARK-13016] [Documentation] Replace example code in 
mllib-dimensionality-reduction.md using include_example

Replaced example example code in mllib-dimensionality-reduction.md using
include_example

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/devaraj-kavali/spark SPARK-13016

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11132


commit 488f09c06d6623920c0cf841af847a47ccb625c2
Author: Devaraj K 
Date:   2016-02-09T09:53:23Z

[SPARK-13016] [Documentation] Replace example code in
mllib-dimensionality-reduction.md using include_example

Replaced example example code in mllib-dimensionality-reduction.md using
include_example




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11565 Replace deprecated DigestUtils.sha...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9532#issuecomment-181824962
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13189] Cleanup build references to Scal...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11092#issuecomment-181824581
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12976][SQL] Add LazilyGenerateOrdering ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10894#issuecomment-181753362
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50957/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org