[GitHub] spark pull request: [SPARK-8975][Streaming] Adds a mechanism to se...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/7471#discussion_r35226421
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverTrackerSuite.scala
---
@@ -72,15 +77,50 @@ class ReceiverTrackerSuite extends TestSuiteBase {
assert(locations(0).length === 1)
assert(locations(3).length === 1)
}
+
+ test(Receiver tracker - propagates rate limit) {
+val newRateLimit = 100L
+val ids = new TestReceiverInputDStream(ssc)
+val tracker = new ReceiverTracker(ssc)
+tracker.start()
+eventually(timeout(5 seconds)) {
+ assert(TestDummyReceiver.started)
+}
+tracker.sendRateUpdate(ids.id, newRateLimit)
+// this is an async message, we need to wait a bit for it to be
processed
+eventually(timeout(3 seconds)) {
+ assert(ids.getCurrentRateLimit.get === newRateLimit)
+}
+ }
}
+/** An input DStream with a hard-coded receiver that gives access to
internals for testing. */
+private class TestReceiverInputDStream(@transient ssc_ : StreamingContext)
+ extends ReceiverInputDStream[Int](ssc_) {
+
+ override def getReceiver(): DummyReceiver = TestDummyReceiver
+
+ def getCurrentRateLimit: Option[Long] = {
+TestDummyReceiver.executor.getCurrentRateLimit
+ }
+}
+
+/**
+ * We need the receiver to be an object, otherwise serialization will
create another one
+ * and we won't be able to read its rate limit.
+ */
+private object TestDummyReceiver extends DummyReceiver
--- End diff --
Then let's call it SingletonReceiver. I find using test and dummy both in
the name very superfluous
On Wednesday, July 22, 2015, Iulian Dragos notificati...@github.com wrote:
In
streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverTrackerSuite.scala
https://github.com/apache/spark/pull/7471#discussion_r35208599:
+/** An input DStream with a hard-coded receiver that gives access to
internals for testing. */
+private class TestReceiverInputDStream(@transient ssc_ :
StreamingContext)
+ extends ReceiverInputDStream[Int](ssc_) {
+
+ override def getReceiver(): DummyReceiver = TestDummyReceiver
+
+ def getCurrentRateLimit: Option[Long] = {
+TestDummyReceiver.executor.getCurrentRateLimit
+ }
+}
+
+/**
+ * We need the receiver to be an object, otherwise serialization will
create another one
+ * and we won't be able to read its rate limit.
+ */
+private object TestDummyReceiver extends DummyReceiver
Is your issue only about naming? I personally find it weird to have the
companion object of a class extend the class itself. It's true that I
*could* do that, but I prefer not to. If you feel strongly about it I can
do the renaming you wish, but I find it unnecessary.
â
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/7471/files#r35208599.
--
Sent from Gmail Mobile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8975][Streaming] Adds a mechanism to se...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/7471#discussion_r35226471
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverTrackerSuite.scala
---
@@ -72,15 +77,50 @@ class ReceiverTrackerSuite extends TestSuiteBase {
assert(locations(0).length === 1)
assert(locations(3).length === 1)
}
+
+ test(Receiver tracker - propagates rate limit) {
+val newRateLimit = 100L
+val ids = new TestReceiverInputDStream(ssc)
+val tracker = new ReceiverTracker(ssc)
+tracker.start()
+eventually(timeout(5 seconds)) {
+ assert(TestDummyReceiver.started)
+}
+tracker.sendRateUpdate(ids.id, newRateLimit)
+// this is an async message, we need to wait a bit for it to be
processed
+eventually(timeout(3 seconds)) {
+ assert(ids.getCurrentRateLimit.get === newRateLimit)
+}
+ }
}
+/** An input DStream with a hard-coded receiver that gives access to
internals for testing. */
+private class TestReceiverInputDStream(@transient ssc_ : StreamingContext)
+ extends ReceiverInputDStream[Int](ssc_) {
+
+ override def getReceiver(): DummyReceiver = TestDummyReceiver
+
+ def getCurrentRateLimit: Option[Long] = {
+TestDummyReceiver.executor.getCurrentRateLimit
+ }
+}
+
+/**
+ * We need the receiver to be an object, otherwise serialization will
create another one
+ * and we won't be able to read its rate limit.
+ */
+private object TestDummyReceiver extends DummyReceiver
--- End diff --
Then let's call it SingletonReceiver. I find using test and dummy both in
the name very superfl
On Wednesday, July 22, 2015, Iulian Dragos notificati...@github.com
javascript:_e(%7B%7D,'cvml','notificati...@github.com'); wrote:
In
streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverTrackerSuite.scala
https://github.com/apache/spark/pull/7471#discussion_r35208599:
+/** An input DStream with a hard-coded receiver that gives access to
internals for testing. */
+private class TestReceiverInputDStream(@transient ssc_ :
StreamingContext)
+ extends ReceiverInputDStream[Int](ssc_) {
+
+ override def getReceiver(): DummyReceiver = TestDummyReceiver
+
+ def getCurrentRateLimit: Option[Long] = {
+TestDummyReceiver.executor.getCurrentRateLimit
+ }
+}
+
+/**
+ * We need the receiver to be an object, otherwise serialization will
create another one
+ * and we won't be able to read its rate limit.
+ */
+private object TestDummyReceiver extends DummyReceiver
Is your issue only about naming? I personally find it weird to have the
companion object of a class extend the class itself. It's true that I
*could* do that, but I prefer not to. If you feel strongly about it I can
do the renaming you wish, but I find it unnecessary.
â
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/7471/files#r35208599.
--
Sent from Gmail Mobile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9067][SQL] Close reader in NewHadoopRDD...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7424#issuecomment-123768234 LGTM. @rxin could you take a final look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9192][SQL] add initialization phase for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7535#issuecomment-123774141 [Test build #38087 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38087/console) for PR 7535 at commit [`4dbed66`](https://github.com/apache/spark/commit/4dbed665182cb56a6c932c98e4f65b2d887f7372). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9192][SQL] add initialization phase for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7535#issuecomment-123773396 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9192][SQL] add initialization phase for...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/7535#discussion_r35230482
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
---
@@ -38,99 +38,105 @@ trait CheckAnalysis {
throw new AnalysisException(msg)
}
- def containsMultipleGenerators(exprs: Seq[Expression]): Boolean = {
+ protected def containsMultipleGenerators(exprs: Seq[Expression]):
Boolean = {
exprs.flatMap(_.collect {
- case e: Generator = true
-}).nonEmpty
+ case e: Generator = e
+}).length 1
}
def checkAnalysis(plan: LogicalPlan): Unit = {
// We transform up and order the rules so as to catch the first
possible failure instead
// of the result of cascading resolution failures.
-plan.foreachUp {
-
- case operator: LogicalPlan =
-operator transformExpressionsUp {
- case a: Attribute if !a.resolved =
-val from = operator.inputSet.map(_.name).mkString(, )
-a.failAnalysis(scannot resolve '${a.prettyString}' given
input columns $from)
-
- case e: Expression if e.checkInputDataTypes().isFailure =
-e.checkInputDataTypes() match {
- case TypeCheckResult.TypeCheckFailure(message) =
-e.failAnalysis(
- scannot resolve '${e.prettyString}' due to data type
mismatch: $message)
-}
-
- case c: Cast if !c.resolved =
-failAnalysis(
- sinvalid cast from ${c.child.dataType.simpleString} to
${c.dataType.simpleString})
-
- case WindowExpression(UnresolvedWindowFunction(name, _), _) =
-failAnalysis(
- sCould not resolve window function '$name'. +
- Note that, using window functions currently requires a
HiveContext)
-
- case w @ WindowExpression(windowFunction, windowSpec) if
windowSpec.validate.nonEmpty =
-// The window spec is not valid.
-val reason = windowSpec.validate.get
-failAnalysis(sWindow specification $windowSpec is not valid
because $reason)
-}
-
-operator match {
- case f: Filter if f.condition.dataType != BooleanType =
-failAnalysis(
- sfilter expression '${f.condition.prettyString}' +
-sof type ${f.condition.dataType.simpleString} is not a
boolean.)
-
- case Aggregate(groupingExprs, aggregateExprs, child) =
-def checkValidAggregateExpression(expr: Expression): Unit =
expr match {
- case _: AggregateExpression = // OK
- case e: Attribute if
!groupingExprs.exists(_.semanticEquals(e)) =
-failAnalysis(
- sexpression '${e.prettyString}' is neither present in
the group by, +
-snor is it an aggregate function. +
-Add to group by or wrap in first() if you don't care
which value you get.)
- case e if groupingExprs.exists(_.semanticEquals(e)) = // OK
- case e if e.references.isEmpty = // OK
- case e = e.children.foreach(checkValidAggregateExpression)
-}
-
-aggregateExprs.foreach(checkValidAggregateExpression)
-
- case _ = // Fallbacks to the following checks
-}
-
-operator match {
- case o if o.children.nonEmpty o.missingInput.nonEmpty =
-val missingAttributes = o.missingInput.mkString(,)
-val input = o.inputSet.mkString(,)
-
-failAnalysis(
- sresolved attribute(s) $missingAttributes missing from
$input +
-sin operator ${operator.simpleString})
-
- case p @ Project(exprs, _) if containsMultipleGenerators(exprs)
=
-failAnalysis(
- sOnly a single table generating function is allowed in a
SELECT clause, found:
- |
${exprs.map(_.prettyString).mkString(,)}.stripMargin)
-
- // Special handling for cases when self-join introduce duplicate
expression ids.
- case j @ Join(left, right, _, _) if
left.outputSet.intersect(right.outputSet).nonEmpty =
-val conflictingAttributes =
left.outputSet.intersect(right.outputSet)
-failAnalysis(
- s
- |Failure when resolving conflicting references in Join:
- |$plan
- |Conflicting attributes:
${conflictingAttributes.mkString(,)}
- |.stripMargin)
-
- case o if !o.resolved =
-failAnalysis(
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123773927 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9192][SQL] add initialization phase for...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/7535#discussion_r35230387
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
---
@@ -38,99 +38,105 @@ trait CheckAnalysis {
throw new AnalysisException(msg)
}
- def containsMultipleGenerators(exprs: Seq[Expression]): Boolean = {
+ protected def containsMultipleGenerators(exprs: Seq[Expression]):
Boolean = {
exprs.flatMap(_.collect {
- case e: Generator = true
-}).nonEmpty
+ case e: Generator = e
+}).length 1
}
def checkAnalysis(plan: LogicalPlan): Unit = {
// We transform up and order the rules so as to catch the first
possible failure instead
// of the result of cascading resolution failures.
-plan.foreachUp {
-
- case operator: LogicalPlan =
--- End diff --
`case operator: LogicalPlan` catches all cases, so why not just use a
normal function here instead of pattern match?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123775036 I think most of the changes are fine. The `!` ones look a little awkward and I want to check the `prev@func` issue to see why lint-r is complaining about that. cc @sun-rui --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123775245 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123776921 [Test build #38088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38088/consoleFull) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123781965 [Test build #38089 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38089/console) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123781936 [Test build #38089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38089/consoleFull) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123781822 LGTM. Thanks for the fix. I am merging it to master. If everything is good, I will merge it to 1.4 branch as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123781906 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/7276#discussion_r35225996
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
---
@@ -354,46 +503,22 @@ class ReceiverTracker(ssc: StreamingContext,
skipReceiverLaunch: Boolean = false
rcvr
})
- val checkpointDirOption = Option(ssc.checkpointDir)
- val serializableHadoopConf =
-new SerializableConfiguration(ssc.sparkContext.hadoopConfiguration)
-
- // Function to start the receiver on the worker node
- val startReceiver = (iterator: Iterator[Receiver[_]]) = {
-if (!iterator.hasNext) {
- throw new SparkException(
-Could not start receiver as object not found.)
-}
-val receiver = iterator.next()
-val supervisor = new ReceiverSupervisorImpl(
- receiver, SparkEnv.get, serializableHadoopConf.value,
checkpointDirOption)
-supervisor.start()
-supervisor.awaitTermination()
- }
-
// Run the dummy Spark job to ensure that all slaves have registered.
// This avoids all the receivers to be scheduled on the same node.
if (!ssc.sparkContext.isLocal) {
ssc.sparkContext.makeRDD(1 to 50, 50).map(x = (x,
1)).reduceByKey(_ + _, 20).collect()
}
- // Get the list of executors and schedule receivers
- val executors = getExecutors(ssc)
- val tempRDD =
-if (!executors.isEmpty) {
- val locations = scheduleReceivers(receivers, executors)
- val roundRobinReceivers = (0 until receivers.length).map(i =
-(receivers(i), locations(i)))
- ssc.sc.makeRDD[Receiver[_]](roundRobinReceivers)
-} else {
- ssc.sc.makeRDD(receivers, receivers.size)
-}
-
// Distribute the receivers and start them
logInfo(Starting + receivers.length + receivers)
running = true
+
try {
-ssc.sparkContext.runJob(tempRDD,
ssc.sparkContext.clean(startReceiver))
+for (receiver - receivers) {
+ endpoint.send(StartReceiver(receiver))
+}
+// Wait until all receivers exit
+receiverJobExitLatch.await()
logInfo(All of the receivers have been terminated)
} finally {
running = false
--- End diff --
Since we don't need to block, the thread is not necessary. Then we can also
remove `ReceiverLauncher`. Therefore, I removed `ReceiverLauncher` and added a
method `launchReceivers` to replace it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123769834 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123768179 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9192][SQL] add initialization phase for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7535#issuecomment-123774128 [Test build #38087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38087/consoleFull) for PR 7535 at commit [`4dbed66`](https://github.com/apache/spark/commit/4dbed665182cb56a6c932c98e4f65b2d887f7372). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9192][SQL] add initialization phase for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7535#issuecomment-123774145 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123780351 jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123780334 yeah, the jar file is truly corrupted. let's retrigger the build and see what happens, and i'll poke around and see if i can't get to the bottom of what happened. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123781967 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123781875 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123784786 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123784916 [Test build #62 has started](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/62/consoleFull) for PR 7587 at commit [`c8e4ea7`](https://github.com/apache/spark/commit/c8e4ea71d1372589a7f6bb6a96ed9074067f2055). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123785455 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123785391 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123787430 [Test build #38090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38090/consoleFull) for PR 7587 at commit [`c8e4ea7`](https://github.com/apache/spark/commit/c8e4ea71d1372589a7f6bb6a96ed9074067f2055). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] sbt-launch-lib.bash shoul...
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/7597 [SPARK-9254] [BUILD] sbt-launch-lib.bash should support HTTP/HTTPS redirection Target file(s) can be hosted on CDN nodes. HTTP/HTTPS redirection must be supported to download these files. You can merge this pull request into a Git repository by running: $ git pull https://github.com/liancheng/spark spark-9254 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7597.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7597 commit a7cbfb32e639c35843d472ce86bb2d17d04ec563 Author: Cheng Lian l...@databricks.com Date: 2015-07-22T15:15:10Z Supports HTTP/HTTPS redirection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9253][EC2] Allow to create machines wit...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7596#issuecomment-123755422 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123765692 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123769830 [Test build #38086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38086/console) for PR 7586 at commit [`ce0038a`](https://github.com/apache/spark/commit/ce0038acc0527495b94a19534ebd5c2b83a9ee19). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9192][SQL] add initialization phase for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7535#issuecomment-123773475 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123776950 [Test build #38088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38088/console) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8975][Streaming] Adds a mechanism to se...
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/7471#issuecomment-123776768 Spurious failure: Building Spark [info] Building Spark (w/Hive 0.13.1) using SBT with these arguments: -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive-thriftserver -Phive package assembly/assembly streaming-kafka-assembly/assembly streaming-flume-assembly/assembly Using /usr/java/latest as default JAVA_HOME. Note, this will be overridden by -java-home if it is set. **Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar** --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123784861 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123757859 [Test build #61 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/61/console) for PR 7385 at commit [`1a19268`](https://github.com/apache/spark/commit/1a19268fda972f6103bcd50f376c0f57a238b394). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123757864 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123757719 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123758137 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123757812 [Test build #61 has started](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/61/consoleFull) for PR 7385 at commit [`1a19268`](https://github.com/apache/spark/commit/1a19268fda972f6103bcd50f376c0f57a238b394). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7276#issuecomment-123760499 [Test build #38084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38084/console) for PR 7276 at commit [`8f93c8d`](https://github.com/apache/spark/commit/8f93c8d1aec224a279c7f247d95e708072063d66). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7276#issuecomment-123763123 @zsxwing, it looks like SparkEnv has an executorId field that you can use. @JoshRosen Thank you. It works. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123768101 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7254][MLlib] Run PowerIterationClusteri...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6054#issuecomment-123782627 [Test build #1164 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1164/consoleFull) for PR 6054 at commit [`ef565a0`](https://github.com/apache/spark/commit/ef565a0ab061647b46dbb35424e86f41db45dad6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8838][SQL] Add config to enable/disable...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7238#issuecomment-123782677 [Test build #1166 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1166/console) for PR 7238 at commit [`ea8f6e5`](https://github.com/apache/spark/commit/ea8f6e5ed8a05019a94fd4bb5aa48766578ab1dd). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123782601 [Test build #1162 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1162/consoleFull) for PR 7385 at commit [`1a19268`](https://github.com/apache/spark/commit/1a19268fda972f6103bcd50f376c0f57a238b394). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123782550 [Test build #1159 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1159/consoleFull) for PR 7586 at commit [`ce0038a`](https://github.com/apache/spark/commit/ce0038acc0527495b94a19534ebd5c2b83a9ee19). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7276#issuecomment-123782647 [Test build #1161 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1161/console) for PR 7276 at commit [`8f93c8d`](https://github.com/apache/spark/commit/8f93c8d1aec224a279c7f247d95e708072063d66). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9251][SPARK-9083][SQL] do not order by ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7593#issuecomment-123782664 [Test build #1165 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1165/consoleFull) for PR 7593 at commit [`0f9b6da`](https://github.com/apache/spark/commit/0f9b6daacf35ad38cd971bdd454f0bfa2b075e00). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8838][SQL] Add config to enable/disable...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7238#issuecomment-123782657 [Test build #1166 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1166/consoleFull) for PR 7238 at commit [`ea8f6e5`](https://github.com/apache/spark/commit/ea8f6e5ed8a05019a94fd4bb5aa48766578ab1dd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123758089 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/7276#discussion_r35226184
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
---
@@ -170,101 +207,64 @@ class ReceiverTracker(ssc: StreamingContext,
skipReceiverLaunch: Boolean = false
// Signal the receivers to delete old block data
if (WriteAheadLogUtils.enableReceiverLog(ssc.conf)) {
logInfo(sCleanup old received batch data: $cleanupThreshTime)
- receiverInfo.values.flatMap { info = Option(info.endpoint) }
-.foreach { _.send(CleanupOldBlocks(cleanupThreshTime)) }
+ endpoint.send(CleanupOldBlocks(cleanupThreshTime))
}
}
- /** Register a receiver */
- private def registerReceiver(
- streamId: Int,
- typ: String,
- host: String,
- receiverEndpoint: RpcEndpointRef,
- senderAddress: RpcAddress
-): Boolean = {
-if (!receiverInputStreamIds.contains(streamId)) {
- throw new SparkException(Register received for unexpected id +
streamId)
-}
-
-if (isTrackerStopping || isTrackerStopped) {
- false
-} else {
- // stopReceivers won't happen at the same time because both
registerReceiver and are
- // called in the event loop. So here we can assume stopReceivers
has not yet been called. If
- // stopReceivers is called later, it should be able to see this
receiver.
- receiverInfo(streamId) = ReceiverInfo(
-streamId, s${typ}-${streamId}, receiverEndpoint, true, host)
-
listenerBus.post(StreamingListenerReceiverStarted(receiverInfo(streamId)))
- logInfo(Registered receiver for stream + streamId + from +
senderAddress)
- true
-}
+ /** Check if any blocks are left to be processed */
+ def hasUnallocatedBlocks: Boolean = {
+receivedBlockTracker.hasUnallocatedReceivedBlocks
}
- /** Deregister a receiver */
- private def deregisterReceiver(streamId: Int, message: String, error:
String) {
-val newReceiverInfo = receiverInfo.get(streamId) match {
- case Some(oldInfo) =
-val lastErrorTime =
- if (error == null || error == ) -1 else
ssc.scheduler.clock.getTimeMillis()
-oldInfo.copy(endpoint = null, active = false, lastErrorMessage =
message,
- lastError = error, lastErrorTime = lastErrorTime)
- case None =
-logWarning(No prior receiver info)
-val lastErrorTime =
- if (error == null || error == ) -1 else
ssc.scheduler.clock.getTimeMillis()
-ReceiverInfo(streamId, , null, false, , lastErrorMessage =
message,
- lastError = error, lastErrorTime = lastErrorTime)
-}
-receiverInfo -= streamId
-listenerBus.post(StreamingListenerReceiverStopped(newReceiverInfo))
-val messageWithError = if (error != null !error.isEmpty) {
- s$message - $error
-} else {
- s$message
-}
-logError(sDeregistered receiver for stream $streamId:
$messageWithError)
+ /**
+ * Get the list of executors excluding driver
+ */
+ private def getExecutors(ssc: StreamingContext): List[String] = {
+val executors =
ssc.sparkContext.getExecutorMemoryStatus.map(_._1.split(:)(0)).toList
+val driver = ssc.sparkContext.getConf.get(spark.driver.host)
+executors.diff(List(driver))
}
- /** Add new blocks for the given stream */
- private def addBlock(receivedBlockInfo: ReceivedBlockInfo): Boolean = {
-receivedBlockTracker.addBlock(receivedBlockInfo)
- }
+ /** Check if tracker has been marked for starting */
+ private def isTrackerStarted(): Boolean = trackerState == Started
- /** Report error sent by a receiver */
- private def reportError(streamId: Int, message: String, error: String) {
-val newReceiverInfo = receiverInfo.get(streamId) match {
- case Some(oldInfo) =
-oldInfo.copy(lastErrorMessage = message, lastError = error)
- case None =
-logWarning(No prior receiver info)
-ReceiverInfo(streamId, , null, false, , lastErrorMessage =
message,
- lastError = error, lastErrorTime =
ssc.scheduler.clock.getTimeMillis())
-}
-receiverInfo(streamId) = newReceiverInfo
-
listenerBus.post(StreamingListenerReceiverError(receiverInfo(streamId)))
-val messageWithError = if (error != null !error.isEmpty) {
- s$message - $error
-} else {
- s$message
-}
-logWarning(sError reported by receiver for stream $streamId:
$messageWithError)
- }
+ /** Check if tracker has been marked for stopping */
+ private def isTrackerStopping():
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7276#issuecomment-123759674 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123775195 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8975][Streaming] Adds a mechanism to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7471#issuecomment-123782489 [Test build #1157 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1157/consoleFull) for PR 7471 at commit [`8941cf9`](https://github.com/apache/spark/commit/8941cf91b03aa7835a78bc756bee1f32cb7bb1d8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123782306 @shaneknapp Can you manually delete the jar ? The script should auto-download a good version then AFAIK --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123757644 [Test build #38082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38082/consoleFull) for PR 7597 at commit [`a7cbfb3`](https://github.com/apache/spark/commit/a7cbfb32e639c35843d472ce86bb2d17d04ec563). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123757752 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123757662 [Test build #38082 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38082/console) for PR 7597 at commit [`a7cbfb3`](https://github.com/apache/spark/commit/a7cbfb32e639c35843d472ce86bb2d17d04ec563). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123757665 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123764826 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123764894 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9192][SQL] add initialization phase for...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/7535#issuecomment-123774434 hi @marmbrus @rxin , I have updated the description and code, can you review it for me? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-12319 @shaneknapp @JoshRosen Looks like there is a problem with Jenkins. The builds seem to be failing with ``` Launching sbt from build/sbt-launch-0.13.7.jar Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar [error] running /home/jenkins/workspace/SparkPullRequestBuilder/build/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive-thriftserver -Phive package assembly/assembly streaming-kafka-assembly/assembly streaming-flume-assembly/assembly ; received return code 1 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9216][Streaming] Define KinesisBackedBl...
Github user huitseeker commented on a diff in the pull request:
https://github.com/apache/spark/pull/7578#discussion_r35231803
--- Diff:
extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.kinesis
+
+import scala.collection.JavaConversions._
+
+import com.amazonaws.auth.{AWSCredentials,
DefaultAWSCredentialsProviderChain}
+import com.amazonaws.services.kinesis.AmazonKinesisClient
+import com.amazonaws.services.kinesis.model._
+
+import org.apache.spark._
+import org.apache.spark.rdd.{BlockRDD, BlockRDDPartition}
+import org.apache.spark.storage.BlockId
+import org.apache.spark.util.NextIterator
+
+
+/** Class representing a range of Kinesis sequence numbers */
+private[kinesis]
+case class SequenceNumberRange(
+streamName: String, shardId: String, fromSeqNumber: String,
toSeqNumber: String)
+
+/** Class representing an array of Kinesis sequence number ranges */
+private[kinesis]
+case class SequenceNumberRanges(ranges: Array[SequenceNumberRange]) {
+ def isEmpty(): Boolean = ranges.isEmpty
+ def nonEmpty(): Boolean = ranges.nonEmpty
+ override def toString(): String =
ranges.mkString(SequenceNumberRanges(, , , ))
+}
+
+private[kinesis]
+object SequenceNumberRanges {
+ def apply(range: SequenceNumberRange): SequenceNumberRanges = {
+new SequenceNumberRanges(Array(range))
+ }
+}
+
+
+/** Partition storing the information of the ranges of Kinesis sequence
numbers to read */
+private[kinesis]
+class KinesisBackedBlockRDDPartition(
+idx: Int,
+blockId: BlockId,
+val isBlockIdValid: Boolean,
+val seqNumberRanges: SequenceNumberRanges
+ ) extends BlockRDDPartition(blockId, idx)
+
+/**
+ * A BlockRDD where the block data is backed by Kinesis, which can
accessed using the
+ * sequence numbers of the corresponding blocks.
+ */
+private[kinesis]
+class KinesisBackedBlockRDD(
+sc: SparkContext,
+regionId: String,
+endpointUrl: String,
+@transient blockIds: Array[BlockId],
+@transient arrayOfseqNumberRanges: Array[SequenceNumberRanges],
+@transient isBlockIdValid: Array[Boolean] = Array.empty,
+awsCredentialsOption: Option[SerializableAWSCredentials] = None
+) extends BlockRDD[Array[Byte]](sc, blockIds) {
+
+ require(blockIds.length == arrayOfseqNumberRanges.length,
+Number of blockIds is not equal to the number of sequence number
ranges)
+
+ override def isValid(): Boolean = true
+
+ override def getPartitions: Array[Partition] = {
+Array.tabulate(blockIds.length) { i =
+ val isValid = if (isBlockIdValid.length == 0) true else
isBlockIdValid(i)
+ new KinesisBackedBlockRDDPartition(i, blockIds(i), isValid,
arrayOfseqNumberRanges(i))
+}
+ }
+
+ override def compute(split: Partition, context: TaskContext):
Iterator[Array[Byte]] = {
+val blockManager = SparkEnv.get.blockManager
+val partition = split.asInstanceOf[KinesisBackedBlockRDDPartition]
+val blockId = partition.blockId
+
+def getBlockFromBlockManager(): Option[Iterator[Array[Byte]]] = {
+ logDebug(sRead partition data of $this from block manager, block
$blockId)
+
blockManager.get(blockId).map(_.data.asInstanceOf[Iterator[Array[Byte]]])
+}
+
+def getBlockFromKinesis(): Iterator[Array[Byte]] = {
+ val credenentials = awsCredentialsOption.getOrElse {
+new DefaultAWSCredentialsProviderChain().getCredentials()
+ }
+ partition.seqNumberRanges.ranges.iterator.flatMap { range =
+new KinesisSequenceRangeIterator(credenentials, endpointUrl,
regionId, range)
+ }
+}
+if (partition.isBlockIdValid) {
+ getBlockFromBlockManager().getOrElse {
[GitHub] spark pull request: [SPARK-4366] [SQL] [Follow-up] Fix SqlParser c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7588#issuecomment-123782833 [Test build #1167 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1167/consoleFull) for PR 7588 at commit [`25f5f36`](https://github.com/apache/spark/commit/25f5f3665cf7fd1641600e57e0821558fa4b2746). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7259#issuecomment-123782721 [Test build #1168 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1168/consoleFull) for PR 7259 at commit [`78d40f1`](https://github.com/apache/spark/commit/78d40f1b1f87cf868c5872ef5e30396371156fdb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7254][MLlib] Run PowerIterationClusteri...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6054#issuecomment-123782719 [Test build #1164 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1164/console) for PR 6054 at commit [`ef565a0`](https://github.com/apache/spark/commit/ef565a0ab061647b46dbb35424e86f41db45dad6). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9024] Unsafe HashJoin/HashOuterJoin/Has...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7480#issuecomment-123782776 [Test build #1160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1160/console) for PR 7480 at commit [`6294b1e`](https://github.com/apache/spark/commit/6294b1e3de357c94646c323eba2d4bde80971c45). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5174][SPARK-5175] provide more APIs in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3984#issuecomment-123782737 [Test build #1169 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1169/console) for PR 3984 at commit [`b020228`](https://github.com/apache/spark/commit/b0202285c077f48a2e6949005d0371ee56755f03). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait ActorHelper extends Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7259#issuecomment-123782746 [Test build #1168 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1168/console) for PR 7259 at commit [`78d40f1`](https://github.com/apache/spark/commit/78d40f1b1f87cf868c5872ef5e30396371156fdb). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class ExpressionInfo ` * `case class DescribeFunction(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9095] [SQL] Removes the old Parquet sup...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7441#issuecomment-123782722 [Test build #1170 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1170/consoleFull) for PR 7441 at commit [`dc93730`](https://github.com/apache/spark/commit/dc93730a8fb22d38e5e1954c747a2c41401624c7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8186] [SPARK-8187] [SQL] datetime funct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7589#issuecomment-123782708 [Test build #1163 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1163/consoleFull) for PR 7589 at commit [`1a68e03`](https://github.com/apache/spark/commit/1a68e0334caee8c2fff822261fd728bd6209b4b6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9024] Unsafe HashJoin/HashOuterJoin/Has...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7480#issuecomment-123782680 [Test build #1160 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1160/consoleFull) for PR 7480 at commit [`6294b1e`](https://github.com/apache/spark/commit/6294b1e3de357c94646c323eba2d4bde80971c45). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5174][SPARK-5175] provide more APIs in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3984#issuecomment-123782717 [Test build #1169 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1169/consoleFull) for PR 3984 at commit [`b020228`](https://github.com/apache/spark/commit/b0202285c077f48a2e6949005d0371ee56755f03). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9192][SQL] add initialization phase for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7535#issuecomment-123782541 [Test build #1158 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1158/console) for PR 7535 at commit [`4dbed66`](https://github.com/apache/spark/commit/4dbed665182cb56a6c932c98e4f65b2d887f7372). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8975][Streaming] Adds a mechanism to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7471#issuecomment-123782588 [Test build #1157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1157/console) for PR 7471 at commit [`8941cf9`](https://github.com/apache/spark/commit/8941cf91b03aa7835a78bc756bee1f32cb7bb1d8). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123782660 [Test build #1162 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1162/console) for PR 7385 at commit [`1a19268`](https://github.com/apache/spark/commit/1a19268fda972f6103bcd50f376c0f57a238b394). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7276#issuecomment-123782563 [Test build #1161 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1161/consoleFull) for PR 7276 at commit [`8f93c8d`](https://github.com/apache/spark/commit/8f93c8d1aec224a279c7f247d95e708072063d66). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123756680 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9253][EC2] Allow to create machines wit...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7596#issuecomment-123756375 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123756671 Jenkins PR builders are down. Because the [1st candidate URL] [1] for `sbt-launch.lib` is down, while [the 2nd] [2] redirects requests to CDN and gives 302. [1]: https://github.com/apache/spark/blob/c03299a18b4e076cabb4b7833a1e7632c5c0dabe/build/sbt-launch-lib.bash#L41 [2]: https://github.com/apache/spark/blob/c03299a18b4e076cabb4b7833a1e7632c5c0dabe/build/sbt-launch-lib.bash#L42 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123759327 [Test build #38083 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38083/console) for PR 7385 at commit [`1a19268`](https://github.com/apache/spark/commit/1a19268fda972f6103bcd50f376c0f57a238b394). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7276#issuecomment-123759937 After considering carefully, I think if a receiver has a preferredLocation, it means it's better to run the receiver in this preferredLocation if possible. So I don't clear it when restart a receiver. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123769452 [Test build #38086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38086/consoleFull) for PR 7586 at commit [`ce0038a`](https://github.com/apache/spark/commit/ce0038acc0527495b94a19534ebd5c2b83a9ee19). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123783668 Jenkins my friend, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9253][EC2] Allow to create machines wit...
GitHub user ziky90 opened a pull request: https://github.com/apache/spark/pull/7596 [SPARK-9253][EC2] Allow to create machines with different AWS credentials than will be used for accessing the S3 Possibility to specify additional/different AWS credentials for security reasons. This is especially important for the security reasons. You can have AWS credentials with limited validity time that will expire with rights to create new machines and other AWS credential that has access only to particular buckets. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ziky90/spark custom_aws_credentials Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7596.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7596 commit 5a3aa4292f4f9658128dd5fbf33b9d4495f8d665 Author: Zikes, Jan jan.zi...@merck.com Date: 2015-07-22T15:08:32Z implemented [SPARK-9253] Allow to create machines with different AWS credentials than will be used for accessing the S3 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123757549 @tdas could you take a look at this PR again? I think it's better to merge this one at first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123757421 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/7276#discussion_r35226649
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverSchedulingPolicy.scala
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.scheduler
+
+import scala.collection.Map
+import scala.collection.mutable
+
+import org.apache.spark.streaming.receiver.Receiver
+
+private[streaming] class ReceiverSchedulingPolicy {
+
+ /**
+ * Try our best to schedule receivers with evenly distributed. However,
if the
+ * `preferredLocation`s of receivers are not even, we may not be able to
schedule them evenly
+ * because we have to respect them.
+ *
+ * This method is called when we start to launch receivers at the first
time.
+ */
+ def scheduleReceivers(
+ receivers: Seq[Receiver[_]], executors: Seq[String]): Map[Int,
Seq[String]] = {
+if (receivers.isEmpty) {
+ return Map.empty
+}
+
+if (executors.isEmpty) {
+ return receivers.map(_.streamId - Seq.empty).toMap
+}
+
+val hostToExecutors = executors.groupBy(_.split(:)(0))
+val locations = new
Array[mutable.ArrayBuffer[String]](receivers.length)
+val numReceiversOnExecutor = mutable.HashMap[String, Int]()
+// Set the initial value to 0
+executors.foreach(numReceiversOnExecutor(_) = 0)
+
+// Firstly, we need to respect preferredLocation. So if a receiver
has preferredLocation,
+// we need to make sure the preferredLocation is in the candidate
location list.
+for (i - 0 until receivers.length) {
+ locations(i) = new mutable.ArrayBuffer[String]()
+ // Note: preferredLocation is host but executors are host:port
+ receivers(i).preferredLocation.foreach { host =
+hostToExecutors.get(host) match {
+ case Some(executorsOnHost) =
+// preferredLocation is a known host. Select an executor that
has the least receivers in
+// this host
+val scheduledLocation =
+ executorsOnHost.minBy(executor =
numReceiversOnExecutor(executor))
+locations(i) += scheduledLocation
+numReceiversOnExecutor(scheduledLocation) =
+ numReceiversOnExecutor(scheduledLocation) + 1
+ case None =
+// preferredLocation is an unknown host.
+// Note: There are two cases:
+// 1. This executor is not up. But it may be up later.
+// 2. This executor is dead, or it's not a host in the cluster.
+// Currently, simply add host to the scheduled locations
+locations(i) += host
+}
+ }
+}
+
+// For those receivers that don't have preferredLocation, make sure we
assign at least one
+// executor to them.
+for (scheduledLocations - locations.filter(_.isEmpty)) {
+ // Select the executor that has the least receivers
+ val (executor, numReceivers) = numReceiversOnExecutor.minBy(_._2)
+ scheduledLocations += executor
+ numReceiversOnExecutor(executor) = numReceivers + 1
+}
+
+// Assign idle executors to receivers that have less executors
+val idleExecutors = numReceiversOnExecutor.filter(_._2 == 0).map(_._1)
+for (executor - idleExecutors) {
+ // Assign an idle executor to the receiver that has least locations.
+ val scheduledLocations = locations.minBy(_.size)
+ scheduledLocations += executor
+}
+
+receivers.map(_.streamId).zip(locations).toMap
+ }
+
+ /**
+ * Return a list of candidate executors to run the receiver. If the list
is empty, the caller can
+ * run this receiver in arbitrary executor.
+ *
+ * This method tries to balance executors' load. Here is the approach to
schedule executors
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7276#issuecomment-123760505 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8882][Streaming]Add a new Receiver sche...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7276#issuecomment-123760472 [Test build #38084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38084/consoleFull) for PR 7276 at commit [`8f93c8d`](https://github.com/apache/spark/commit/8f93c8d1aec224a279c7f247d95e708072063d66). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9024] Unsafe HashJoin/HashOuterJoin/Has...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7480#issuecomment-123766536 [Test build #1156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1156/console) for PR 7480 at commit [`6294b1e`](https://github.com/apache/spark/commit/6294b1e3de357c94646c323eba2d4bde80971c45). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123767150 [Test build #38085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38085/consoleFull) for PR 7597 at commit [`fd266ca`](https://github.com/apache/spark/commit/fd266ca86d96767afc973ff37383a0abd82796ca). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9024] Unsafe HashJoin/HashOuterJoin/Has...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7480#issuecomment-123766464 [Test build #1156 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1156/consoleFull) for PR 7480 at commit [`6294b1e`](https://github.com/apache/spark/commit/6294b1e3de357c94646c323eba2d4bde80971c45). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7597#issuecomment-123756712 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123759335 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7385#issuecomment-123759293 [Test build #38083 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38083/consoleFull) for PR 7385 at commit [`1a19268`](https://github.com/apache/spark/commit/1a19268fda972f6103bcd50f376c0f57a238b394). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
