[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167855990 What first two days of work are you referring to? The problem is we just can't use Kryo for serializing task results because there might be unregistered classes. Because of this constraint we can't use `spark.serializer` here since the user can specify Kryo there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167859672 Thanks for putting the effort into this @tedyu. It did move us forward in the sense that now we know we can't use Kryo for serializing task results. I'm going to close the issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu closed the pull request at: https://github.com/apache/spark/pull/10368 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167802141 I think the first two days' work aligns with SPARK-12414. Is it possible to keep that (without modification to KryoSerializer and the new test) ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on a diff in the pull request: https://github.com/apache/spark/pull/10368#discussion_r48507734 --- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala --- @@ -109,6 +111,9 @@ class KryoSerializer(conf: SparkConf) kryo.register(classOf[SerializableJobConf], new KryoJavaSerializer()) kryo.register(classOf[HttpBroadcast[_]], new KryoJavaSerializer()) kryo.register(classOf[PythonBroadcast], new KryoJavaSerializer()) +kryo.register(classOf[TaskMetrics], new KryoJavaSerializer()) +kryo.register(classOf[DirectTaskResult[_]], new KryoJavaSerializer()) +kryo.register(classOf[IndirectTaskResult[_]], new KryoJavaSerializer()) --- End diff -- bq. people may forget to register new classes if they just add an Option field to TaskMetrics in future Addition to TaskMetrics would be reviewed, right ? A comment can be added to TaskMetrics reminding them to register corresponding class. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10368#discussion_r48507496 --- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala --- @@ -109,6 +111,9 @@ class KryoSerializer(conf: SparkConf) kryo.register(classOf[SerializableJobConf], new KryoJavaSerializer()) kryo.register(classOf[HttpBroadcast[_]], new KryoJavaSerializer()) kryo.register(classOf[PythonBroadcast], new KryoJavaSerializer()) +kryo.register(classOf[TaskMetrics], new KryoJavaSerializer()) +kryo.register(classOf[DirectTaskResult[_]], new KryoJavaSerializer()) +kryo.register(classOf[IndirectTaskResult[_]], new KryoJavaSerializer()) --- End diff -- I just found there are more classes need to register, such as `ShuffleReadMetrics`, `OutputMetrics` and `ShuffleWriteMetrics`. The problem is that most fields in `TaskMetrics` are `Option` so the new added unit tests cannot cover them. Not sure how many people uses `spark.kryo.registrationRequired`. But if they enable it, we will break their codes. In addition, even if we register all necessary classes now, people may forget to register new classes if they just add an `Option` field in future. I prefer to close this PR since I don't think this patch buys us much due to the tricky thing introduced. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10368#discussion_r48508806 --- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala --- @@ -109,6 +111,9 @@ class KryoSerializer(conf: SparkConf) kryo.register(classOf[SerializableJobConf], new KryoJavaSerializer()) kryo.register(classOf[HttpBroadcast[_]], new KryoJavaSerializer()) kryo.register(classOf[PythonBroadcast], new KryoJavaSerializer()) +kryo.register(classOf[TaskMetrics], new KryoJavaSerializer()) +kryo.register(classOf[DirectTaskResult[_]], new KryoJavaSerializer()) +kryo.register(classOf[IndirectTaskResult[_]], new KryoJavaSerializer()) --- End diff -- Moreover, `_accumulatorUpdates` is much more complicated. Its value could be any type, e.g., `SQLMetricValue`. If someone adds a new `SQLMetricValue` type, it's hard for reviewers to recall it will be stored in `TaskMetrics`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/10368#discussion_r48511217 --- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala --- @@ -109,6 +111,9 @@ class KryoSerializer(conf: SparkConf) kryo.register(classOf[SerializableJobConf], new KryoJavaSerializer()) kryo.register(classOf[HttpBroadcast[_]], new KryoJavaSerializer()) kryo.register(classOf[PythonBroadcast], new KryoJavaSerializer()) +kryo.register(classOf[TaskMetrics], new KryoJavaSerializer()) +kryo.register(classOf[DirectTaskResult[_]], new KryoJavaSerializer()) +kryo.register(classOf[IndirectTaskResult[_]], new KryoJavaSerializer()) --- End diff -- I see, that does seem like a real problem. Let's move this discussion to the main thread so more people can see. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167673871 As @zsxwing pointed out, the existing unit tests are not sufficient because they didn't catch the fact that `ShuffleReadMetrics` et al must also be registered. This is because these fields are options. In the future, people are free to add more options fields and it will be hard for reviewers to maintain this additional requirement of registering it with Kryo. Given this the suggestion is to close this issue for now. Eventually when we get rid of closure serializer altogether (SPARK-12414) we can just replace this one with Java serializer, and add a huge comment why it can't be Kryo. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48513944
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala ---
@@ -81,6 +81,16 @@ class TaskResultGetterSuite extends SparkFunSuite with
BeforeAndAfter with Local
// as we can make it) so the tests don't take too long.
def conf: SparkConf = new SparkConf().set("spark.akka.frameSize", "1")
+ test("Kryo serializer for TaskResult") {
+val conf1 = new SparkConf(false)
+conf1.set("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
--- End diff --
I wasn't aware of SPARK-12414 previously.
If Java serializer is to be used everywhere, is there some code in this PR
that is useful for SPARK-12414 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48458317
--- Diff: core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala
---
@@ -220,7 +222,7 @@ class HeartbeatReceiverSuite
} else {
assert(!response.reregisterBlockManager)
// Additionally verify that the scheduler callback is called with
the correct parameters
- verify(scheduler).executorHeartbeatReceived(
+ verify(scheduler,
VerificationModeFactory.atLeast(1)).executorHeartbeatReceived(
--- End diff --
@tedyu this still needs attention. Is the rest OK to commit @andrewor14 or
are you interested in refining the tests?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167101101 **[Test build #48304 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48304/consoleFull)** for PR 10368 at commit [`a4173eb`](https://github.com/apache/spark/commit/a4173ebf248617fd16864df156516e6c038a09a1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167101251 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167101252 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48304/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167081572 **[Test build #48304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48304/consoleFull)** for PR 10368 at commit [`a4173eb`](https://github.com/apache/spark/commit/a4173ebf248617fd16864df156516e6c038a09a1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/10368#issuecomment-167086261
```
sbt.ForkMain$ForkError:
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
eventually never returned normally. Attempted 209 times over 10.019028894
seconds. Last failure message: strings.forall({
((elem: Any) => DirectKafkaStreamSuite.collectedData.contains(elem))
}) was false.
at
org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)
at
org.apache.spark.streaming.kafka.DirectKafkaStreamSuite.eventually(DirectKafkaStreamSuite.scala:44)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307)
at
org.apache.spark.streaming.kafka.DirectKafkaStreamSuite.eventually(DirectKafkaStreamSuite.scala:44)
at
org.apache.spark.streaming.kafka.DirectKafkaStreamSuite$$anonfun$5.org$apache$spark$streaming$kafka$DirectKafkaStreamSuite$$anonfun$$sendDataAndWaitForReceive$1(DirectKafkaStreamSuite.scala:249)
at
org.apache.spark.streaming.kafka.DirectKafkaStreamSuite$$anonfun$5.apply$mcV$sp(DirectKafkaStreamSuite.scala:310)
at
org.apache.spark.streaming.kafka.DirectKafkaStreamSuite$$anonfun$5.apply(DirectKafkaStreamSuite.scala:235)
at
org.apache.spark.streaming.kafka.DirectKafkaStreamSuite$$anonfun$5.apply(DirectKafkaStreamSuite.scala:235)
```
Not sure if the test failure in build #48288 was related.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167079477 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48372722
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala ---
@@ -154,6 +155,19 @@ class KryoSerializerSuite extends SparkFunSuite with
SharedSparkContext {
mutable.HashMap(1 -> "one", 2 -> "two", 3 -> "three")))
}
+ test("Bug: SPARK-12415") {
+val ser = new
KryoSerializer(conf.clone.set("spark.kryo.registrationRequired", "true"))
+ .newInstance()
+def check[T: ClassTag](t: T) {
+ val ret = ser.deserialize[T](ser.serialize(t))
+ assert(ret.equals(t), "Deser " + ret + " orig " + t)
--- End diff --
Nit: Use `==` insead
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48371027
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala
---
@@ -46,6 +46,26 @@ class DirectTaskResult[T](var valueBytes: ByteBuffer,
var accumUpdates: Map[Long
def this() = this(null.asInstanceOf[ByteBuffer], null, null)
+ override def toString: String = valueBytes.toString + " " + accumUpdates
+ " " + metrics
+
+ override def equals(other: Any): Boolean = other match {
+case that: DirectTaskResult[_] => {
+ if (!this.valueBytes.equals(that.valueBytes)) return false
+
+ val accumSize = if (accumUpdates != null) accumUpdates.size else 0
+ val thatAccumSize = if (that.accumUpdates != null)
that.accumUpdates.size else 0
+ if (accumSize != thatAccumSize) return false
+ if (accumSize > 0) {
+val b = this.accumUpdates.keys.forall { key =>
+ this.accumUpdates.get(key) == that.accumUpdates.get(key)
+}
+if (!b) return false;
+ }
+ this.metrics.equals(that.metrics)
+}
--- End diff --
similarly, this can be simplified
```
val accumEquals =
if (accumUpdates != null) {
accumUpdates.keys.forall { k => accumUpdates.get(k) ==
that.accumUpdates.get(k) }
} else {
that.accumUpdates == null
}
valueBytes.equals(that.valueBytes) &&
metrics.equals(that.metrics) &&
accumEquals
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48370672
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -215,6 +215,19 @@ class TaskMetrics extends Serializable {
inputMetrics.foreach(_.updateBytesRead())
}
+ override def equals(other: Any): Boolean = other match {
+case that: TaskMetrics => {
+ if (this.executorDeserializeTime != that.executorDeserializeTime)
return false
+ if (this.executorRunTime != that.executorRunTime) return false
+ if (this.resultSize != that.resultSize) return false
+ if (this.jvmGCTime != that.jvmGCTime) return false
+ if (this.resultSerializationTime != that.resultSerializationTime)
return false
+ if (this.hostname == null && that.hostname != null) return false
+ if (this.hostname != null) this.hostname.equals(that.hostname) else
true
+}
--- End diff --
it would be better if this looked like:
```
this.executorDeserializeTime == that.executorDeserializeTime &&
this.executorRunTime == that.executorRunTime &&
...
this.hostname == that.hostname
```
no need to check `equals` here because we're in scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48372093
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala
---
@@ -46,6 +46,26 @@ class DirectTaskResult[T](var valueBytes: ByteBuffer,
var accumUpdates: Map[Long
def this() = this(null.asInstanceOf[ByteBuffer], null, null)
+ override def toString: String = valueBytes.toString + " " + accumUpdates
+ " " + metrics
+
+ override def equals(other: Any): Boolean = other match {
+case that: DirectTaskResult[_] => {
+ if (!this.valueBytes.equals(that.valueBytes)) return false
+
+ val accumSize = if (accumUpdates != null) accumUpdates.size else 0
+ val thatAccumSize = if (that.accumUpdates != null)
that.accumUpdates.size else 0
+ if (accumSize != thatAccumSize) return false
+ if (accumSize > 0) {
+val b = this.accumUpdates.keys.forall { key =>
--- End diff --
Why compare two Maps manually? I think `Map` already implements `equals`.
Right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48372644
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala
---
@@ -46,6 +46,26 @@ class DirectTaskResult[T](var valueBytes: ByteBuffer,
var accumUpdates: Map[Long
def this() = this(null.asInstanceOf[ByteBuffer], null, null)
+ override def toString: String = valueBytes.toString + " " + accumUpdates
+ " " + metrics
+
+ override def equals(other: Any): Boolean = other match {
+case that: DirectTaskResult[_] => {
+ if (!this.valueBytes.equals(that.valueBytes)) return false
+
+ val accumSize = if (accumUpdates != null) accumUpdates.size else 0
+ val thatAccumSize = if (that.accumUpdates != null)
that.accumUpdates.size else 0
+ if (accumSize != thatAccumSize) return false
+ if (accumSize > 0) {
+val b = this.accumUpdates.keys.forall { key =>
+ this.accumUpdates.get(key) == that.accumUpdates.get(key)
+}
+if (!b) return false;
+ }
+ this.metrics.equals(that.metrics)
+}
--- End diff --
Also use `==` instead of `equals`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166972579 The rest of it looks OK. @zsxwing can you take another pass? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/10368#discussion_r48371257 --- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala --- @@ -109,6 +111,9 @@ class KryoSerializer(conf: SparkConf) kryo.register(classOf[SerializableJobConf], new KryoJavaSerializer()) kryo.register(classOf[HttpBroadcast[_]], new KryoJavaSerializer()) kryo.register(classOf[PythonBroadcast], new KryoJavaSerializer()) +kryo.register(classOf[TaskMetrics], new KryoJavaSerializer()) +kryo.register(classOf[DirectTaskResult[_]], new KryoJavaSerializer()) +kryo.register(classOf[IndirectTaskResult[_]], new KryoJavaSerializer()) --- End diff -- FYI, each of these calls is actually somewhat expensive. We create thousands of new Kryo serializer instances because we don't reuse them and so we end up calling `register` here many times. There is an outstanding issue [SPARK-12473](https://issues.apache.org/jira/browse/SPARK-12473) to resolve this, but until that is fixed this commit will likely cause minor performance regressions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48389119
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -215,6 +215,18 @@ class TaskMetrics extends Serializable {
inputMetrics.foreach(_.updateBytesRead())
}
+ override def equals(other: Any): Boolean = other match {
--- End diff --
yeah, that would be preferable actually. In the future if we add more
fields to `TaskMetrics` we might forget to add it to `equals`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167022107 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167022108 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48273/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167022099 **[Test build #48273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48273/consoleFull)** for PR 10368 at commit [`710dc44`](https://github.com/apache/spark/commit/710dc448e2b25fad734c624d834a47ff8990b840). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/10368#discussion_r48389780 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala --- @@ -46,6 +46,8 @@ class DirectTaskResult[T](var valueBytes: ByteBuffer, var accumUpdates: Map[Long def this() = this(null.asInstanceOf[ByteBuffer], null, null) + override def toString: String = valueBytes.toString + " " + accumUpdates + " " + metrics --- End diff -- This representation isn't really useful. It doesn't even tell you the task ID. I would just revert this change since it's unrelated and doesn't add much value. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167015292 **[Test build #48269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48269/consoleFull)** for PR 10368 at commit [`4b5657b`](https://github.com/apache/spark/commit/4b5657b25658d8e06021a55c48d0277d8a569ba3). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167015301 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167015302 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48269/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167019833 **[Test build #48273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48273/consoleFull)** for PR 10368 at commit [`710dc44`](https://github.com/apache/spark/commit/710dc448e2b25fad734c624d834a47ff8990b840). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167039836 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167039814 **[Test build #48279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48279/consoleFull)** for PR 10368 at commit [`5011926`](https://github.com/apache/spark/commit/501192630532e37223f9f9d9e5fb7a5c7b8936aa). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167039837 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48279/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167016785 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48257/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167016754 **[Test build #48257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48257/consoleFull)** for PR 10368 at commit [`7cfde71`](https://github.com/apache/spark/commit/7cfde714ba2f7716ba2459f5165f44a254959237). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167016784 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167024446 **[Test build #48275 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48275/consoleFull)** for PR 10368 at commit [`846b939`](https://github.com/apache/spark/commit/846b93900b5fcc0f35b30a42c0c6a48f61063014). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167038607 **[Test build #48283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48283/consoleFull)** for PR 10368 at commit [`a4173eb`](https://github.com/apache/spark/commit/a4173ebf248617fd16864df156516e6c038a09a1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167025357 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48275/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167025330 **[Test build #48275 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48275/consoleFull)** for PR 10368 at commit [`846b939`](https://github.com/apache/spark/commit/846b93900b5fcc0f35b30a42c0c6a48f61063014). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167025356 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167028329 **[Test build #48278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48278/consoleFull)** for PR 10368 at commit [`ac0199a`](https://github.com/apache/spark/commit/ac0199a11cfc885fa8f062e73b8c439bf9f3cab3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167033459 **[Test build #48279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48279/consoleFull)** for PR 10368 at commit [`5011926`](https://github.com/apache/spark/commit/501192630532e37223f9f9d9e5fb7a5c7b8936aa). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10368#discussion_r48389313 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala --- @@ -46,6 +46,8 @@ class DirectTaskResult[T](var valueBytes: ByteBuffer, var accumUpdates: Map[Long def this() = this(null.asInstanceOf[ByteBuffer], null, null) + override def toString: String = valueBytes.toString + " " + accumUpdates + " " + metrics --- End diff -- nit: remove this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48389325
--- Diff: core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala
---
@@ -220,7 +222,7 @@ class HeartbeatReceiverSuite
} else {
assert(!response.reregisterBlockManager)
// Additionally verify that the scheduler callback is called with
the correct parameters
- verify(scheduler).executorHeartbeatReceived(
+ verify(scheduler,
VerificationModeFactory.atLeast(1)).executorHeartbeatReceived(
--- End diff --
nit: revert changes to `HeartbeatReceiverSuite`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167014071 LGTM except some nits. Thanks a lot for fixing this @tedyu --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167014373 **[Test build #48269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48269/consoleFull)** for PR 10368 at commit [`4b5657b`](https://github.com/apache/spark/commit/4b5657b25658d8e06021a55c48d0277d8a569ba3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on a diff in the pull request: https://github.com/apache/spark/pull/10368#discussion_r48389541 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala --- @@ -46,6 +46,8 @@ class DirectTaskResult[T](var valueBytes: ByteBuffer, var accumUpdates: Map[Long def this() = this(null.asInstanceOf[ByteBuffer], null, null) + override def toString: String = valueBytes.toString + " " + accumUpdates + " " + metrics --- End diff -- This would make String form of TaskResult more readable. Can this be kept ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/10368#issuecomment-167018650
Here was why I implemented equals for Map (from #48257):
{code}
[info] - Bug: SPARK-12415 *** FAILED *** (3 milliseconds)
[info] java.nio.HeapByteBuffer[pos=0 lim=2 cap=2] null
org.apache.spark.executor.TaskMetrics@60d6ae17 did not equal
java.nio.HeapByteBuffer[pos=0 lim=2 cap=2] Map()
org.apache.spark.executor.TaskMetrics@2122833f Deser
java.nio.HeapByteBuffer[pos=0 lim=2 cap=2] null
org.apache.spark.executor.TaskMetrics@60d6ae17 orig
java.nio.HeapByteBuffer[pos=0 lim=2 cap=2] Map()
org.apache.spark.executor.TaskMetrics@2122833f (KryoSerializerSuite.scala:163)
{code}
One Map was null and one Map was empty.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167030489 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48278/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167030479 **[Test build #48278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48278/consoleFull)** for PR 10368 at commit [`ac0199a`](https://github.com/apache/spark/commit/ac0199a11cfc885fa8f062e73b8c439bf9f3cab3). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167030488 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166999771 **[Test build #48257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48257/consoleFull)** for PR 10368 at commit [`7cfde71`](https://github.com/apache/spark/commit/7cfde714ba2f7716ba2459f5165f44a254959237). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48385365
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -215,6 +215,18 @@ class TaskMetrics extends Serializable {
inputMetrics.foreach(_.updateBytesRead())
}
+ override def equals(other: Any): Boolean = other match {
--- End diff --
This is not a correct `equals`. There are other fields in `TaskMetrics`. To
make `TaskMetrics.equals` work, there are also other classes need to be
updated. Instead, could you just manually compare fields of DirectTaskResult in
the tests rather than adding an incorrect `equals`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10368#discussion_r48384718
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala
---
@@ -46,6 +46,23 @@ class DirectTaskResult[T](var valueBytes: ByteBuffer,
var accumUpdates: Map[Long
def this() = this(null.asInstanceOf[ByteBuffer], null, null)
+ override def toString: String = valueBytes.toString + " " + accumUpdates
+ " " + metrics
+
+ override def equals(other: Any): Boolean = other match {
+case that: DirectTaskResult[_] => {
+ val accumEquals =
+if (accumUpdates != null) {
+ accumUpdates.keys.forall { k => accumUpdates.get(k) ==
that.accumUpdates.get(k) }
+} else {
+ that.accumUpdates == null
+}
+ valueBytes == that.valueBytes &&
+ metrics == that.metrics &&
+ accumEquals
--- End diff --
We can just use `accumUpdates == that.accumUpdates` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167047677 **[Test build #48288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48288/consoleFull)** for PR 10368 at commit [`a4173eb`](https://github.com/apache/spark/commit/a4173ebf248617fd16864df156516e6c038a09a1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167046527 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167046506 ``` [info] PostgresIntegrationSuite: [info] Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.jdbc.PostgresIntegrationSuite *** ABORTED *** (9 seconds, 292 milliseconds) [info] com.spotify.docker.client.DockerRequestException: Request error: POST unix://localhost:80/containers/bc67f5d74a6cfb298d238cbc53eb6540564c4e642ac630e6448da4dd0871fc37/start: 500 [info] at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1133) [info] at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1104) [info] at com.spotify.docker.client.DefaultDockerClient.startContainer(DefaultDockerClient.java:401) ``` Don't think the above was related to this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167061036 **[Test build #48288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48288/consoleFull)** for PR 10368 at commit [`a4173eb`](https://github.com/apache/spark/commit/a4173ebf248617fd16864df156516e6c038a09a1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167061069 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167044665 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48283/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167044643 **[Test build #48283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48283/consoleFull)** for PR 10368 at commit [`a4173eb`](https://github.com/apache/spark/commit/a4173ebf248617fd16864df156516e6c038a09a1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167044664 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-167061071 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48288/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166387628 @andrewor14 @zsxwing Please take another look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166387458 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48119/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166387455 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166387289 **[Test build #48119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48119/consoleFull)** for PR 10368 at commit [`bc7699c`](https://github.com/apache/spark/commit/bc7699cc2793aeaa74f6a51bd8518dc91f327999). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166358855 **[Test build #48119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48119/consoleFull)** for PR 10368 at commit [`bc7699c`](https://github.com/apache/spark/commit/bc7699cc2793aeaa74f6a51bd8518dc91f327999). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166358959 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48118/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166358956 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166149468 **[Test build #48087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48087/consoleFull)** for PR 10368 at commit [`86dbca8`](https://github.com/apache/spark/commit/86dbca8e399e260f56c2a27ca1ab9748c83e2962). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166154566 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166154567 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48087/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166154468 **[Test build #48087 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48087/consoleFull)** for PR 10368 at commit [`86dbca8`](https://github.com/apache/spark/commit/86dbca8e399e260f56c2a27ca1ab9748c83e2962). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166032102 **[Test build #48059 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48059/consoleFull)** for PR 10368 at commit [`b65aeb6`](https://github.com/apache/spark/commit/b65aeb67ea3a8a8bfabead4a7ba86a2dfeb83fa0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-165997773 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48053/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-165997757 **[Test build #48053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48053/consoleFull)** for PR 10368 at commit [`cdd07e2`](https://github.com/apache/spark/commit/cdd07e2f61ba8f1f58954fe1c689c8096ed07985). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-165997771 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166028352 **[Test build #48058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48058/consoleFull)** for PR 10368 at commit [`422ac1c`](https://github.com/apache/spark/commit/422ac1cfd5369e9a00e3d52693bb868048e30ba2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166028405 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166028404 **[Test build #48058 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48058/consoleFull)** for PR 10368 at commit [`422ac1c`](https://github.com/apache/spark/commit/422ac1cfd5369e9a00e3d52693bb868048e30ba2). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166028406 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48058/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166046523 **[Test build #2239 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2239/consoleFull)** for PR 10368 at commit [`facf246`](https://github.com/apache/spark/commit/facf2460e184900c66393edd3652607cb4e40b12). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166046525 **[Test build #2239 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2239/consoleFull)** for PR 10368 at commit [`facf246`](https://github.com/apache/spark/commit/facf2460e184900c66393edd3652607cb4e40b12). * This patch **fails executing the `dev/run-tests` script**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166061058 **[Test build #48068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48068/consoleFull)** for PR 10368 at commit [`facf246`](https://github.com/apache/spark/commit/facf2460e184900c66393edd3652607cb4e40b12). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166059915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166059903 **[Test build #48067 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48067/consoleFull)** for PR 10368 at commit [`facf246`](https://github.com/apache/spark/commit/facf2460e184900c66393edd3652607cb4e40b12). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166033780 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166034111 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166034112 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48063/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166034107 **[Test build #48063 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48063/consoleFull)** for PR 10368 at commit [`facf246`](https://github.com/apache/spark/commit/facf2460e184900c66393edd3652607cb4e40b12). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166034110 **[Test build #48063 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48063/consoleFull)** for PR 10368 at commit [`facf246`](https://github.com/apache/spark/commit/facf2460e184900c66393edd3652607cb4e40b12). * This patch **fails executing the `dev/run-tests` script**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166059917 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48067/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166070595 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48070/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user tedyu commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166052576 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10368#issuecomment-166055164 **[Test build #48067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48067/consoleFull)** for PR 10368 at commit [`facf246`](https://github.com/apache/spark/commit/facf2460e184900c66393edd3652607cb4e40b12). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
