[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12913 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570784
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoBenchmark.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.serializer
+
+import scala.reflect.ClassTag
+import scala.util.Random
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.serializer.KryoTest._
+import org.apache.spark.util.Benchmark
+
+class KryoBenchmark extends SparkFunSuite {
+ val benchmark = new Benchmark("Benchmark Kryo Unsafe vs safe
Serialization", 1024 * 1024 * 15, 10)
+
+ ignore(s"Benchmark Kryo Unsafe vs safe Serialization") {
+Seq (true, false).foreach (runBenchmark)
+benchmark.run()
+
+// scalastyle:off
+/*
+ Benchmark Kryo Unsafe vs safe Serialization: Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
+
+ basicTypes: Int unsafe:true160 / 178
98.5 10.1 1.0X
+ basicTypes: Long unsafe:true 210 / 218
74.9 13.4 0.8X
+ basicTypes: Float unsafe:true 203 / 213
77.5 12.9 0.8X
+ basicTypes: Double unsafe:true 226 / 235
69.5 14.4 0.7X
+ Array: Int unsafe:true1087 / 1101
14.5 69.1 0.1X
+ Array: Long unsafe:true 2758 / 2844
5.7 175.4 0.1X
+ Array: Float unsafe:true 1511 / 1552
10.4 96.1 0.1X
+ Array: Double unsafe:true 2942 / 2972
5.3 187.0 0.1X
+ Map of string->Double unsafe:true 2645 / 2739
5.9 168.2 0.1X
+ basicTypes: Int unsafe:false 211 / 218
74.7 13.4 0.8X
+ basicTypes: Long unsafe:false 247 / 253
63.6 15.7 0.6X
+ basicTypes: Float unsafe:false 211 / 216
74.5 13.4 0.8X
+ basicTypes: Double unsafe:false227 / 233
69.2 14.4 0.7X
+ Array: Int unsafe:false 3012 / 3032
5.2 191.5 0.1X
+ Array: Long unsafe:false 4463 / 4515
3.5 283.8 0.0X
+ Array: Float unsafe:false 2788 / 2868
5.6 177.2 0.1X
+ Array: Double unsafe:false3558 / 3752
4.4 226.2 0.0X
+ Map of string->Double unsafe:false2806 / 2933
5.6 178.4 0.1X
+*/
+// scalastyle:on
+ }
+
+ private def runBenchmark(useUnsafe: Boolean): Unit = {
+def addBenchmark(name: String, values: Long)(f: => Long): Unit = {
+ benchmark.addCase(s"$name unsafe:$useUnsafe") { _ =>
+f
+ }
+}
+
+def check[T: ClassTag](t: T, ser: SerializerInstance): Int = {
+ if (ser.deserialize[T](ser.serialize(t)) === t) 1 else 0
+}
+
+val N1 = 100
+def basicTypes[T: ClassTag](name: String, fn: () => T): Unit = {
+ lazy val ser = createSerializer(useUnsafe)
+ addBenchmark(s"basicTypes: $name", N1) {
+var sum = 0L
+var i = 0
+while (i < N1) {
+ sum += check(fn(), ser)
+ i += 1
+}
+sum
+ }
+}
+basicTypes("Int", Random.nextInt)
+basicTypes("Long", Random.nextLong)
+basicTypes("Float", Random.nextFloat)
+basicTypes("Double", Random.nextDouble)
+
+val N2 = 1
+def
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570771
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoBenchmark.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.serializer
+
+import scala.reflect.ClassTag
+import scala.util.Random
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.serializer.KryoTest._
+import org.apache.spark.util.Benchmark
+
+class KryoBenchmark extends SparkFunSuite {
+ val benchmark = new Benchmark("Benchmark Kryo Unsafe vs safe
Serialization", 1024 * 1024 * 15, 10)
+
+ ignore(s"Benchmark Kryo Unsafe vs safe Serialization") {
+Seq (true, false).foreach (runBenchmark)
+benchmark.run()
+
+// scalastyle:off
+/*
+ Benchmark Kryo Unsafe vs safe Serialization: Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
+
+ basicTypes: Int unsafe:true160 / 178
98.5 10.1 1.0X
+ basicTypes: Long unsafe:true 210 / 218
74.9 13.4 0.8X
+ basicTypes: Float unsafe:true 203 / 213
77.5 12.9 0.8X
+ basicTypes: Double unsafe:true 226 / 235
69.5 14.4 0.7X
+ Array: Int unsafe:true1087 / 1101
14.5 69.1 0.1X
+ Array: Long unsafe:true 2758 / 2844
5.7 175.4 0.1X
+ Array: Float unsafe:true 1511 / 1552
10.4 96.1 0.1X
+ Array: Double unsafe:true 2942 / 2972
5.3 187.0 0.1X
+ Map of string->Double unsafe:true 2645 / 2739
5.9 168.2 0.1X
+ basicTypes: Int unsafe:false 211 / 218
74.7 13.4 0.8X
+ basicTypes: Long unsafe:false 247 / 253
63.6 15.7 0.6X
+ basicTypes: Float unsafe:false 211 / 216
74.5 13.4 0.8X
+ basicTypes: Double unsafe:false227 / 233
69.2 14.4 0.7X
+ Array: Int unsafe:false 3012 / 3032
5.2 191.5 0.1X
+ Array: Long unsafe:false 4463 / 4515
3.5 283.8 0.0X
+ Array: Float unsafe:false 2788 / 2868
5.6 177.2 0.1X
+ Array: Double unsafe:false3558 / 3752
4.4 226.2 0.0X
+ Map of string->Double unsafe:false2806 / 2933
5.6 178.4 0.1X
+*/
+// scalastyle:on
+ }
+
+ private def runBenchmark(useUnsafe: Boolean): Unit = {
+def addBenchmark(name: String, values: Long)(f: => Long): Unit = {
+ benchmark.addCase(s"$name unsafe:$useUnsafe") { _ =>
+f
+ }
+}
+
+def check[T: ClassTag](t: T, ser: SerializerInstance): Int = {
+ if (ser.deserialize[T](ser.serialize(t)) === t) 1 else 0
+}
+
+val N1 = 100
+def basicTypes[T: ClassTag](name: String, fn: () => T): Unit = {
--- End diff --
the use of call by name for fn is really confusing, especially the way you
are calling it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570758
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoBenchmark.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.serializer
+
+import scala.reflect.ClassTag
+import scala.util.Random
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.serializer.KryoTest._
+import org.apache.spark.util.Benchmark
+
+class KryoBenchmark extends SparkFunSuite {
+ val benchmark = new Benchmark("Benchmark Kryo Unsafe vs safe
Serialization", 1024 * 1024 * 15, 10)
+
+ ignore(s"Benchmark Kryo Unsafe vs safe Serialization") {
+Seq (true, false).foreach (runBenchmark)
+benchmark.run()
+
+// scalastyle:off
+/*
+ Benchmark Kryo Unsafe vs safe Serialization: Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
+
+ basicTypes: Int unsafe:true160 / 178
98.5 10.1 1.0X
+ basicTypes: Long unsafe:true 210 / 218
74.9 13.4 0.8X
+ basicTypes: Float unsafe:true 203 / 213
77.5 12.9 0.8X
+ basicTypes: Double unsafe:true 226 / 235
69.5 14.4 0.7X
+ Array: Int unsafe:true1087 / 1101
14.5 69.1 0.1X
+ Array: Long unsafe:true 2758 / 2844
5.7 175.4 0.1X
+ Array: Float unsafe:true 1511 / 1552
10.4 96.1 0.1X
+ Array: Double unsafe:true 2942 / 2972
5.3 187.0 0.1X
+ Map of string->Double unsafe:true 2645 / 2739
5.9 168.2 0.1X
+ basicTypes: Int unsafe:false 211 / 218
74.7 13.4 0.8X
+ basicTypes: Long unsafe:false 247 / 253
63.6 15.7 0.6X
+ basicTypes: Float unsafe:false 211 / 216
74.5 13.4 0.8X
+ basicTypes: Double unsafe:false227 / 233
69.2 14.4 0.7X
+ Array: Int unsafe:false 3012 / 3032
5.2 191.5 0.1X
+ Array: Long unsafe:false 4463 / 4515
3.5 283.8 0.0X
+ Array: Float unsafe:false 2788 / 2868
5.6 177.2 0.1X
+ Array: Double unsafe:false3558 / 3752
4.4 226.2 0.0X
+ Map of string->Double unsafe:false2806 / 2933
5.6 178.4 0.1X
+*/
+// scalastyle:on
+ }
+
+ private def runBenchmark(useUnsafe: Boolean): Unit = {
+def addBenchmark(name: String, values: Long)(f: => Long): Unit = {
--- End diff --
I find this function is over abstracted and actually makes it more
confusing what's going on. I'd just remove it since it is almost no-op.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570741
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -248,8 +266,9 @@ class KryoDeserializationStream(
}
}
-private[spark] class KryoSerializerInstance(ks: KryoSerializer) extends
SerializerInstance {
-
+private[spark] class KryoSerializerInstance(
--- End diff --
```
private[spark] class KryoSerializerInstance(ks: KryoSerializer, useUnsafe:
Boolean)
extends SerializerInstance {
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570738
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -219,9 +232,14 @@ class KryoSerializationStream(
private[spark]
class KryoDeserializationStream(
serInstance: KryoSerializerInstance,
-inStream: InputStream) extends DeserializationStream {
+inStream: InputStream,
+useUnsafe: Boolean) extends DeserializationStream {
- private[this] var input: KryoInput = new KryoInput(inStream)
+ private[this] var input: KryoInput = if (useUnsafe) {
+new KryoUnsafeInput(inStream)
--- End diff --
one line here too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570736
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -186,9 +193,15 @@ class KryoSerializer(conf: SparkConf)
private[spark]
class KryoSerializationStream(
serInstance: KryoSerializerInstance,
-outStream: OutputStream) extends SerializationStream {
+outStream: OutputStream,
+useUnsafe: Boolean) extends SerializationStream {
- private[this] var output: KryoOutput = new KryoOutput(outStream)
+ private[this] var output: KryoOutput =
+if (useUnsafe) {
--- End diff --
put this in one line?
```
if (useUnsafe) new KryoUnsafeOutput(outStream) else new
KryoOutput(outStream)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570727
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -78,8 +79,14 @@ class KryoSerializer(conf: SparkConf)
.filter(!_.isEmpty)
private val avroSchemas = conf.getAvroSchema
+ private val useUnsafe = conf.getBoolean("spark.kryo.unsafe", false)
--- End diff --
we should document what this means.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570704
--- Diff:
core/src/test/scala/org/apache/spark/serializer/UnsafeKryoSerializerSuite.scala
---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.serializer
+
+class UnsafeKryoSerializerSuite extends KryoSerializerSuite {
+
+ // This test suite should run all tests in KryoSerializerSuite with kryo
unsafe.
+
+ override def beforeAll() {
--- End diff --
don't you need to reset this back in afterAll?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user techaddict commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570678
--- Diff:
core/src/test/scala/org/apache/spark/serializer/UnsafeKryoSerializerSuite.scala
---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.serializer
+
+class UnsafeKryoSerializerSuite extends KryoSerializerSuite {
+
+ // This test suite should run all tests in KryoSerializerSuite with kryo
unsafe.
+
+ override def beforeAll() {
+super.beforeAll()
+conf.set("spark.kryo.unsafe", "true")
--- End diff --
Ohh yes, fixed and tested ð
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r84570152
--- Diff:
core/src/test/scala/org/apache/spark/serializer/UnsafeKryoSerializerSuite.scala
---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.serializer
+
+class UnsafeKryoSerializerSuite extends KryoSerializerSuite {
+
+ // This test suite should run all tests in KryoSerializerSuite with kryo
unsafe.
+
+ override def beforeAll() {
+super.beforeAll()
+conf.set("spark.kryo.unsafe", "true")
--- End diff --
Have you checked that this runs in the right order w.r.t. the code that
initializes a SparkContext in KryoSerializerSuite / SharedSparkContext? You may
want to print sc.conf in a test. Otherwise, it would be better to put
conf.set("spark.kryo.unsafe", "true") at the top-level inside this class.
SharedSparkContext does create a SparkContext in its beforeAll, but you seem to
editing the conf after that is done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r83972911
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala ---
@@ -75,9 +75,11 @@ class KryoSerializerSuite extends SparkFunSuite with
SharedSparkContext {
}
test("basic types") {
-val ser = new KryoSerializer(conf).newInstance()
def check[T: ClassTag](t: T) {
- assert(ser.deserialize[T](ser.serialize(t)) === t)
+ testBothUnsafeAndSafe(conf) { sparkConf =>
--- End diff --
(We have a few other examples suites where we just inherit from existing
suites, such as SortShuffleSuite vs ShuffleSuite.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r83971706
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala ---
@@ -75,9 +75,11 @@ class KryoSerializerSuite extends SparkFunSuite with
SharedSparkContext {
}
test("basic types") {
-val ser = new KryoSerializer(conf).newInstance()
def check[T: ClassTag](t: T) {
- assert(ser.deserialize[T](ser.serialize(t)) === t)
+ testBothUnsafeAndSafe(conf) { sparkConf =>
--- End diff --
Is there any way to refactor this into two Suites, where one extends from
the other and has additional setup code (or a setup method that it overrides)?
Otherwise we have to remember to add this construct to all tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r83971396
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -78,8 +79,14 @@ class KryoSerializer(conf: SparkConf)
.filter(!_.isEmpty)
private val avroSchemas = conf.getAvroSchema
+ private val useUnsafe = conf.getBoolean("spark.kryo.useUnsafe", false)
--- End diff --
Nit: I'd call this spark.kryo.unsafe to make it easier to remember
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user techaddict commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r73454617
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala ---
@@ -399,6 +399,14 @@ class KryoSerializerSuite extends SparkFunSuite with
SharedSparkContext {
assert(!ser2.getAutoReset)
}
+ private def testBothUnsafeAndSafe(f: SparkConf => Unit): Unit = {
--- End diff --
Yes will update the pr today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12913: [SPARK-928][CORE] Add support for Unsafe-based se...
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/12913#discussion_r73431033
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala ---
@@ -399,6 +399,14 @@ class KryoSerializerSuite extends SparkFunSuite with
SharedSparkContext {
assert(!ser2.getAutoReset)
}
+ private def testBothUnsafeAndSafe(f: SparkConf => Unit): Unit = {
--- End diff --
At first glance I don't see this function used anywhere I'm assuming this
was meant to wrap some of the tests?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
