Repository: spark
Updated Branches:
refs/heads/master 74d8f7dda -> e3b5d6cb2
[SPARK-10684] [SQL] StructType.interpretedOrdering need not to be serialized
Kryo fails with buffer overflow even with max value (2G).
{noformat}
org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow.
Available: 0, required: 1
Serialization trace:
containsChild (org.apache.spark.sql.catalyst.expressions.BoundReference)
child (org.apache.spark.sql.catalyst.expressions.SortOrder)
array (scala.collection.mutable.ArraySeq)
ordering (org.apache.spark.sql.catalyst.expressions.InterpretedOrdering)
interpretedOrdering (org.apache.spark.sql.types.StructType)
schema (org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema). To
avoid this, increase spark.kryoserializer.buffer.max value.
at
org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:263)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Author: navis.ryu <[email protected]>
Closes #8808 from navis/SPARK-10684.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e3b5d6cb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e3b5d6cb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e3b5d6cb
Branch: refs/heads/master
Commit: e3b5d6cb29e0f983fcc55920619e6433298955f5
Parents: 74d8f7d
Author: navis.ryu <[email protected]>
Authored: Fri Sep 18 00:43:02 2015 -0700
Committer: Reynold Xin <[email protected]>
Committed: Fri Sep 18 00:43:02 2015 -0700
----------------------------------------------------------------------
.../src/main/scala/org/apache/spark/sql/types/StructType.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/e3b5d6cb/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
----------------------------------------------------------------------
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
index d8968ef..b29cf22 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
@@ -305,7 +305,9 @@ case class StructType(fields: Array[StructField]) extends
DataType with Seq[Stru
f(this) || fields.exists(field => field.dataType.existsRecursively(f))
}
- private[sql] val interpretedOrdering =
InterpretedOrdering.forSchema(this.fields.map(_.dataType))
+ @transient
+ private[sql] lazy val interpretedOrdering =
+ InterpretedOrdering.forSchema(this.fields.map(_.dataType))
}
object StructType extends AbstractDataType {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
