[
https://issues.apache.org/jira/browse/SPARK-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614421#comment-14614421
]
Renxu Sun commented on SPARK-7483:
----------------------------------
I've tried to change the nodes property of Summay class inside object FPTree
from ListBuffer to ArrayBuffer. And it seems to fix the exception problem as
well as producing the right item counts so far. Suspect the problem with
KryoSerializer dealing with ListBuffer class, but haven't looked into details.
The code looks like:
private[fpm] object FPTree {
/** Representing a node in an FP-Tree. */
class Node[T](val parent: Node[T]) extends Serializable {
var item: T = _
var count: Long = 0L
val children: mutable.Map[T, Node[T]] = mutable.Map.empty
def isRoot: Boolean = parent == null
}
/** Summary of a item in an FP-Tree. */
private class Summary[T] extends Serializable {
var count: Long = 0L
val nodes: mutable.ArrayBuffer[Node[T]] = mutable.ArrayBuffer.empty
}
}
> [MLLib] Using Kryo with FPGrowth fails with an exception
> --------------------------------------------------------
>
> Key: SPARK-7483
> URL: https://issues.apache.org/jira/browse/SPARK-7483
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 1.3.1
> Reporter: Tomasz Bartczak
> Priority: Minor
>
> When using FPGrowth algorithm with KryoSerializer - Spark fails with
> {code}
> Job aborted due to stage failure: Task 0 in stage 9.0 failed 1 times, most
> recent failure: Lost task 0.0 in stage 9.0 (TID 16, localhost):
> com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException:
> Can not set final scala.collection.mutable.ListBuffer field
> org.apache.spark.mllib.fpm.FPTree$Summary.nodes to
> scala.collection.mutable.ArrayBuffer
> Serialization trace:
> nodes (org.apache.spark.mllib.fpm.FPTree$Summary)
> org$apache$spark$mllib$fpm$FPTree$$summaries
> (org.apache.spark.mllib.fpm.FPTree)
> {code}
> This can be easily reproduced in spark codebase by setting
> {code}
> conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
> {code} and running FPGrowthSuite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
