[
https://issues.apache.org/jira/browse/SPARK-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181282#comment-14181282
]
Gen TANG commented on SPARK-2652:
---------------------------------
If we set
{code}
"spark.serializer", "org.apache.spark.serializer.KryoSerializer"
{code}
We will meet the problem about transformation between HashSet and BitSet in
Kyro when the algorithm uses BitSet , such as:
{code}
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException:
Can not set final scala.collection.mutable.BitSet to
scala.collection.mutable.HashSet
{code}
I met this problem when I used ALS and Spark HIVE. The problem will be fixed if
we register mutable.BitSet manually.
However, it is quite strange, because it seems that twitter.chill support
mutable.BitSet https://github.com/twitter/chill/pull/185
> Turning default configurations for PySpark
> ------------------------------------------
>
> Key: SPARK-2652
> URL: https://issues.apache.org/jira/browse/SPARK-2652
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 1.0.0
> Reporter: Davies Liu
> Assignee: Davies Liu
> Labels: Configuration, Python
> Fix For: 1.1.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Some default value of configuration does not make sense for PySpark, change
> them to reasonable ones, such as spark.serializer and
> spark.kryo.referenceTracking
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
