[
https://issues.apache.org/jira/browse/SPARK-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519940#comment-14519940
]
ASF GitHub Bot commented on SPARK-6069:
---------------------------------------
GitHub user pferrel opened a pull request:
https://github.com/apache/mahout/pull/125
Spark 1.2
Created a Scala based BiMap and BiDictionary and removed usage of Guava
HashBiMap (actually no Guava is used now), I hope this gets past the
Javaserializer bug in https://issues.apache.org/jira/browse/SPARK-6069
Also found an error that did an unneeded collect, which will drastically
reduce memory requirements in reading in data for cooccurrence.
Passing around ```Option[BiDictionary] = None``` instead of an empty
dictionary, which seems a better idiom when using an immutable BiDictionary.
Questions:
* Passes local build tests but haven't tried on a cluster to verify it
works in spite of SPARK-6069
* Need to try it on epinions data before pushing
* Where should this go, it's in 0.11.0-SNAPSHOT now
* Still need to try putting Scala libs in mahout shell module only. This is
to check to see if they are needed in non-shell Scala modules
I'm having trouble getting HDFS+Spark running, even standalone, so if
anyone wants to try this I can provide instructions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pferrel/mahout spark-1.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/mahout/pull/125.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #125
----
commit 71165a54b9bc99f23c2cce6e8c63720b7fc64001
Author: Andrew Palumbo <[email protected]>
Date: 2015-02-24T22:54:15Z
added scala dependencies in math-scala to fix spark-shell
commit 75054c195ed71fe51f6e3de85e598f6ce6d51c3a
Author: pferrel <[email protected]>
Date: 2015-04-29T14:36:58Z
initial cut at making cooccurrence work on Spark 1.2--runs locally but
cleanup needed, also major memory use optimizations
commit 990347b16258b3bf410c130314a2225654100181
Author: pferrel <[email protected]>
Date: 2015-04-29T15:49:24Z
refactored BiDictionary and BiMap to carve off dicictionary specific
behavior in the companion object
commit 15555d897176e4d2e50552d476741e5c314c072f
Author: pferrel <[email protected]>
Date: 2015-04-29T16:19:08Z
runs build tests now removing the .create factory method, which will
require creation only when keys are already available
commit f3611855fbf2b06bc3faa7083484d426e84e1b73
Author: pferrel <[email protected]>
Date: 2015-04-29T18:37:38Z
using Option[BiDictionary] = None now instread of passing around an empty
BiDictionary
----
> Deserialization Error ClassNotFoundException with Kryo, Guava 14
> ----------------------------------------------------------------
>
> Key: SPARK-6069
> URL: https://issues.apache.org/jira/browse/SPARK-6069
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.1
> Environment: Standalone one worker cluster on localhost, or any
> cluster
> Reporter: Pat Ferrel
> Priority: Critical
>
> A class is contained in the jars passed in when creating a context. It is
> registered with kryo. The class (Guava HashBiMap) is created correctly from
> an RDD and broadcast but the deserialization fails with ClassNotFound.
> The work around is to hard code the path to the jar and make it available on
> all workers. Hard code because we are creating a library so there is no easy
> way to pass in to the app something like:
> spark.executor.extraClassPath /path/to/some.jar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
