Hi All, There an issue with serializing Spark's MatrixFactorizationModel object. The object contains a huge RDD and as I have read in many blogs, this model cannot be serialized as a java object. Therefore when retrieving the model I get the same exception as above;
*Caused by: java.lang.ClassNotFoundException: org.apache.spark.OneToOneDependency cannot be found by org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* I have asked this question in Spark mailing lists and they recommended me to use the built in save and load functions other than using Java serializing. So I have used following method to persist the model, model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), outPath);[1] <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> Then nothing happens when this line executes. No error is thrown as well. Any solution for this? [1] - https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 On 16 August 2015 at 18:06, Madawa Soysa <[email protected]> wrote: > Yes I was able to resolve the issue by removing RDD fields from the > SummaryModel object as @Mano pointed out. Still I have the same exception > when retrieving the model. Trying to fix that issue. > > On 14 August 2015 at 10:43, Nirmal Fernando <[email protected]> wrote: > >> Thanks Niranda, this doc is useful. >> >> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera <[email protected]> >> wrote: >> >>> From what I know, OneToOneDependancy come into play when spark tries to >>> create the RDD dependency tree. >>> >>> Just thought of sharing that. this would be a good resource [1] :-) >>> >>> [1] >>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>> >>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando <[email protected]> >>> wrote: >>> >>>> What is *org.apache.spark.OneToOneDependency ? Is it something you >>>> use?* >>>> >>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa <[email protected] >>>> > wrote: >>>> >>>>> Hi, >>>>> >>>>> I created a model summary in order to show the model data in the >>>>> analysis.jag page. >>>>> But when refreshing the page after building the model I get the >>>>> following error. >>>>> >>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: An >>>>> error has occurred while extracting all the models of analysis id: 13 >>>>> at >>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>> at >>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>> Caused by: >>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: An error >>>>> has occurred while extracting all the models of analysis id: 13 >>>>> at >>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>> at >>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>> ... 52 more >>>>> >>>>> *Caused by: java.lang.ClassNotFoundException: >>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>> at >>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>> at >>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>> ... 53 more >>>>> >>>>> I guess there is an error in persistence of the model summary object, >>>>> what should be the cause for this error? [1] >>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>> contains >>>>> the commit which I introduced the model summary. >>>>> >>>>> [1] - >>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>> >>>>> -- >>>>> >>>>> *_________________________________**Madawa Soysa* >>>>> >>>>> Undergraduate, >>>>> >>>>> Department of Computer Science and Engineering, >>>>> >>>>> University of Moratuwa. >>>>> >>>>> >>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>> [email protected] >>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> *Niranda Perera* >>> Software Engineer, WSO2 Inc. >>> Mobile: +94-71-554-8430 >>> Twitter: @n1r44 <https://twitter.com/N1R44> >>> https://pythagoreanscript.wordpress.com/ >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > > *_________________________________**Madawa Soysa* > > Undergraduate, > > Department of Computer Science and Engineering, > > University of Moratuwa. > > > Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: > [email protected] > LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter > <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> > -- *_________________________________**Madawa Soysa* Undergraduate, Department of Computer Science and Engineering, University of Moratuwa. Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: [email protected] LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
