Also I asked this question in StackOverflow[1] <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> and there they have mentioned a version incompatibility between Scala and Spark versions
[1] - http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java On 21 August 2015 at 13:31, Madawa Soysa <[email protected]> wrote: > Yes path is valid, I explicitly set the path here from the > MLModelHandler persistModel method. > > On 21 August 2015 at 10:26, Nirmal Fernando <[email protected]> wrote: > >> >> >> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa <[email protected]> >> wrote: >> >>> Hi All, >>> >>> There an issue with serializing Spark's MatrixFactorizationModel object. >>> The object contains a huge RDD and as I have read in many blogs, this model >>> cannot be serialized as a java object. Therefore when retrieving the model >>> I get the same exception as above; >>> >>> *Caused by: java.lang.ClassNotFoundException: >>> org.apache.spark.OneToOneDependency cannot be found by >>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>> >>> I have asked this question in Spark mailing lists and they recommended >>> me to use the built in save and load functions other than using Java >>> serializing. So I have used following method to persist the model, >>> >>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>> outPath);[1] >>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>> >>> Then nothing happens when this line executes. No error is thrown as >>> well. Any solution for this? >>> >> >> Can you print outPath and see whether it's a valid file path? >> >> >>> >>> [1] - >>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>> >>> On 16 August 2015 at 18:06, Madawa Soysa <[email protected]> >>> wrote: >>> >>>> Yes I was able to resolve the issue by removing RDD fields from the >>>> SummaryModel object as @Mano pointed out. Still I have the same exception >>>> when retrieving the model. Trying to fix that issue. >>>> >>>> On 14 August 2015 at 10:43, Nirmal Fernando <[email protected]> wrote: >>>> >>>>> Thanks Niranda, this doc is useful. >>>>> >>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera <[email protected]> >>>>> wrote: >>>>> >>>>>> From what I know, OneToOneDependancy come into play when spark tries >>>>>> to create the RDD dependency tree. >>>>>> >>>>>> Just thought of sharing that. this would be a good resource [1] :-) >>>>>> >>>>>> [1] >>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>> >>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it something you >>>>>>> use?* >>>>>>> >>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I created a model summary in order to show the model data in the >>>>>>>> analysis.jag page. >>>>>>>> But when refreshing the page after building the model I get the >>>>>>>> following error. >>>>>>>> >>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: An >>>>>>>> error has occurred while extracting all the models of analysis id: 13 >>>>>>>> at >>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>> at >>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>> Caused by: >>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: An >>>>>>>> error >>>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>>> at >>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>> at >>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>> ... 52 more >>>>>>>> >>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>> at >>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>> at >>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>> ... 53 more >>>>>>>> >>>>>>>> I guess there is an error in persistence of the model summary >>>>>>>> object, what should be the cause for this error? [1] >>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>> contains >>>>>>>> the commit which I introduced the model summary. >>>>>>>> >>>>>>>> [1] - >>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>> >>>>>>>> Undergraduate, >>>>>>>> >>>>>>>> Department of Computer Science and Engineering, >>>>>>>> >>>>>>>> University of Moratuwa. >>>>>>>> >>>>>>>> >>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>> [email protected] >>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>> <http://madawas.tumblr.com/> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks & regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Team Lead - WSO2 Machine Learner >>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>> Mobile: +94715779733 >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Niranda Perera* >>>>>> Software Engineer, WSO2 Inc. >>>>>> Mobile: +94-71-554-8430 >>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>> https://pythagoreanscript.wordpress.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Team Lead - WSO2 Machine Learner >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *_________________________________**Madawa Soysa* >>>> >>>> Undergraduate, >>>> >>>> Department of Computer Science and Engineering, >>>> >>>> University of Moratuwa. >>>> >>>> >>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>> [email protected] >>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>> >>> >>> >>> >>> -- >>> >>> *_________________________________**Madawa Soysa* >>> >>> Undergraduate, >>> >>> Department of Computer Science and Engineering, >>> >>> University of Moratuwa. >>> >>> >>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>> [email protected] >>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > > *_________________________________**Madawa Soysa* > > Undergraduate, > > Department of Computer Science and Engineering, > > University of Moratuwa. > > > Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: > [email protected] > LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter > <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> > -- *_________________________________**Madawa Soysa* Undergraduate, Department of Computer Science and Engineering, University of Moratuwa. Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: [email protected] LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
