On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa <madawa...@cse.mrt.ac.lk> wrote:
> Hi All, > > There an issue with serializing Spark's MatrixFactorizationModel object. > The object contains a huge RDD and as I have read in many blogs, this model > cannot be serialized as a java object. Therefore when retrieving the model > I get the same exception as above; > > *Caused by: java.lang.ClassNotFoundException: > org.apache.spark.OneToOneDependency cannot be found by > org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* > > I have asked this question in Spark mailing lists and they recommended me > to use the built in save and load functions other than using Java > serializing. So I have used following method to persist the model, > > model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), > outPath);[1] > <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> > > Then nothing happens when this line executes. No error is thrown as well. > Any solution for this? > Can you print outPath and see whether it's a valid file path? > > [1] - > https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 > > On 16 August 2015 at 18:06, Madawa Soysa <madawa...@cse.mrt.ac.lk> wrote: > >> Yes I was able to resolve the issue by removing RDD fields from the >> SummaryModel object as @Mano pointed out. Still I have the same exception >> when retrieving the model. Trying to fix that issue. >> >> On 14 August 2015 at 10:43, Nirmal Fernando <nir...@wso2.com> wrote: >> >>> Thanks Niranda, this doc is useful. >>> >>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera <nira...@wso2.com> >>> wrote: >>> >>>> From what I know, OneToOneDependancy come into play when spark tries to >>>> create the RDD dependency tree. >>>> >>>> Just thought of sharing that. this would be a good resource [1] :-) >>>> >>>> [1] >>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>> >>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando <nir...@wso2.com> >>>> wrote: >>>> >>>>> What is *org.apache.spark.OneToOneDependency ? Is it something you >>>>> use?* >>>>> >>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I created a model summary in order to show the model data in the >>>>>> analysis.jag page. >>>>>> But when refreshing the page after building the model I get the >>>>>> following error. >>>>>> >>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: An >>>>>> error has occurred while extracting all the models of analysis id: 13 >>>>>> at >>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>> at >>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>> Caused by: >>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: An >>>>>> error >>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>> at >>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>> at >>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>> ... 52 more >>>>>> >>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>> at >>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>> at >>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>> ... 53 more >>>>>> >>>>>> I guess there is an error in persistence of the model summary object, >>>>>> what should be the cause for this error? [1] >>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>> contains >>>>>> the commit which I introduced the model summary. >>>>>> >>>>>> [1] - >>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>> >>>>>> -- >>>>>> >>>>>> *_________________________________**Madawa Soysa* >>>>>> >>>>>> Undergraduate, >>>>>> >>>>>> Department of Computer Science and Engineering, >>>>>> >>>>>> University of Moratuwa. >>>>>> >>>>>> >>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>> madawa...@cse.mrt.ac.lk >>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Team Lead - WSO2 Machine Learner >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> Dev@wso2.org >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Niranda Perera* >>>> Software Engineer, WSO2 Inc. >>>> Mobile: +94-71-554-8430 >>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>> https://pythagoreanscript.wordpress.com/ >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> >> *_________________________________**Madawa Soysa* >> >> Undergraduate, >> >> Department of Computer Science and Engineering, >> >> University of Moratuwa. >> >> >> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >> madawa...@cse.mrt.ac.lk >> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >> > > > > -- > > *_________________________________**Madawa Soysa* > > Undergraduate, > > Department of Computer Science and Engineering, > > University of Moratuwa. > > > Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: > madawa...@cse.mrt.ac.lk > LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter > <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
