I don't think it's correct. Scala version is 2.10.4 even in the mvn repo On Fri, Aug 21, 2015, 13:46 Madawa Soysa <[email protected]> wrote:
> Also I asked this question in StackOverflow[1] > <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> > and there they have mentioned a version incompatibility between Scala and > Spark versions > > [1] - > http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java > > On 21 August 2015 at 13:31, Madawa Soysa <[email protected]> wrote: > >> Yes path is valid, I explicitly set the path here from the >> MLModelHandler persistModel method. >> >> On 21 August 2015 at 10:26, Nirmal Fernando <[email protected]> wrote: >> >>> >>> >>> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa <[email protected]> >>> wrote: >>> >>>> Hi All, >>>> >>>> There an issue with serializing Spark's MatrixFactorizationModel >>>> object. The object contains a huge RDD and as I have read in many blogs, >>>> this model cannot be serialized as a java object. Therefore when retrieving >>>> the model I get the same exception as above; >>>> >>>> *Caused by: java.lang.ClassNotFoundException: >>>> org.apache.spark.OneToOneDependency cannot be found by >>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>> >>>> I have asked this question in Spark mailing lists and they recommended >>>> me to use the built in save and load functions other than using Java >>>> serializing. So I have used following method to persist the model, >>>> >>>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>>> outPath);[1] >>>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>>> >>>> Then nothing happens when this line executes. No error is thrown as >>>> well. Any solution for this? >>>> >>> >>> Can you print outPath and see whether it's a valid file path? >>> >>> >>>> >>>> [1] - >>>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>>> >>>> On 16 August 2015 at 18:06, Madawa Soysa <[email protected]> >>>> wrote: >>>> >>>>> Yes I was able to resolve the issue by removing RDD fields from the >>>>> SummaryModel object as @Mano pointed out. Still I have the same exception >>>>> when retrieving the model. Trying to fix that issue. >>>>> >>>>> On 14 August 2015 at 10:43, Nirmal Fernando <[email protected]> wrote: >>>>> >>>>>> Thanks Niranda, this doc is useful. >>>>>> >>>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> From what I know, OneToOneDependancy come into play when spark tries >>>>>>> to create the RDD dependency tree. >>>>>>> >>>>>>> Just thought of sharing that. this would be a good resource [1] :-) >>>>>>> >>>>>>> [1] >>>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>>> >>>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it something you >>>>>>>> use?* >>>>>>>> >>>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I created a model summary in order to show the model data in the >>>>>>>>> analysis.jag page. >>>>>>>>> But when refreshing the page after building the model I get the >>>>>>>>> following error. >>>>>>>>> >>>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: An >>>>>>>>> error has occurred while extracting all the models of analysis id: 13 >>>>>>>>> at >>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>>> Caused by: >>>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: An >>>>>>>>> error >>>>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>>>> at >>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>>> ... 52 more >>>>>>>>> >>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>> at >>>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>>> ... 53 more >>>>>>>>> >>>>>>>>> I guess there is an error in persistence of the model summary >>>>>>>>> object, what should be the cause for this error? [1] >>>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>>> contains >>>>>>>>> the commit which I introduced the model summary. >>>>>>>>> >>>>>>>>> [1] - >>>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>> >>>>>>>>> Undergraduate, >>>>>>>>> >>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>> >>>>>>>>> University of Moratuwa. >>>>>>>>> >>>>>>>>> >>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>> [email protected] >>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks & regards, >>>>>>>> Nirmal >>>>>>>> >>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>> Mobile: +94715779733 >>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Niranda Perera* >>>>>>> Software Engineer, WSO2 Inc. >>>>>>> Mobile: +94-71-554-8430 >>>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>>> https://pythagoreanscript.wordpress.com/ >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Team Lead - WSO2 Machine Learner >>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *_________________________________**Madawa Soysa* >>>>> >>>>> Undergraduate, >>>>> >>>>> Department of Computer Science and Engineering, >>>>> >>>>> University of Moratuwa. >>>>> >>>>> >>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>> [email protected] >>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> *_________________________________**Madawa Soysa* >>>> >>>> Undergraduate, >>>> >>>> Department of Computer Science and Engineering, >>>> >>>> University of Moratuwa. >>>> >>>> >>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>> [email protected] >>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> >> *_________________________________**Madawa Soysa* >> >> Undergraduate, >> >> Department of Computer Science and Engineering, >> >> University of Moratuwa. >> >> >> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >> [email protected] >> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >> > > > > -- > > *_________________________________**Madawa Soysa* > > Undergraduate, > > Department of Computer Science and Engineering, > > University of Moratuwa. > > > Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: > [email protected] > LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter > <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
