I have tested the model.save() with a simple Java program. It works fine. I have noticed that scala-library:2.11.6 is a dependency of spark:spark-core_2.11:1.4.1 [1] <http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar> In ML scala version is explicitly specified as 2.10.4. Is there a specific reason to use scala 2.10.4? I guess this version incompatibility could be the reason for this issue.
[1] - http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar On 24 August 2015 at 10:24, Nirmal Fernando <[email protected]> wrote: > Madawa, seems this is a Spark issue :-( can you try a simple Java program > and see whether model.save() works? > > On Sat, Aug 22, 2015 at 8:19 AM, Madawa Soysa <[email protected]> > wrote: > >> Hi Nirmal, >> >> outPath is correct and the directory gets created, but the process >> becomes idle after that. Attached the only part that was written to a file. >> >> Also the method doesn't throw an exception as well. >> >> On 21 August 2015 at 21:31, Nirmal Fernando <[email protected]> wrote: >> >>> Hi Madawa, >>> >>> According to Spark API [1], outPath shouldn't be exist. >>> >>> [1] >>> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L200 >>> >>> On Fri, Aug 21, 2015 at 1:59 PM, Niranda Perera <[email protected]> >>> wrote: >>> >>>> I don't think it's correct. Scala version is 2.10.4 even in the mvn repo >>>> >>>> On Fri, Aug 21, 2015, 13:46 Madawa Soysa <[email protected]> >>>> wrote: >>>> >>>>> Also I asked this question in StackOverflow[1] >>>>> <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> >>>>> and there they have mentioned a version incompatibility between Scala and >>>>> Spark versions >>>>> >>>>> [1] - >>>>> http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java >>>>> >>>>> On 21 August 2015 at 13:31, Madawa Soysa <[email protected]> >>>>> wrote: >>>>> >>>>>> Yes path is valid, I explicitly set the path here from the >>>>>> MLModelHandler persistModel method. >>>>>> >>>>>> On 21 August 2015 at 10:26, Nirmal Fernando <[email protected]> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> There an issue with serializing Spark's MatrixFactorizationModel >>>>>>>> object. The object contains a huge RDD and as I have read in many >>>>>>>> blogs, >>>>>>>> this model cannot be serialized as a java object. Therefore when >>>>>>>> retrieving >>>>>>>> the model I get the same exception as above; >>>>>>>> >>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>> >>>>>>>> I have asked this question in Spark mailing lists and they >>>>>>>> recommended me to use the built in save and load functions other than >>>>>>>> using >>>>>>>> Java serializing. So I have used following method to persist the >>>>>>>> model, >>>>>>>> >>>>>>>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>>>>>>> outPath);[1] >>>>>>>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>>>>>>> >>>>>>>> Then nothing happens when this line executes. No error is thrown as >>>>>>>> well. Any solution for this? >>>>>>>> >>>>>>> >>>>>>> Can you print outPath and see whether it's a valid file path? >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> [1] - >>>>>>>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>>>>>>> >>>>>>>> On 16 August 2015 at 18:06, Madawa Soysa <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes I was able to resolve the issue by removing RDD fields from >>>>>>>>> the SummaryModel object as @Mano pointed out. Still I have the same >>>>>>>>> exception when retrieving the model. Trying to fix that issue. >>>>>>>>> >>>>>>>>> On 14 August 2015 at 10:43, Nirmal Fernando <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks Niranda, this doc is useful. >>>>>>>>>> >>>>>>>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> From what I know, OneToOneDependancy come into play when spark >>>>>>>>>>> tries to create the RDD dependency tree. >>>>>>>>>>> >>>>>>>>>>> Just thought of sharing that. this would be a good resource [1] >>>>>>>>>>> :-) >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it something >>>>>>>>>>>> you use?* >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I created a model summary in order to show the model data in >>>>>>>>>>>>> the analysis.jag page. >>>>>>>>>>>>> But when refreshing the page after building the model I get >>>>>>>>>>>>> the following error. >>>>>>>>>>>>> >>>>>>>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: >>>>>>>>>>>>> An error has occurred while extracting all the models of >>>>>>>>>>>>> analysis id: 13 >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>>>>>>> Caused by: >>>>>>>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: >>>>>>>>>>>>> An error >>>>>>>>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>>>>>>> ... 52 more >>>>>>>>>>>>> >>>>>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>>>>>>> ... 53 more >>>>>>>>>>>>> >>>>>>>>>>>>> I guess there is an error in persistence of the model summary >>>>>>>>>>>>> object, what should be the cause for this error? [1] >>>>>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>>>>>>> contains >>>>>>>>>>>>> the commit which I introduced the model summary. >>>>>>>>>>>>> >>>>>>>>>>>>> [1] - >>>>>>>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>>>> >>>>>>>>>>>>> Undergraduate, >>>>>>>>>>>>> >>>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>>> >>>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>> Nirmal >>>>>>>>>>>> >>>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Dev mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Niranda Perera* >>>>>>>>>>> Software Engineer, WSO2 Inc. >>>>>>>>>>> Mobile: +94-71-554-8430 >>>>>>>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>>>>>>> https://pythagoreanscript.wordpress.com/ >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks & regards, >>>>>>>>>> Nirmal >>>>>>>>>> >>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>> Mobile: +94715779733 >>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>> >>>>>>>>> Undergraduate, >>>>>>>>> >>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>> >>>>>>>>> University of Moratuwa. >>>>>>>>> >>>>>>>>> >>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>> [email protected] >>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>> >>>>>>>> Undergraduate, >>>>>>>> >>>>>>>> Department of Computer Science and Engineering, >>>>>>>> >>>>>>>> University of Moratuwa. >>>>>>>> >>>>>>>> >>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>> [email protected] >>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>> <http://madawas.tumblr.com/> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks & regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Team Lead - WSO2 Machine Learner >>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>> Mobile: +94715779733 >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *_________________________________**Madawa Soysa* >>>>>> >>>>>> Undergraduate, >>>>>> >>>>>> Department of Computer Science and Engineering, >>>>>> >>>>>> University of Moratuwa. >>>>>> >>>>>> >>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>> [email protected] >>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *_________________________________**Madawa Soysa* >>>>> >>>>> Undergraduate, >>>>> >>>>> Department of Computer Science and Engineering, >>>>> >>>>> University of Moratuwa. >>>>> >>>>> >>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>> [email protected] >>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>> >>>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> >> *_________________________________**Madawa Soysa* >> >> Undergraduate, >> >> Department of Computer Science and Engineering, >> >> University of Moratuwa. >> >> >> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >> [email protected] >> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *_________________________________**Madawa Soysa* Undergraduate, Department of Computer Science and Engineering, University of Moratuwa. Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: [email protected] LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
