Madawa, seems this is a Spark issue :-( can you try a simple Java program and see whether model.save() works?
On Sat, Aug 22, 2015 at 8:19 AM, Madawa Soysa <madawa...@cse.mrt.ac.lk> wrote: > Hi Nirmal, > > outPath is correct and the directory gets created, but the process becomes > idle after that. Attached the only part that was written to a file. > > Also the method doesn't throw an exception as well. > > On 21 August 2015 at 21:31, Nirmal Fernando <nir...@wso2.com> wrote: > >> Hi Madawa, >> >> According to Spark API [1], outPath shouldn't be exist. >> >> [1] >> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L200 >> >> On Fri, Aug 21, 2015 at 1:59 PM, Niranda Perera <nira...@wso2.com> wrote: >> >>> I don't think it's correct. Scala version is 2.10.4 even in the mvn repo >>> >>> On Fri, Aug 21, 2015, 13:46 Madawa Soysa <madawa...@cse.mrt.ac.lk> >>> wrote: >>> >>>> Also I asked this question in StackOverflow[1] >>>> <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> >>>> and there they have mentioned a version incompatibility between Scala and >>>> Spark versions >>>> >>>> [1] - >>>> http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java >>>> >>>> On 21 August 2015 at 13:31, Madawa Soysa <madawa...@cse.mrt.ac.lk> >>>> wrote: >>>> >>>>> Yes path is valid, I explicitly set the path here from the >>>>> MLModelHandler persistModel method. >>>>> >>>>> On 21 August 2015 at 10:26, Nirmal Fernando <nir...@wso2.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa < >>>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> There an issue with serializing Spark's MatrixFactorizationModel >>>>>>> object. The object contains a huge RDD and as I have read in many blogs, >>>>>>> this model cannot be serialized as a java object. Therefore when >>>>>>> retrieving >>>>>>> the model I get the same exception as above; >>>>>>> >>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>> >>>>>>> I have asked this question in Spark mailing lists and they >>>>>>> recommended me to use the built in save and load functions other than >>>>>>> using >>>>>>> Java serializing. So I have used following method to persist the model, >>>>>>> >>>>>>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>>>>>> outPath);[1] >>>>>>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>>>>>> >>>>>>> Then nothing happens when this line executes. No error is thrown as >>>>>>> well. Any solution for this? >>>>>>> >>>>>> >>>>>> Can you print outPath and see whether it's a valid file path? >>>>>> >>>>>> >>>>>>> >>>>>>> [1] - >>>>>>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>>>>>> >>>>>>> On 16 August 2015 at 18:06, Madawa Soysa <madawa...@cse.mrt.ac.lk> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes I was able to resolve the issue by removing RDD fields from the >>>>>>>> SummaryModel object as @Mano pointed out. Still I have the same >>>>>>>> exception >>>>>>>> when retrieving the model. Trying to fix that issue. >>>>>>>> >>>>>>>> On 14 August 2015 at 10:43, Nirmal Fernando <nir...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks Niranda, this doc is useful. >>>>>>>>> >>>>>>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera <nira...@wso2.com >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> From what I know, OneToOneDependancy come into play when spark >>>>>>>>>> tries to create the RDD dependency tree. >>>>>>>>>> >>>>>>>>>> Just thought of sharing that. this would be a good resource [1] >>>>>>>>>> :-) >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>>>>>> >>>>>>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando < >>>>>>>>>> nir...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it something >>>>>>>>>>> you use?* >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>>>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I created a model summary in order to show the model data in >>>>>>>>>>>> the analysis.jag page. >>>>>>>>>>>> But when refreshing the page after building the model I get the >>>>>>>>>>>> following error. >>>>>>>>>>>> >>>>>>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: >>>>>>>>>>>> An error has occurred while extracting all the models of analysis >>>>>>>>>>>> id: 13 >>>>>>>>>>>> at >>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>>>>>> at >>>>>>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>>>>>> Caused by: >>>>>>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: >>>>>>>>>>>> An error >>>>>>>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>>>>>>> at >>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>>>>>> at >>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>>>>>> ... 52 more >>>>>>>>>>>> >>>>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>>>> at >>>>>>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>>>>>> at >>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>>>>>> ... 53 more >>>>>>>>>>>> >>>>>>>>>>>> I guess there is an error in persistence of the model summary >>>>>>>>>>>> object, what should be the cause for this error? [1] >>>>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>>>>>> contains >>>>>>>>>>>> the commit which I introduced the model summary. >>>>>>>>>>>> >>>>>>>>>>>> [1] - >>>>>>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>>> >>>>>>>>>>>> Undergraduate, >>>>>>>>>>>> >>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>> >>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Thanks & regards, >>>>>>>>>>> Nirmal >>>>>>>>>>> >>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> Dev@wso2.org >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Niranda Perera* >>>>>>>>>> Software Engineer, WSO2 Inc. >>>>>>>>>> Mobile: +94-71-554-8430 >>>>>>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>>>>>> https://pythagoreanscript.wordpress.com/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks & regards, >>>>>>>>> Nirmal >>>>>>>>> >>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>> Mobile: +94715779733 >>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>> >>>>>>>> Undergraduate, >>>>>>>> >>>>>>>> Department of Computer Science and Engineering, >>>>>>>> >>>>>>>> University of Moratuwa. >>>>>>>> >>>>>>>> >>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>> <http://madawas.tumblr.com/> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *_________________________________**Madawa Soysa* >>>>>>> >>>>>>> Undergraduate, >>>>>>> >>>>>>> Department of Computer Science and Engineering, >>>>>>> >>>>>>> University of Moratuwa. >>>>>>> >>>>>>> >>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>> madawa...@cse.mrt.ac.lk >>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>> <http://madawas.tumblr.com/> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Team Lead - WSO2 Machine Learner >>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *_________________________________**Madawa Soysa* >>>>> >>>>> Undergraduate, >>>>> >>>>> Department of Computer Science and Engineering, >>>>> >>>>> University of Moratuwa. >>>>> >>>>> >>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>> madawa...@cse.mrt.ac.lk >>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> *_________________________________**Madawa Soysa* >>>> >>>> Undergraduate, >>>> >>>> Department of Computer Science and Engineering, >>>> >>>> University of Moratuwa. >>>> >>>> >>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>> madawa...@cse.mrt.ac.lk >>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>> >>> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > > *_________________________________**Madawa Soysa* > > Undergraduate, > > Department of Computer Science and Engineering, > > University of Moratuwa. > > > Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: > madawa...@cse.mrt.ac.lk > LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter > <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
