Hi Nirmal, outPath is correct and the directory gets created, but the process becomes idle after that. Attached the only part that was written to a file.
Also the method doesn't throw an exception as well. On 21 August 2015 at 21:31, Nirmal Fernando <nir...@wso2.com> wrote: > Hi Madawa, > > According to Spark API [1], outPath shouldn't be exist. > > [1] > https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L200 > > On Fri, Aug 21, 2015 at 1:59 PM, Niranda Perera <nira...@wso2.com> wrote: > >> I don't think it's correct. Scala version is 2.10.4 even in the mvn repo >> >> On Fri, Aug 21, 2015, 13:46 Madawa Soysa <madawa...@cse.mrt.ac.lk> wrote: >> >>> Also I asked this question in StackOverflow[1] >>> <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> >>> and there they have mentioned a version incompatibility between Scala and >>> Spark versions >>> >>> [1] - >>> http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java >>> >>> On 21 August 2015 at 13:31, Madawa Soysa <madawa...@cse.mrt.ac.lk> >>> wrote: >>> >>>> Yes path is valid, I explicitly set the path here from the >>>> MLModelHandler persistModel method. >>>> >>>> On 21 August 2015 at 10:26, Nirmal Fernando <nir...@wso2.com> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa <madawa...@cse.mrt.ac.lk >>>>> > wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> There an issue with serializing Spark's MatrixFactorizationModel >>>>>> object. The object contains a huge RDD and as I have read in many blogs, >>>>>> this model cannot be serialized as a java object. Therefore when >>>>>> retrieving >>>>>> the model I get the same exception as above; >>>>>> >>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>> >>>>>> I have asked this question in Spark mailing lists and they >>>>>> recommended me to use the built in save and load functions other than >>>>>> using >>>>>> Java serializing. So I have used following method to persist the model, >>>>>> >>>>>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>>>>> outPath);[1] >>>>>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>>>>> >>>>>> Then nothing happens when this line executes. No error is thrown as >>>>>> well. Any solution for this? >>>>>> >>>>> >>>>> Can you print outPath and see whether it's a valid file path? >>>>> >>>>> >>>>>> >>>>>> [1] - >>>>>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>>>>> >>>>>> On 16 August 2015 at 18:06, Madawa Soysa <madawa...@cse.mrt.ac.lk> >>>>>> wrote: >>>>>> >>>>>>> Yes I was able to resolve the issue by removing RDD fields from the >>>>>>> SummaryModel object as @Mano pointed out. Still I have the same >>>>>>> exception >>>>>>> when retrieving the model. Trying to fix that issue. >>>>>>> >>>>>>> On 14 August 2015 at 10:43, Nirmal Fernando <nir...@wso2.com> wrote: >>>>>>> >>>>>>>> Thanks Niranda, this doc is useful. >>>>>>>> >>>>>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera <nira...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> From what I know, OneToOneDependancy come into play when spark >>>>>>>>> tries to create the RDD dependency tree. >>>>>>>>> >>>>>>>>> Just thought of sharing that. this would be a good resource [1] >>>>>>>>> :-) >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>>>>> >>>>>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando <nir...@wso2.com >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it something >>>>>>>>>> you use?* >>>>>>>>>> >>>>>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I created a model summary in order to show the model data in the >>>>>>>>>>> analysis.jag page. >>>>>>>>>>> But when refreshing the page after building the model I get the >>>>>>>>>>> following error. >>>>>>>>>>> >>>>>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: >>>>>>>>>>> An error has occurred while extracting all the models of analysis >>>>>>>>>>> id: 13 >>>>>>>>>>> at >>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>>>>> at >>>>>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>>>>> Caused by: >>>>>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: >>>>>>>>>>> An error >>>>>>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>>>>>> at >>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>>>>> at >>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>>>>> ... 52 more >>>>>>>>>>> >>>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>>> at >>>>>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>>>>> at >>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>>>>> ... 53 more >>>>>>>>>>> >>>>>>>>>>> I guess there is an error in persistence of the model summary >>>>>>>>>>> object, what should be the cause for this error? [1] >>>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>>>>> contains >>>>>>>>>>> the commit which I introduced the model summary. >>>>>>>>>>> >>>>>>>>>>> [1] - >>>>>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>> >>>>>>>>>>> Undergraduate, >>>>>>>>>>> >>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>> >>>>>>>>>>> University of Moratuwa. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks & regards, >>>>>>>>>> Nirmal >>>>>>>>>> >>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>> Mobile: +94715779733 >>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> Dev@wso2.org >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Niranda Perera* >>>>>>>>> Software Engineer, WSO2 Inc. >>>>>>>>> Mobile: +94-71-554-8430 >>>>>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>>>>> https://pythagoreanscript.wordpress.com/ >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks & regards, >>>>>>>> Nirmal >>>>>>>> >>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>> Mobile: +94715779733 >>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *_________________________________**Madawa Soysa* >>>>>>> >>>>>>> Undergraduate, >>>>>>> >>>>>>> Department of Computer Science and Engineering, >>>>>>> >>>>>>> University of Moratuwa. >>>>>>> >>>>>>> >>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>> madawa...@cse.mrt.ac.lk >>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>> <http://madawas.tumblr.com/> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *_________________________________**Madawa Soysa* >>>>>> >>>>>> Undergraduate, >>>>>> >>>>>> Department of Computer Science and Engineering, >>>>>> >>>>>> University of Moratuwa. >>>>>> >>>>>> >>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>> madawa...@cse.mrt.ac.lk >>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Team Lead - WSO2 Machine Learner >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *_________________________________**Madawa Soysa* >>>> >>>> Undergraduate, >>>> >>>> Department of Computer Science and Engineering, >>>> >>>> University of Moratuwa. >>>> >>>> >>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>> madawa...@cse.mrt.ac.lk >>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>> >>> >>> >>> >>> -- >>> >>> *_________________________________**Madawa Soysa* >>> >>> Undergraduate, >>> >>> Department of Computer Science and Engineering, >>> >>> University of Moratuwa. >>> >>> >>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>> madawa...@cse.mrt.ac.lk >>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>> >> > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *_________________________________**Madawa Soysa* Undergraduate, Department of Computer Science and Engineering, University of Moratuwa. Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: madawa...@cse.mrt.ac.lk LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/>
part-00000
Description: Binary data
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
