Hi Nirmal, Not actually. I have added parquet-common, parquet-encoding, parquet-column, parquet-hadoop as mentioned at https://github.com/Parquet/parquet-mr <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FParquet%2Fparquet-mr&sa=D&sntz=1&usg=AFQjCNEHFd3g0eqXClZCkN35QgFYl0FVww> . But still get the following exception even though the class parquet.hadoop.metadata.CompressionCodecName is there in the parquet-hadoop.jar
java.lang.ClassNotFoundException: parquet.hadoop.metadata.CompressionCodecName cannot be found by spark-sql_2.10_1.4.1.wso2v1 I kept trying with different versions, still the issue is there. On 26 August 2015 at 08:27, Nirmal Fernando <[email protected]> wrote: > Madawa, hope the issue is resolved now after the instructions given > offline. > > On Mon, Aug 24, 2015 at 8:34 PM, Madawa Soysa <[email protected]> > wrote: > >> I have tested the model.save() with a simple Java program. It works fine. >> >> I have noticed that scala-library:2.11.6 is a dependency of >> spark:spark-core_2.11:1.4.1 [1] >> <http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar> >> In ML scala version is explicitly specified as 2.10.4. Is there a specific >> reason to use scala 2.10.4? I guess this version incompatibility could be >> the reason for this issue. >> >> [1] - >> http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar >> >> On 24 August 2015 at 10:24, Nirmal Fernando <[email protected]> wrote: >> >>> Madawa, seems this is a Spark issue :-( can you try a simple Java >>> program and see whether model.save() works? >>> >>> On Sat, Aug 22, 2015 at 8:19 AM, Madawa Soysa <[email protected]> >>> wrote: >>> >>>> Hi Nirmal, >>>> >>>> outPath is correct and the directory gets created, but the process >>>> becomes idle after that. Attached the only part that was written to a file. >>>> >>>> Also the method doesn't throw an exception as well. >>>> >>>> On 21 August 2015 at 21:31, Nirmal Fernando <[email protected]> wrote: >>>> >>>>> Hi Madawa, >>>>> >>>>> According to Spark API [1], outPath shouldn't be exist. >>>>> >>>>> [1] >>>>> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L200 >>>>> >>>>> On Fri, Aug 21, 2015 at 1:59 PM, Niranda Perera <[email protected]> >>>>> wrote: >>>>> >>>>>> I don't think it's correct. Scala version is 2.10.4 even in the mvn >>>>>> repo >>>>>> >>>>>> On Fri, Aug 21, 2015, 13:46 Madawa Soysa <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Also I asked this question in StackOverflow[1] >>>>>>> <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> >>>>>>> and there they have mentioned a version incompatibility between Scala >>>>>>> and >>>>>>> Spark versions >>>>>>> >>>>>>> [1] - >>>>>>> http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java >>>>>>> >>>>>>> On 21 August 2015 at 13:31, Madawa Soysa <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes path is valid, I explicitly set the path here from the >>>>>>>> MLModelHandler persistModel method. >>>>>>>> >>>>>>>> On 21 August 2015 at 10:26, Nirmal Fernando <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> There an issue with serializing Spark's MatrixFactorizationModel >>>>>>>>>> object. The object contains a huge RDD and as I have read in many >>>>>>>>>> blogs, >>>>>>>>>> this model cannot be serialized as a java object. Therefore when >>>>>>>>>> retrieving >>>>>>>>>> the model I get the same exception as above; >>>>>>>>>> >>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>> >>>>>>>>>> I have asked this question in Spark mailing lists and they >>>>>>>>>> recommended me to use the built in save and load functions other >>>>>>>>>> than using >>>>>>>>>> Java serializing. So I have used following method to persist the >>>>>>>>>> model, >>>>>>>>>> >>>>>>>>>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>>>>>>>>> outPath);[1] >>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>>>>>>>>> >>>>>>>>>> Then nothing happens when this line executes. No error is thrown >>>>>>>>>> as well. Any solution for this? >>>>>>>>>> >>>>>>>>> >>>>>>>>> Can you print outPath and see whether it's a valid file path? >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] - >>>>>>>>>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>>>>>>>>> >>>>>>>>>> On 16 August 2015 at 18:06, Madawa Soysa <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Yes I was able to resolve the issue by removing RDD fields from >>>>>>>>>>> the SummaryModel object as @Mano pointed out. Still I have the same >>>>>>>>>>> exception when retrieving the model. Trying to fix that issue. >>>>>>>>>>> >>>>>>>>>>> On 14 August 2015 at 10:43, Nirmal Fernando <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks Niranda, this doc is useful. >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> From what I know, OneToOneDependancy come into play when spark >>>>>>>>>>>>> tries to create the RDD dependency tree. >>>>>>>>>>>>> >>>>>>>>>>>>> Just thought of sharing that. this would be a good resource >>>>>>>>>>>>> [1] :-) >>>>>>>>>>>>> >>>>>>>>>>>>> [1] >>>>>>>>>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it >>>>>>>>>>>>>> something you use?* >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I created a model summary in order to show the model data in >>>>>>>>>>>>>>> the analysis.jag page. >>>>>>>>>>>>>>> But when refreshing the page after building the model I get >>>>>>>>>>>>>>> the following error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: >>>>>>>>>>>>>>> An error has occurred while extracting all the models of >>>>>>>>>>>>>>> analysis id: 13 >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>>>>>>>>> Caused by: >>>>>>>>>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: >>>>>>>>>>>>>>> An error >>>>>>>>>>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>>>>>>>>> ... 52 more >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>>>>>>>>> ... 53 more >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I guess there is an error in persistence of the model >>>>>>>>>>>>>>> summary object, what should be the cause for this error? [1] >>>>>>>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>>>>>>>>> contains >>>>>>>>>>>>>>> the commit which I introduced the model summary. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] - >>>>>>>>>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Undergraduate, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>> >>>>>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Niranda Perera* >>>>>>>>>>>>> Software Engineer, WSO2 Inc. >>>>>>>>>>>>> Mobile: +94-71-554-8430 >>>>>>>>>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>>>>>>>>> https://pythagoreanscript.wordpress.com/ >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>> Nirmal >>>>>>>>>>>> >>>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>> >>>>>>>>>>> Undergraduate, >>>>>>>>>>> >>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>> >>>>>>>>>>> University of Moratuwa. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>> [email protected] >>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>> >>>>>>>>>> Undergraduate, >>>>>>>>>> >>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>> >>>>>>>>>> University of Moratuwa. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>> [email protected] >>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks & regards, >>>>>>>>> Nirmal >>>>>>>>> >>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>> Mobile: +94715779733 >>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>> >>>>>>>> Undergraduate, >>>>>>>> >>>>>>>> Department of Computer Science and Engineering, >>>>>>>> >>>>>>>> University of Moratuwa. >>>>>>>> >>>>>>>> >>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>> [email protected] >>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>> <http://madawas.tumblr.com/> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *_________________________________**Madawa Soysa* >>>>>>> >>>>>>> Undergraduate, >>>>>>> >>>>>>> Department of Computer Science and Engineering, >>>>>>> >>>>>>> University of Moratuwa. >>>>>>> >>>>>>> >>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>> [email protected] >>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>> <http://madawas.tumblr.com/> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Team Lead - WSO2 Machine Learner >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *_________________________________**Madawa Soysa* >>>> >>>> Undergraduate, >>>> >>>> Department of Computer Science and Engineering, >>>> >>>> University of Moratuwa. >>>> >>>> >>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>> [email protected] >>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> >> *_________________________________**Madawa Soysa* >> >> Undergraduate, >> >> Department of Computer Science and Engineering, >> >> University of Moratuwa. >> >> >> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >> [email protected] >> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *_________________________________**Madawa Soysa* Undergraduate, Department of Computer Science and Engineering, University of Moratuwa. Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: [email protected] LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
