Gives following output. parquet.hadoop.metadata; version="0.0.0"<parquet_common_1.6.0_1.0.0 [321]> parquet.hadoop.metadata; version="0.0.0"<parquet_hadoop_1.6.0_1.0.0 [323]>
On 26 August 2015 at 08:35, Nirmal Fernando <[email protected]> wrote: > You added them to repository/components/lib right? Can you start the > server in OSGi mode (./wso2server.sh -DosgiConsole ) and issue following > command in OSGi console; > > osgi> p parquet.hadoop.metadata > > On Wed, Aug 26, 2015 at 8:32 AM, Madawa Soysa <[email protected]> > wrote: > >> Hi Nirmal, >> >> Not actually. I have added parquet-common, parquet-encoding, >> parquet-column, parquet-hadoop as mentioned at >> https://github.com/Parquet/parquet-mr >> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FParquet%2Fparquet-mr&sa=D&sntz=1&usg=AFQjCNEHFd3g0eqXClZCkN35QgFYl0FVww> >> . But still get the following exception even though the >> class parquet.hadoop.metadata.CompressionCodecName is there in the >> parquet-hadoop.jar >> >> java.lang.ClassNotFoundException: >> parquet.hadoop.metadata.CompressionCodecName cannot be found by >> spark-sql_2.10_1.4.1.wso2v1 >> >> I kept trying with different versions, still the issue is there. >> >> On 26 August 2015 at 08:27, Nirmal Fernando <[email protected]> wrote: >> >>> Madawa, hope the issue is resolved now after the instructions given >>> offline. >>> >>> On Mon, Aug 24, 2015 at 8:34 PM, Madawa Soysa <[email protected]> >>> wrote: >>> >>>> I have tested the model.save() with a simple Java program. It works >>>> fine. >>>> >>>> I have noticed that scala-library:2.11.6 is a dependency of >>>> spark:spark-core_2.11:1.4.1 [1] >>>> <http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar> >>>> In ML scala version is explicitly specified as 2.10.4. Is there a specific >>>> reason to use scala 2.10.4? I guess this version incompatibility could be >>>> the reason for this issue. >>>> >>>> [1] - >>>> http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar >>>> >>>> On 24 August 2015 at 10:24, Nirmal Fernando <[email protected]> wrote: >>>> >>>>> Madawa, seems this is a Spark issue :-( can you try a simple Java >>>>> program and see whether model.save() works? >>>>> >>>>> On Sat, Aug 22, 2015 at 8:19 AM, Madawa Soysa <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi Nirmal, >>>>>> >>>>>> outPath is correct and the directory gets created, but the process >>>>>> becomes idle after that. Attached the only part that was written to a >>>>>> file. >>>>>> >>>>>> Also the method doesn't throw an exception as well. >>>>>> >>>>>> On 21 August 2015 at 21:31, Nirmal Fernando <[email protected]> wrote: >>>>>> >>>>>>> Hi Madawa, >>>>>>> >>>>>>> According to Spark API [1], outPath shouldn't be exist. >>>>>>> >>>>>>> [1] >>>>>>> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L200 >>>>>>> >>>>>>> On Fri, Aug 21, 2015 at 1:59 PM, Niranda Perera <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I don't think it's correct. Scala version is 2.10.4 even in the mvn >>>>>>>> repo >>>>>>>> >>>>>>>> On Fri, Aug 21, 2015, 13:46 Madawa Soysa <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Also I asked this question in StackOverflow[1] >>>>>>>>> <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> >>>>>>>>> and there they have mentioned a version incompatibility between Scala >>>>>>>>> and >>>>>>>>> Spark versions >>>>>>>>> >>>>>>>>> [1] - >>>>>>>>> http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java >>>>>>>>> >>>>>>>>> On 21 August 2015 at 13:31, Madawa Soysa <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yes path is valid, I explicitly set the path here from the >>>>>>>>>> MLModelHandler persistModel method. >>>>>>>>>> >>>>>>>>>> On 21 August 2015 at 10:26, Nirmal Fernando <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> There an issue with serializing Spark's >>>>>>>>>>>> MatrixFactorizationModel object. The object contains a huge RDD >>>>>>>>>>>> and as I >>>>>>>>>>>> have read in many blogs, this model cannot be serialized as a java >>>>>>>>>>>> object. >>>>>>>>>>>> Therefore when retrieving the model I get the same exception as >>>>>>>>>>>> above; >>>>>>>>>>>> >>>>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>>>> >>>>>>>>>>>> I have asked this question in Spark mailing lists and they >>>>>>>>>>>> recommended me to use the built in save and load functions other >>>>>>>>>>>> than using >>>>>>>>>>>> Java serializing. So I have used following method to persist the >>>>>>>>>>>> model, >>>>>>>>>>>> >>>>>>>>>>>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>>>>>>>>>>> outPath);[1] >>>>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>>>>>>>>>>> >>>>>>>>>>>> Then nothing happens when this line executes. No error is >>>>>>>>>>>> thrown as well. Any solution for this? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Can you print outPath and see whether it's a valid file path? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [1] - >>>>>>>>>>>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>>>>>>>>>>> >>>>>>>>>>>> On 16 August 2015 at 18:06, Madawa Soysa < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yes I was able to resolve the issue by removing RDD fields >>>>>>>>>>>>> from the SummaryModel object as @Mano pointed out. Still I have >>>>>>>>>>>>> the same >>>>>>>>>>>>> exception when retrieving the model. Trying to fix that issue. >>>>>>>>>>>>> >>>>>>>>>>>>> On 14 August 2015 at 10:43, Nirmal Fernando <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks Niranda, this doc is useful. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> From what I know, OneToOneDependancy come into play when >>>>>>>>>>>>>>> spark tries to create the RDD dependency tree. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Just thought of sharing that. this would be a good resource >>>>>>>>>>>>>>> [1] :-) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it >>>>>>>>>>>>>>>> something you use?* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I created a model summary in order to show the model data >>>>>>>>>>>>>>>>> in the analysis.jag page. >>>>>>>>>>>>>>>>> But when refreshing the page after building the model I >>>>>>>>>>>>>>>>> get the following error. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: >>>>>>>>>>>>>>>>> An error has occurred while extracting all the models of >>>>>>>>>>>>>>>>> analysis id: 13 >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>>>>>>>>>>> Caused by: >>>>>>>>>>>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: >>>>>>>>>>>>>>>>> An error >>>>>>>>>>>>>>>>> has occurred while extracting all the models of analysis id: >>>>>>>>>>>>>>>>> 13 >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>>>>>>>>>>> ... 52 more >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>>>>>>>>>>> ... 53 more >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I guess there is an error in persistence of the model >>>>>>>>>>>>>>>>> summary object, what should be the cause for this error? >>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>>>>>>>>>>> contains >>>>>>>>>>>>>>>>> the commit which I introduced the model summary. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1] - >>>>>>>>>>>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Undergraduate, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | >>>>>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> *Niranda Perera* >>>>>>>>>>>>>>> Software Engineer, WSO2 Inc. >>>>>>>>>>>>>>> Mobile: +94-71-554-8430 >>>>>>>>>>>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>>>>>>>>>>> https://pythagoreanscript.wordpress.com/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>> >>>>>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>>>> >>>>>>>>>>>>> Undergraduate, >>>>>>>>>>>>> >>>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>>> >>>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>>> >>>>>>>>>>>> Undergraduate, >>>>>>>>>>>> >>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>> >>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>>> [email protected] >>>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Thanks & regards, >>>>>>>>>>> Nirmal >>>>>>>>>>> >>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>> >>>>>>>>>> Undergraduate, >>>>>>>>>> >>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>> >>>>>>>>>> University of Moratuwa. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>> [email protected] >>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>> >>>>>>>>> Undergraduate, >>>>>>>>> >>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>> >>>>>>>>> University of Moratuwa. >>>>>>>>> >>>>>>>>> >>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>> [email protected] >>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks & regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Team Lead - WSO2 Machine Learner >>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>> Mobile: +94715779733 >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *_________________________________**Madawa Soysa* >>>>>> >>>>>> Undergraduate, >>>>>> >>>>>> Department of Computer Science and Engineering, >>>>>> >>>>>> University of Moratuwa. >>>>>> >>>>>> >>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>> [email protected] >>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Team Lead - WSO2 Machine Learner >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *_________________________________**Madawa Soysa* >>>> >>>> Undergraduate, >>>> >>>> Department of Computer Science and Engineering, >>>> >>>> University of Moratuwa. >>>> >>>> >>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>> [email protected] >>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> >> *_________________________________**Madawa Soysa* >> >> Undergraduate, >> >> Department of Computer Science and Engineering, >> >> University of Moratuwa. >> >> >> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >> [email protected] >> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *_________________________________**Madawa Soysa* Undergraduate, Department of Computer Science and Engineering, University of Moratuwa. Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: [email protected] LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
