Re: Unnecessary Variable

2015-09-02 Thread Renato Marroquín Mogrovejo
Hi Furkan, Thanks for taking the time and looking into this. I agree with you, there is something fuzzy going on there. So our persistent classes extent org.apache.avro.specific.SpecificRecord and use their specific DatumReader/Writer which in turn use the the methods that you are pointing out. An

Re: Unnecessary Variable

2015-09-02 Thread Furkan KAMACI
Also, here is the issue for general serialization mechanism for Map Reduce: https://issues.apache.org/jira/browse/HADOOP-1986 Here is the main usage of that parameter: https://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/generic/GenericDatumReader.html#read(D, org.apache.avro.io.Decoder) I

Re: Unnecessary Variable

2015-08-26 Thread Renato Marroquín Mogrovejo
Yeah in the GoraMapReduceUtils it doesn't get used, I vaguely remember that I used it in some code a while ago, but I tried looking out for it, but I didn't find anything. The other thing I found was the PersistentDeserializer that also uses it [1] and that gets created in the GoraMapReduceUtils, b

Re: Unnecessary Variable

2015-08-17 Thread Furkan KAMACI
They are all passed to GoraMapReduceUtils.setIOSerializations(conf, reuseObjects); at that two examples, am I right? However, it is not used at there. 18 Ağu 2015 00:47 tarihinde "Renato Marroquín Mogrovejo" < renatoj.marroq...@gmail.com> yazdı: > Sorry, I pressed enter too fast there. > As poin

Re: Unnecessary Variable

2015-08-17 Thread Renato Marroquín Mogrovejo
Sorry, I pressed enter too fast there. As pointed out in the JIRA issue, it is for not creating too many objects while mapping or reducing. You can also find it in the GoraMapper. It is also used in here: https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/

Re: Unnecessary Variable

2015-08-17 Thread Renato Marroquín Mogrovejo
What about ... https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/GoraReducer.java#L47 2015-08-17 14:38 GMT-07:00 Furkan KAMACI : > Whether you set *reuseObjects *to true or false, it is not considered at > source code.. > > On Tue, Aug 18, 2015 at 12:35

Re: Unnecessary Variable

2015-08-17 Thread Furkan KAMACI
Whether you set *reuseObjects *to true or false, it is not considered at source code.. On Tue, Aug 18, 2015 at 12:35 AM, Furkan KAMACI wrote: > A job conf is passed to Spark to store RDD results. I needed a similar > piece of code at my implementation (as like storing sth at data store via > Apa

Re: Unnecessary Variable

2015-08-17 Thread Furkan KAMACI
A job conf is passed to Spark to store RDD results. I needed a similar piece of code at my implementation (as like storing sth at data store via Apache Gora). When I check the code, I thought that *reuseObjects *variable is not necessary for setIOSerializations method at Apache Gora and wanted to b

Re: Unnecessary Variable

2015-08-17 Thread Renato Marroquín Mogrovejo
I think it was this [1], but this was specific to hadoop I think. I think you could safely pass there, so we can keep it to stay compatible with Hadoop API. Just a quick question, how does the HadoopRDD deals with it? does it expose this parameter? Maybe it's worth checking it out as well. Renat

Re: Unnecessary Variable

2015-08-17 Thread Henry Saputra
I think it was from way back, didn't remember why. Why do you need to remove it to support Spark? - Henry On Mon, Aug 17, 2015 at 1:37 PM, Furkan KAMACI wrote: > Hi All, > > There is a method as: > > public static void setIOSerializations(Configuration conf, boolean > reuseObjects) > > at G

Unnecessary Variable

2015-08-17 Thread Furkan KAMACI
Hi All, There is a method as: public static void setIOSerializations(Configuration conf, boolean reuseObjects) at GoraMapReduceUtils.java However, reuseObjects is never used at that method. I've removed it at my Spark implementation. Is it necessary for future use? Kind Regards, Furkan KAM