Hi Furkan,
Thanks for taking the time and looking into this.
I agree with you, there is something fuzzy going on there. So our
persistent classes extent org.apache.avro.specific.SpecificRecord and use
their specific DatumReader/Writer which in turn use the the methods that
you are pointing out.
An
Also, here is the issue for general serialization mechanism for Map Reduce:
https://issues.apache.org/jira/browse/HADOOP-1986
Here is the main usage of that parameter:
https://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/generic/GenericDatumReader.html#read(D,
org.apache.avro.io.Decoder)
I
Yeah in the GoraMapReduceUtils it doesn't get used, I vaguely remember that
I used it in some code a while ago, but I tried looking out for it, but I
didn't find anything. The other thing I found was the
PersistentDeserializer that also uses it [1] and that gets created in the
GoraMapReduceUtils, b
They are all passed to
GoraMapReduceUtils.setIOSerializations(conf, reuseObjects);
at that two examples, am I right? However, it is not used at there.
18 Ağu 2015 00:47 tarihinde "Renato Marroquín Mogrovejo" <
renatoj.marroq...@gmail.com> yazdı:
> Sorry, I pressed enter too fast there.
> As poin
Sorry, I pressed enter too fast there.
As pointed out in the JIRA issue, it is for not creating too many objects
while mapping or reducing. You can also find it in the GoraMapper.
It is also used in here:
https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/
What about ...
https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/GoraReducer.java#L47
2015-08-17 14:38 GMT-07:00 Furkan KAMACI :
> Whether you set *reuseObjects *to true or false, it is not considered at
> source code..
>
> On Tue, Aug 18, 2015 at 12:35
Whether you set *reuseObjects *to true or false, it is not considered at
source code..
On Tue, Aug 18, 2015 at 12:35 AM, Furkan KAMACI
wrote:
> A job conf is passed to Spark to store RDD results. I needed a similar
> piece of code at my implementation (as like storing sth at data store via
> Apa
A job conf is passed to Spark to store RDD results. I needed a similar
piece of code at my implementation (as like storing sth at data store via
Apache Gora). When I check the code, I thought that *reuseObjects *variable
is not necessary for setIOSerializations method at Apache Gora and wanted
to b
I think it was this [1], but this was specific to hadoop I think. I think
you could safely pass there, so we can keep it to stay compatible
with Hadoop API.
Just a quick question, how does the HadoopRDD deals with it? does it expose
this parameter? Maybe it's worth checking it out as well.
Renat
I think it was from way back, didn't remember why.
Why do you need to remove it to support Spark?
- Henry
On Mon, Aug 17, 2015 at 1:37 PM, Furkan KAMACI wrote:
> Hi All,
>
> There is a method as:
>
> public static void setIOSerializations(Configuration conf, boolean
> reuseObjects)
>
> at G
Hi All,
There is a method as:
public static void setIOSerializations(Configuration conf, boolean
reuseObjects)
at GoraMapReduceUtils.java
However, reuseObjects is never used at that method. I've removed it at my
Spark implementation. Is it necessary for future use?
Kind Regards,
Furkan KAM
11 matches
Mail list logo
