[
https://issues.apache.org/jira/browse/AVRO-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163006#comment-13163006
]
Doug Cutting commented on AVRO-969:
-----------------------------------
ReflectDatumWriter should be able to correctly write a superset of the types
that SpecificDatumWriter can write, so this property should not be needed.
That said, it might be good to be able to override the DatumWriter and/or
DatumReader classes used by Avro's mapred API. This might permit, e.g.,
ThriftDatumWriter to be used. So the patch that might be best is to switch to
using an "avro.input.datumReader", "avro.map_output.datumWriter", and
avro.output.datumWriter" properties that name classes whose constructor accepts
a schema parameter.
> Make possible usage of SpecificDatumWriter in avro-mapred
> ---------------------------------------------------------
>
> Key: AVRO-969
> URL: https://issues.apache.org/jira/browse/AVRO-969
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.6.1
> Reporter: Vyacheslav Zholudev
>
> I realized that ReflectDatumWriter is always used when running mapred job (in
> AvroOutputFormat.java). Sometimes it leads to bugs like in AVRO-966.
> Why not just provide a property like {{WRITER_IS_REFLECT =
> "avro.map.writer.is.reflect";}} to make a decision which DatumWriter should
> be used.
> I created a small patch to solve this:
> {code:title=avro-mapred.patch|borderStyle=solid}
> Index: lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroJob.java
> ===================================================================
> --- lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroJob.java
> (revision 1209417)
> +++ lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroJob.java
> (revision )
> @@ -53,6 +53,8 @@
> /** The configuration key for reflection-based map output representation.
> */
> public static final String MAP_OUTPUT_IS_REFLECT =
> "avro.map.output.is.reflect";
>
> + public static final String WRITER_IS_REFLECT =
> "avro.map.writer.is.reflect";
> +
> /** Configure a job's map input schema. */
> public static void setInputSchema(JobConf job, Schema s) {
> job.set(INPUT_SCHEMA, s.toString());
> Index:
> lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroOutputFormat.java
> ===================================================================
> ---
> lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroOutputFormat.java
> (revision 1209417)
> +++
> lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroOutputFormat.java
> (revision )
> @@ -23,6 +23,7 @@
> import java.util.Map;
> import java.net.URLDecoder;
>
> +import org.apache.avro.specific.SpecificDatumWriter;
> import org.apache.hadoop.io.NullWritable;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> @@ -102,8 +103,9 @@
> ? AvroJob.getMapOutputSchema(job)
> : AvroJob.getOutputSchema(job);
>
> - final DataFileWriter<T> writer =
> - new DataFileWriter<T>(new ReflectDatumWriter<T>());
> + final DataFileWriter<T> writer =
> job.getBoolean(AvroJob.WRITER_IS_REFLECT, false) ?
> + new DataFileWriter<T>(new ReflectDatumWriter<T>()) :
> + new DataFileWriter<T>(new SpecificDatumWriter<T>());
>
> configureDataFileWriter(writer, job);
> {code}
> Does it make sense?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
