[
https://issues.apache.org/jira/browse/FLINK-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726553#comment-15726553
]
ASF GitHub Bot commented on FLINK-5039:
---------------------------------------
GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/2953
[FLINK-5039] Bump Avro version
This is a critical issue for some of our users.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink flink5039
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2953.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2953
----
commit b3da788361b5ffe72edb033d663a58433a695c73
Author: Robert Metzger <[email protected]>
Date: 2016-12-06T20:03:10Z
[FLINK-5039] Bump Avro version
----
> Avro GenericRecord support is broken
> ------------------------------------
>
> Key: FLINK-5039
> URL: https://issues.apache.org/jira/browse/FLINK-5039
> Project: Flink
> Issue Type: Bug
> Components: Batch Connectors and Input/Output Formats
> Affects Versions: 1.1.3
> Reporter: Bruno Dumon
> Priority: Blocker
> Fix For: 1.2.0, 1.1.4
>
>
> Avro GenericRecord support was introduced in FLINK-3691, but it seems like
> the GenericRecords are not properly (de)serialized.
> This can be easily seen with a program like this:
> {noformat}
> env.createInput(new AvroInputFormat<>(new Path("somefile.avro"),
> GenericRecord.class))
> .first(10)
> .print();
> {noformat}
> which will print records in which all fields have the same value:
> {noformat}
> {"foo": 1478628723066, "bar": 1478628723066, "baz": 1478628723066, ...}
> {"foo": 1478628723179, "bar": 1478628723179, "baz": 1478628723179, ...}
> {noformat}
> If I'm not mistaken, the AvroInputFormat does essentially
> TypeExtractor.getForClass(GenericRecord.class), but GenericRecords are not
> POJOs.
> Furthermore, each GenericRecord contains a pointer to the record schema. I
> guess the current naive approach will serialize this schema with each record,
> which is quite inefficient (the schema is typically more complex and much
> larger than the data). We probably need a TypeInformation and TypeSerializer
> specific to Avro GenericRecords, which could just use avro serialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
