[jira] [Commented] (AVRO-1562) Add support for types extending Maps/Collections
[
https://issues.apache.org/jira/browse/AVRO-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133233#comment-15133233
]
ASF GitHub Bot commented on AVRO-1562:
--
Github user asfgit closed the pull request at:
https://github.com/apache/avro/pull/25
> Add support for types extending Maps/Collections
>
>
> Key: AVRO-1562
> URL: https://issues.apache.org/jira/browse/AVRO-1562
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.7.6
>Reporter: Sachin Goyal
> Attachments: custom_map_and_collections1.patch
>
>
> Consider the following code:
> {code}
> import java.io.ByteArrayOutputStream;
> import java.util.*;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileWriter;
> import org.apache.avro.reflect.ReflectData;
> import org.apache.avro.reflect.ReflectDatumWriter;
> public class AvroDerivingMaps
> {
> public static void main (String [] args) throws Exception
> {
> MapDerivedContainer orig = new MapDerivedContainer();
> ReflectData rdata = ReflectData.AllowNull.get();
> Schema schema = rdata.getSchema(MapDerivedContainer.class);
> System.out.println(schema);
>
> ReflectDatumWriter datumWriter = new
> ReflectDatumWriter (MapDerivedContainer.class, rdata);
> DataFileWriter fileWriter = new
> DataFileWriter (datumWriter);
> ByteArrayOutputStream baos = new ByteArrayOutputStream();
> fileWriter.create(schema, baos);
> fileWriter.append(orig);
> fileWriter.close();
> }
> }
> class MapDerived extends HashMap
> {
> Integer a = 1;
> String b = "b";
> }
> class MapDerivedContainer
> {
> MapDerived2 map = new MapDerived2();
> }
> class MapDerived2 extends MapDerived
> {
> String c = "c";
> }
> {code}
> \\
> \\
> It throws the following exception:
> {code:javascript}
> {"type":"record","name":"MapDerivedContainer","namespace":"avro","fields":[{"name":"map","type":["null",{"type":"record","name":"MapDerived2","fields":[{"name":"c","type":["null","string"],"default":null},{"name":"a","type":["null","int"],"default":null},{"name":"b","type":["null","string"],"default":null}]}],"default":null}]}
> {code}
> {color:brown}
> Exception in thread "main"
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException:
> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"record","name":"MapDerived2","namespace":"avro","fields":[{"name":"c","type":["null","string"],"default":null},{"name":"a","type":["null","int"],"default":null},{"name":"b","type":["null","string"],"default":null}]}]:
> {}
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:203)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
> ... 1 more
> {color}
> \\
> \\
> It appears that ReflectData#createSchema() checks for "type instanceof
> ParameterizedType" and because of this, it skips handling of the map.
> The same is not true of GenericData#isMap() and GenericData#resolveUnion()
> fails because of this.
> The same may be true for classes extending ArrayList, Collection, Set etc.
> Also, note the schema for the class extending Map:
> {code:javascript}
> {
>"type":"record",
>"name":"MapDerived2",
>"fields":[
> {
> "name":"c",
> "type":[
> "null",
> "string"
> ],
> "default":null
> },
> {
> "name":"a",
> "type":[
> "null",
> "int"
> ],
> "default":null
> },
> {
> "name":"b",
> "type":[
> "null",
> "string"
> ],
> "default":null
> }
>]
> }
> {code}
> This schema ignores the Map completely.
> Probably, for such a class, the schema should look like:
> {code:javascript}
> {
>"type":"record",
>
[jira] [Commented] (AVRO-1562) Add support for types extending Maps/Collections
[
https://issues.apache.org/jira/browse/AVRO-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357628#comment-14357628
]
ASF GitHub Bot commented on AVRO-1562:
--
GitHub user sachingsachin opened a pull request:
https://github.com/apache/avro/pull/25
AVRO-1562: Support classes extending Maps/Collections in Avro
For classes extending collections and maps, _avro_implicit_collection_
and _avro_implicit_map_ fields are added in the Avro representation while the
class itself continues to hold other fields normally like a record.
The implicit field-names are currently hardcoded as above.
We will need to have an API in ReflectData for making this configurable.
We may also need some name-mangling to have different names for classes
extending different parameterized collections/maps.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sachingsachin/avro AVRO-1562
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/avro/pull/25.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #25
commit f13568f3064f56064edddeb2d122e11888e23831
Author: Sachin Goyal sgo...@walmart.com
Date: 2015-03-11T21:12:18Z
AVRO-1562: Support classes extending Maps/Collections in Avro
For classes extending collections and maps, _avro_implicit_collection_
and _avro_implicit_map_ fields are added in the Avro representation while the
class itself continues to hold other fields normally like a record.
The implicit field-names are currently hardcoded as above.
We will need to have an API in ReflectData for making this configurable.
We may also need some name-mangling to have different names for classes
extending different parameterized collections/maps.
Add support for types extending Maps/Collections
Key: AVRO-1562
URL: https://issues.apache.org/jira/browse/AVRO-1562
Project: Avro
Issue Type: Bug
Affects Versions: 1.7.6
Reporter: Sachin Goyal
Attachments: custom_map_and_collections1.patch
Consider the following code:
{code}
import java.io.ByteArrayOutputStream;
import java.util.*;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.reflect.ReflectDatumWriter;
public class AvroDerivingMaps
{
public static void main (String [] args) throws Exception
{
MapDerivedContainer orig = new MapDerivedContainer();
ReflectData rdata = ReflectData.AllowNull.get();
Schema schema = rdata.getSchema(MapDerivedContainer.class);
System.out.println(schema);
ReflectDatumWriterMapDerivedContainer datumWriter = new
ReflectDatumWriter (MapDerivedContainer.class, rdata);
DataFileWriterMapDerivedContainer fileWriter = new
DataFileWriterMapDerivedContainer (datumWriter);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
fileWriter.create(schema, baos);
fileWriter.append(orig);
fileWriter.close();
}
}
class MapDerived extends HashMapString, Integer
{
Integer a = 1;
String b = b;
}
class MapDerivedContainer
{
MapDerived2 map = new MapDerived2();
}
class MapDerived2 extends MapDerived
{
String c = c;
}
{code}
\\
\\
It throws the following exception:
{code:javascript}
{type:record,name:MapDerivedContainer,namespace:avro,fields:[{name:map,type:[null,{type:record,name:MapDerived2,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}],default:null}]}
{code}
{color:brown}
Exception in thread main
org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.UnresolvedUnionException:
Caused by: org.apache.avro.UnresolvedUnionException: Not in union
[null,{type:record,name:MapDerived2,namespace:avro,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}]:
{}
at
org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
at
org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
at
org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:203)
at
[jira] [Commented] (AVRO-1562) Add support for types extending Maps/Collections
[
https://issues.apache.org/jira/browse/AVRO-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357633#comment-14357633
]
Sachin Goyal commented on AVRO-1562:
I have submitted a PR for this, just in case this is considered a desirable
feature in the future.
Add support for types extending Maps/Collections
Key: AVRO-1562
URL: https://issues.apache.org/jira/browse/AVRO-1562
Project: Avro
Issue Type: Bug
Affects Versions: 1.7.6
Reporter: Sachin Goyal
Attachments: custom_map_and_collections1.patch
Consider the following code:
{code}
import java.io.ByteArrayOutputStream;
import java.util.*;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.reflect.ReflectDatumWriter;
public class AvroDerivingMaps
{
public static void main (String [] args) throws Exception
{
MapDerivedContainer orig = new MapDerivedContainer();
ReflectData rdata = ReflectData.AllowNull.get();
Schema schema = rdata.getSchema(MapDerivedContainer.class);
System.out.println(schema);
ReflectDatumWriterMapDerivedContainer datumWriter = new
ReflectDatumWriter (MapDerivedContainer.class, rdata);
DataFileWriterMapDerivedContainer fileWriter = new
DataFileWriterMapDerivedContainer (datumWriter);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
fileWriter.create(schema, baos);
fileWriter.append(orig);
fileWriter.close();
}
}
class MapDerived extends HashMapString, Integer
{
Integer a = 1;
String b = b;
}
class MapDerivedContainer
{
MapDerived2 map = new MapDerived2();
}
class MapDerived2 extends MapDerived
{
String c = c;
}
{code}
\\
\\
It throws the following exception:
{code:javascript}
{type:record,name:MapDerivedContainer,namespace:avro,fields:[{name:map,type:[null,{type:record,name:MapDerived2,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}],default:null}]}
{code}
{color:brown}
Exception in thread main
org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.UnresolvedUnionException:
Caused by: org.apache.avro.UnresolvedUnionException: Not in union
[null,{type:record,name:MapDerived2,namespace:avro,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}]:
{}
at
org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
at
org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
at
org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:203)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
... 1 more
{color}
\\
\\
It appears that ReflectData#createSchema() checks for type instanceof
ParameterizedType and because of this, it skips handling of the map.
The same is not true of GenericData#isMap() and GenericData#resolveUnion()
fails because of this.
The same may be true for classes extending ArrayList, Collection, Set etc.
Also, note the schema for the class extending Map:
{code:javascript}
{
type:record,
name:MapDerived2,
fields:[
{
name:c,
type:[
null,
string
],
default:null
},
{
name:a,
type:[
null,
int
],
default:null
},
{
name:b,
type:[
null,
string
],
default:null
}
]
}
{code}
This schema ignores the Map completely.
Probably, for such a class, the schema should look like:
{code:javascript}
{
type:record,
name:MapDerived2,
fields:[
{
name:c,
type:[
null,
string
],
default:null
},
// Other fields in the class extending the Map
{
[jira] [Commented] (AVRO-1562) Add support for types extending Maps/Collections
[
https://issues.apache.org/jira/browse/AVRO-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125830#comment-14125830
]
Sachin Goyal commented on AVRO-1562:
Thank you [~martinkl] for weighing in on this.
Please help me understand if you had something else in mind about
multi-language compatibility:
{quote}
It needs to be implemented in every supported language.
{quote}
The output of the proposed solution is readable in any language.
For classes extending maps/collections, an implicit map/collection type field
is added to the schema.
Same is used when writing such fields.
This makes it readable with any language.
Add support for types extending Maps/Collections
Key: AVRO-1562
URL: https://issues.apache.org/jira/browse/AVRO-1562
Project: Avro
Issue Type: Bug
Affects Versions: 1.7.6
Reporter: Sachin Goyal
Attachments: custom_map_and_collections1.patch
Consider the following code:
{code}
import java.io.ByteArrayOutputStream;
import java.util.*;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.reflect.ReflectDatumWriter;
public class AvroDerivingMaps
{
public static void main (String [] args) throws Exception
{
MapDerivedContainer orig = new MapDerivedContainer();
ReflectData rdata = ReflectData.AllowNull.get();
Schema schema = rdata.getSchema(MapDerivedContainer.class);
System.out.println(schema);
ReflectDatumWriterMapDerivedContainer datumWriter = new
ReflectDatumWriter (MapDerivedContainer.class, rdata);
DataFileWriterMapDerivedContainer fileWriter = new
DataFileWriterMapDerivedContainer (datumWriter);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
fileWriter.create(schema, baos);
fileWriter.append(orig);
fileWriter.close();
}
}
class MapDerived extends HashMapString, Integer
{
Integer a = 1;
String b = b;
}
class MapDerivedContainer
{
MapDerived2 map = new MapDerived2();
}
class MapDerived2 extends MapDerived
{
String c = c;
}
{code}
\\
\\
It throws the following exception:
{code:javascript}
{type:record,name:MapDerivedContainer,namespace:avro,fields:[{name:map,type:[null,{type:record,name:MapDerived2,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}],default:null}]}
{code}
{color:brown}
Exception in thread main
org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.UnresolvedUnionException:
Caused by: org.apache.avro.UnresolvedUnionException: Not in union
[null,{type:record,name:MapDerived2,namespace:avro,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}]:
{}
at
org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
at
org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
at
org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:203)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
... 1 more
{color}
\\
\\
It appears that ReflectData#createSchema() checks for type instanceof
ParameterizedType and because of this, it skips handling of the map.
The same is not true of GenericData#isMap() and GenericData#resolveUnion()
fails because of this.
The same may be true for classes extending ArrayList, Collection, Set etc.
Also, note the schema for the class extending Map:
{code:javascript}
{
type:record,
name:MapDerived2,
fields:[
{
name:c,
type:[
null,
string
],
default:null
},
{
name:a,
type:[
null,
int
],
default:null
},
{
name:b,
type:[
null,
string
],
default:null
}
]
}
{code}
This schema
[jira] [Commented] (AVRO-1562) Add support for types extending Maps/Collections
[
https://issues.apache.org/jira/browse/AVRO-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115825#comment-14115825
]
Doug Cutting commented on AVRO-1562:
I am not convinced it's worthwhile to get Avro reflect to correctly model every
Java datastructure. Avro schemas do not support inheritance. This permits
simple interoperability with programming languages without inheritance, like C.
And languages that do support inheritance do so in different ways such that it
would be difficult to map a single inheritance-based schema language to every
feature of all of them. Avro data structures are meant to be a common subset,
not the superset of all features.
Continued efforts to force inheritance into Avro's existing schema language
results in odder and odder schemas, schemas that only make sense to Avro
reflect and are not a good basis for data interoperability.
What do others think?
Add support for types extending Maps/Collections
Key: AVRO-1562
URL: https://issues.apache.org/jira/browse/AVRO-1562
Project: Avro
Issue Type: Bug
Affects Versions: 1.7.6
Reporter: Sachin Goyal
Attachments: custom_map_and_collections1.patch
Consider the following code:
{code}
import java.io.ByteArrayOutputStream;
import java.util.*;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.reflect.ReflectDatumWriter;
public class AvroDerivingMaps
{
public static void main (String [] args) throws Exception
{
MapDerivedContainer orig = new MapDerivedContainer();
ReflectData rdata = ReflectData.AllowNull.get();
Schema schema = rdata.getSchema(MapDerivedContainer.class);
System.out.println(schema);
ReflectDatumWriterMapDerivedContainer datumWriter = new
ReflectDatumWriter (MapDerivedContainer.class, rdata);
DataFileWriterMapDerivedContainer fileWriter = new
DataFileWriterMapDerivedContainer (datumWriter);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
fileWriter.create(schema, baos);
fileWriter.append(orig);
fileWriter.close();
}
}
class MapDerived extends HashMapString, Integer
{
Integer a = 1;
String b = b;
}
class MapDerivedContainer
{
MapDerived2 map = new MapDerived2();
}
class MapDerived2 extends MapDerived
{
String c = c;
}
{code}
\\
\\
It throws the following exception:
{code:javascript}
{type:record,name:MapDerivedContainer,namespace:avro,fields:[{name:map,type:[null,{type:record,name:MapDerived2,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}],default:null}]}
{code}
{color:brown}
Exception in thread main
org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.UnresolvedUnionException:
Caused by: org.apache.avro.UnresolvedUnionException: Not in union
[null,{type:record,name:MapDerived2,namespace:avro,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}]:
{}
at
org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
at
org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
at
org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:203)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
... 1 more
{color}
\\
\\
It appears that ReflectData#createSchema() checks for type instanceof
ParameterizedType and because of this, it skips handling of the map.
The same is not true of GenericData#isMap() and GenericData#resolveUnion()
fails because of this.
The same may be true for classes extending ArrayList, Collection, Set etc.
Also, note the schema for the class extending Map:
{code:javascript}
{
type:record,
name:MapDerived2,
fields:[
{
name:c,
type:[
null,
string
],
default:null
},
{
[jira] [Commented] (AVRO-1562) Add support for types extending Maps/Collections
[
https://issues.apache.org/jira/browse/AVRO-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098980#comment-14098980
]
Sachin Goyal commented on AVRO-1562:
I have a patch ready for this.
Will be submitting that shortly.
Add support for types extending Maps/Collections
Key: AVRO-1562
URL: https://issues.apache.org/jira/browse/AVRO-1562
Project: Avro
Issue Type: Bug
Affects Versions: 1.7.6
Reporter: Sachin Goyal
Consider the following code:
{code}
import java.io.ByteArrayOutputStream;
import java.util.*;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.reflect.ReflectDatumWriter;
public class AvroDerivingMaps
{
public static void main (String [] args) throws Exception
{
MapDerivedContainer orig = new MapDerivedContainer();
ReflectData rdata = ReflectData.AllowNull.get();
Schema schema = rdata.getSchema(MapDerivedContainer.class);
System.out.println(schema);
ReflectDatumWriterMapDerivedContainer datumWriter = new
ReflectDatumWriter (MapDerivedContainer.class, rdata);
DataFileWriterMapDerivedContainer fileWriter = new
DataFileWriterMapDerivedContainer (datumWriter);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
fileWriter.create(schema, baos);
fileWriter.append(orig);
fileWriter.close();
}
}
class MapDerived extends HashMapString, Integer
{
Integer a = 1;
String b = b;
}
class MapDerivedContainer
{
MapDerived2 map = new MapDerived2();
}
class MapDerived2 extends MapDerived
{
String c = c;
}
{code}
\\
\\
It throws the following exception:
{code:javascript}
{type:record,name:MapDerivedContainer,namespace:avro,fields:[{name:map,type:[null,{type:record,name:MapDerived2,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}],default:null}]}
{code}
{color:brown}
Exception in thread main
org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.UnresolvedUnionException:
Caused by: org.apache.avro.UnresolvedUnionException: Not in union
[null,{type:record,name:MapDerived2,namespace:avro,fields:[{name:c,type:[null,string],default:null},{name:a,type:[null,int],default:null},{name:b,type:[null,string],default:null}]}]:
{}
at
org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
at
org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
at
org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:203)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
... 1 more
{color}
\\
\\
It appears that ReflectData#createSchema() checks for type instanceof
ParameterizedType and because of this, it skips handling of the map.
The same is not true of GenericData#isMap() and GenericData#resolveUnion()
fails because of this.
The same may be true for classes extending ArrayList, Collection, Set etc.
Also, note the schema for the class extending Map:
{code:javascript}
{
type:record,
name:MapDerived2,
fields:[
{
name:c,
type:[
null,
string
],
default:null
},
{
name:a,
type:[
null,
int
],
default:null
},
{
name:b,
type:[
null,
string
],
default:null
}
]
}
{code}
This schema ignores the Map completely.
Probably, for such a class, the schema should look like:
{code:javascript}
{
type:record,
name:MapDerived2,
fields:[
{
name:c,
type:[
null,
string
],
default:null
},
// Other fields in the class extending the Map
{
name:BASE_MAP,
type:[
null,
map ... // Normal map which
