You cannot define the same type twice within the same schema so you need to
change your "munge" step to produce the following:
{
"name": "address2",
"type": "record",
"namespace" : "some.domain",
"fields" :
[
{
"name": "street",
"type": "string"
},
{
"name": "city",
"type": "string"
},
{
"name": "position1",
"type":
{"type":"record","name":"location","namespace":"some.domain","fields":[{"name":"latitude","type":"float"},{"name":"longitude","type":"float"}]}
},
{
"name": "position2",
"type": "some.domain.location"
}
]
}
~ Nick
On May 1, 2012, at 6:55 PM, Peter Cameron wrote:
> I'm having a problem with nesting schemas. A very brief overview of why we're
> using Avro (successfully so far) is:
>
> o code generation not required
> o small binary format
> o dynamic use of schemas at runtime
>
> We're doing a flavour of RPC, and the reason we're not using Avro's IDL and
> flavour of RPC is because the endpoint is not necessarily a Java platform (C#
> and Java for our purposes), and only the Java implementation of Avro has RPC.
> Hence no Avro RPC for us.
>
> I'm aware that Avro doesn't import nested schemas out of the box. We need
> that functionality as we're exposed to schemas over which we have no control,
> and in the interests of maintainability, these schemas are nicely partitioned
> and are referenced as types from within other schemas. So, for example, a
> address schema refers to a some.domain.location object by having a field of
> type "some.domain.location". Note that our runtime has no knowledge of any
> some.domain package (e.g. address or location objects). Only the endpoints
> know about some.domain. (A layer at our endpoint runtime serialises any
> unknown i.e. non-primitive objects as bytestreams.)
>
> I implemented a schema cache which intelligently imports schemas on the fly,
> so adding the address schema to the cache, automatically adds the location
> schema that it refers to. The cache uses Avro's schema to parse an added
> schema, catches parse exceptions, looks at the exception message to see
> whether or not the error is due to a missing or undefined type, and thus goes
> off to import the needed schema. Brittle, I know, but no other way for us. We
> need this functionality, and nothing else comes close to Avro.
>
> So far so good, until today when I hit a corner case.
>
> Say I have an address object that has two fields, called position1 and
> position2. If position1 and position2 are non-primitive types, then the
> address schema doesn't parse so presumably is an invalid Avro schema. The
> error concerns redefining the location type. Here's the example:
>
> location schema
> ==============
>
> {
> "name": "location",
> "type": "record",
> "namespace" : "some.domain",
> "fields" :
> [
> {
> "name": "latitude",
> "type": "float"
> },
> {
> "name": "longitude",
> "type": "float"
> }
> ]
> }
>
> address schema
> ==============
>
> {
> "name": "address",
> "type": "record",
> "namespace" : "some.domain",
> "fields" :
> [
> {
> "name": "street",
> "type": "string"
> },
> {
> "name": "city",
> "type": "string"
> },
> {
> "name": "position1",
> "type": "some.domain.location"
> },
> {
> "name": "position2",
> "type": "some.domain.location"
> }
> ]
> }
>
>
> Now, an answer of having a list of positions as a field is not an answer for
> us, as we need to solve the general issue of a schema with more than one
> instance of the same nested type i.e. my problem is not with an address or
> location schema.
>
> The problematic schema constructed by my schema cache is:
>
> {
> "name": "address2",
> "type": "record",
> "namespace" : "some.domain",
> "fields" :
> [
> {
> "name": "street",
> "type": "string"
> },
> {
> "name": "city",
> "type": "string"
> },
> {
> "name": "position1",
> "type":
> {"type":"record","name":"location","namespace":"some.domain","fields":[{"name":"latitude","type":"float"},{"name":"longitude","type":"float"}]}
> },
> {
> "name": "position2",
> "type":
> {"type":"record","name":"location","namespace":"some.domain","fields":[{"name":"latitude","type":"float"},{"name":"longitude","type":"float"}]}
> }
> ]
> }
>
>
> Can this be done? This is potentially a blocker for us.
>
> cheers,
> Peter
>
