Thanks! I'm not familiar with logical types (sounds like this is a new, unreleased feature?) so I'll have to look into it.
- Peter On Fri, Aug 28, 2015 at 10:09 AM, Farkas, Zoltan <[email protected]> wrote: > Hi Peter, > > I have recently implemented this with the logicalType concept introduced > recently in avro. > (I have my own fork (https://github.com/zolyfarkas/avro ) that I use until I > find some time to merge Ryan's implementation, but I have other improvements > that I rely on like idl forward declarations, improved json encoding...) > > Here is how I implemented the any type: > > /** a unknown serialized java object */ > @logicalType("unknown") > record Unknown { > /** maven schema ID (optional for future extension, with different ID > types) */ > union {null, MavenSchemaId} mavenSchemaId = null; > > /** the avro serialized object */ > union {null, string, bytes} serObj; > > } > > The maven schema ID contains enough info to retrieve the schema that the > record is serialized into.(the serObj field). > > In my case I store all schemas in a maven repo, and my MavenSchemaId looks > like: > > > /** A maven artifact ID */ > record MavenArtifactId { > > /** The maven group id */ > string groupId; > > /** The maven artifactId */ > string artifactId; > > /** The schema version */ > string version; > } > > /** A maven schema ID*/ > record MavenSchemaId { > > /** The maven artifact */ > MavenArtifactId artifactId; > > /** The record name (namespace + name) */ > string recordName; > } > > But a schemaID can really be anything, (a number, a string...), as long as > you have a system/service to resolve it. You can even put the schema in the > Unknown record if that works for you... > > So every time I need a "Any"(Unknown) field I use it like: > > Import idl "common.avdl" > > record MyRecord { > ... > Unknown any; > ... > } > > The generated DTOs set and get an Object (just like unions), when you > deseralize you will get either a SpecificRecord (if you have a generated > DTO..) or a GenericRecord... > > Let me know if you have any questions... > (would be interested to know if you encounter any issues implementing this > with the official avro logical type implementation...) > > cheers > > --Z > > -----Original Message----- > From: Peter Amstutz [mailto:[email protected]] > Sent: Friday, August 28, 2015 6:26 AM > To: [email protected] > Subject: handling fields with "any" structure > > Hello everyone, > > I am using Avro to load and validate JSON documents. Mostly this works very > well and it is straightforward to express the structure of my document using > Avro schema. However, I have a few fields which can have "any" content. It > is impossible to declare all possible structures in advance, and I can't use > a union type of primitives because the fields may also contain complex types > (nested lists/maps) and Avro doesn't allow named unions. > > So far as I have been able to determine, this is impossible with standard > Avro schema, so I am curious if anyone else has dealt with this problem and > can suggest any workarounds. Currently my best (least bad) idea is to > preprocess the JSON to pull out the "any" > fields and store them on the side before handing the document to Avro for > loading. This is awkward so I would love to hear if anyone has any other > ideas. > > Thanks, > Peter > This message contains confidential information and is intended only for the > individual named. If you are not the named addressee, you should not > disseminate, distribute, alter or copy this e-mail. Please notify the sender > immediately by e-mail if you have received this e-mail by mistake and delete > this e-mail from your system. E-mail transmissions cannot be guaranteed to be > secure or without error as information could be intercepted, corrupted, lost, > destroyed, arrive late or incomplete, or contain viruses. The sender, > therefore, does not accept liability for any errors or omissions in the > contents of this message which arise during or as a result of e-mail > transmission. If verification is required, please request a hard-copy > version. This message is provided for information purposes and should not be > construed as a solicitation or offer to buy or sell any securities or related > financial instruments in any jurisdiction. Securities are offered in the > U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.
