Hey Steinar, When reading data from an Avro schema, you can specify a subset of the schema as the "expected" schema and only those fields will be unpacked. I don't know the exact details of how this works, but I have requested documentation: https://issues.apache.org/jira/browse/AVRO-235. Perhaps you could vote up that issue?
Thanks, Jeff On Wed, Nov 25, 2009 at 2:19 AM, Steinar Knutsen <[email protected]>wrote: > Hi, > > I work in vertical search in Yahoo!, we would like to use for an > application > where it would be extremely beneficial to be able to extract single fields > from binary Avro data without unpacking the entire datagram. This is hard > to > do efficiently, as far as I can tell. Avro supports skip information, but > it > is somewhat inefficient to skip across a block of an array, a record or a > map, if any of these contain a variable length object. The headers only > contain the number of objects contained, not the length in bytes. > > Is the correct way to solve this kind of scenario to create a schema with > extra meta data (which feels like kind of a kludge), or is there some other > way? Hopefully, I have overlooked something. > > Regards, > -- > Steinar Knutsen <[email protected]> > Yahoo! Technologies Norway AS >
