On Apr 3, 2009, at 1:02 PM, Doug Cutting wrote:
George Porter wrote:
While this representation would certainly be as compact as
possible, wouldn't it prevent evolving the data structure over
time? One of the nice features of Google Protocol Buffers and
Thrift is that you can evolve the set of fields over time, and
older/newer clients can talk to older/newer services. If the
proposed Avro is evolvable, then perhaps I'm misunderstanding your
statement about the lack of IDs in the serialized data.
Avro supports schema evolution. In Avro, the schema used to write
the data must be available when the data is read. (In files, it is
typically stored in the file metadata.)
If you have the schema that was used to write the data, and you're
expecting a slightly different schema, then you simply keep those
fields that are in both schemas and skip those not. This is
equivalent to Thrift and Protocol Buffer's support for schema
evolution, but does not require manually assigning numeric field ids.
This feature can also be used to support projection. If you have
records with many large fields, but only need a single field in a
particular computation, then you can specify an expected schema with
only that field, and the runtime will efficiently skip all of the
other fields, returning a record with just the single, expected field.
Thanks for the clarification--I better understand the schema
relationship. The projection feature is a nice feature, especially
since it seems like it would be able to support "sparse files" where
you want to just peek at large structs without invoking a lot of disk-
io (for data serialized on-disk).
I also agree with Bryan, in that it would be unfortunate to have
two different Apache projects with overlapping goals.
We already have both Thrift and Etch in the incubator, which have
similar goals. Apache does not attempt to mandate that projects
have disjoint goals. There are many ways to slice things, and
Apache prefers to rely on survival of the fittest rather than
forcing things together.
Regardless of features, both protocol buffers and thrift have the
advantage of being debugged in mission-critical production
environments.
Yes, but, as I've argued in other messages in this thread, they do
not support the dynamic features we need. Adding those features
would add new code that would share little with existing code in
those projects. So, while the projects are conceptually similar, the
implementations are necessarily different, and, without significant
code overlap, separate projects seem more natural.
Doug
Makes sense. Thanks,
George
