>> 3. AVRO C cannot write values that are bigger than the file writer's >> specified block_size. I don't think there's enough memory to hold both the >> original structures and a gigantic block_size. > > I don't know enough about the C implementation to verify this one and > will leave it to others.
#3 is also true. AVRO-724 [1] is the relevant issue. Matt Massie's comment on that issue discusses a couple of proposed solutions. If your data structures are large arrays, then one option is to have a separate file for each array, and have the array elements correspond to records in the file. Then it's only each individual array element that needs to fit in the file's block_size.
signature.asc
Description: OpenPGP digital signature
