Hi,
it's a blocker in my project as well and as noone was working on it afaik
at this point I started doing it. I am half way through it (turning the
default values from the schema json into avro values and adding them to the
record field schema struct, the other half is filling the gaps already
marked in the code with TODOs using this piece). Currently I am on
vacation, but I plan to finish this in August.
Regards,
Jeno
2014.07.30. 6:38 ezt írta ("Steve Roehrs" <[email protected]>):
> Hi Chris
>
> The C API doesn't support default values, or schema evolution. It does
> support schema projection, where the reader schema has less fields than
> the writer schema.
>
> For this reason we switched to using the C++ API for our particular
> project. C++ Schema evolution has only recently been baselined in
> version 1.7.7 of Avro.
>
> I did some work on trying to implement it in the C version, but gave up,
> as I found the C code quite difficult to work with.
>
> I agree the documentation should be clarified. Schema projection and
> type promotion are a subset of schema resolution - but schema evolution
> is definitely missing!
>
>
> Steve Roehrs
>
> Senior Software Engineer | Lockheed Martin
>
>
>
> | p: +61 8 7389 4525 | m: +61 4 3891 5622 | f: +61 8 7389 4551
>
> | w: www.rlmgroup.com.au | e: [email protected]
>
> | Company address: 82-86 Woomera Ave, Edinburgh, SA 5111
>
> This email and any attachment to it remains the property of Lockheed
> Martin and is intended only to be read or used by the named addressee.
> It may contain information that is confidential, commercially valuable
> or subject to legal privilege. If you receive this email in error,
> please immediately delete it and notify the sender. Opinions,
> conclusions and other information in this message that do not relate to
> the official business of Lockheed Martin or any companies within
> Lockheed Martin shall be understood as neither given nor endorsed by
> them.
>
> -----Original Message-----
> From: capo hatsoft [mailto:[email protected]]
> Sent: Wednesday, July 30, 2014 12:05 PM
> To: [email protected]
> Subject: Schema default values in C implimentation
>
> I recently extended a tool used at my work by adding an Avro output
> module.
> The module works fine except that it appears to ignore default values in
> the schema.
>
> My program does something like this:
>
> presuming schemaBuffer contains
>
> {"type":"record",
> "name":"test",
> "fields":[
> {"name":"foo", "type":"int"},
> {"name":"hat", "type":"int", "default":12},
> {"name":"bar", "type":"string"}
> ]
> }
>
> avro_schema_t schema;
> avro_schema_from_json_length(schemaBuffer, schemaLen, &schema);
>
> avro_value_iface_t * iface;
> avro_value_t writer_value, field;
> avro_file_writer_t avro_writer;
>
> iface = avro_generic_class_from_schema(schema);
> avro_generic_value_new(iface, &writer_value);
> avro_file_writer_create_with_codec(outFilePath, schema, &avro_writer,
> "defalte", blockSize);
>
> //iterate over some data structure containing src data {
> //for int values {
> avro_value_set_int(&field, someIntValue);
> }
> //similar code for other types...
> }
> avro_file_writer_append_value(avro_writer, &writer_value)
>
>
> //clean up
> //flush etc on program exit
>
> The result is that the program correctly creates an avro encoded file
> with
> one record for each of my input records with all the correct values
> etc....
>
> Except! The schema at the top of the output file created is different to
> the input schema. It now looks like:
>
> {"type":"record",
> "name":"test",
> "fields":[
> {"name":"foo", "type":"int"},
> {"name":"hat", "type":"int"},
> {"name":"bar", "type":"string"}
> ]
> }
>
> The default property just seems to be completely ignored by the schema
> parser or otherwise not reproduced by the schema writer.
>
> Having a look at the source code I came across this concerning struct in
> schema.h:
>
> struct avro_record_field_t {
> int index;
> char *name;
> avro_schema_t type;
> /*
> * TODO: default values
> */
> };
>
> So it appears that default values are not supported by Avro C?
>
> I'm pretty confused however as the documentation at
> http://avro.apache.org/docs/1.7.7/api/c/index.html states:
>
> The C implementation supports:
>
> -
>
> binary encoding/decoding of all primitive and complex data types
> -
>
> storage to an Avro Object Container File
> -
>
> schema resolution, promotion and projection
> -
>
> validating and non-validating mode for writing Avro data
>
> The C implementation is lacking:
>
> -
>
> RPC
>
> Is the documentation wrong or am I just missing something?
>
> I couldn't find any evidence that default values are supported after
> reading over the source. If this feature still planned to be
> implemented?
> Should the documentation be updated to reflect that the C implementation
> does not support default values?
>
> This is a blocker for me so I was considering extending Avro C to
> support
> default values myself but I thought I should check with the mailing list
> first.
>
> Thanks in advance,
> Chris.
>
