RE: Serialization with optional fields using C++ library

[Anton]

John,

 

There was “default” originally. I changed it to “defaults” after I’ve met such 
spelling in some stackoverflow answer. None of variants worked for nonempty 
data-field.

 

From: John McClean [mailto:jmccl...@gmail.com] 
Sent: Wednesday, December 22, 2021 8:11 PM
To: user@avro.apache.org
Subject: Re: Serialization with optional fields using C++ library

 

I only skimmed this, but the schema should read "default", not "defaults". I've 
no idea if that's the only issue.

 

J

 

On Wed, Dec 22, 2021 at 8:40 AM Anton <anton...@yandex.ru 
<mailto:anton...@yandex.ru> > wrote:

Hi Martin,

 

SchemaTests.cc checks only test compilation and my schema compiles fine 
according to my code.

Is there any test that checks full cycle of serialization/deserialization 
process with json/schema provided?

 

 

From: Martin Grigorov [mailto:mgrigo...@apache.org 
<mailto:mgrigo...@apache.org> ] 
Sent: Wednesday, December 22, 2021 3:24 PM
To: user@avro.apache.org <mailto:user@avro.apache.org> 
Subject: Re: Serialization with optional fields using C++ library

 

Hi Anton,

 

I don't see any unit tests / examples for this at 
https://github.com/apache/avro/tree/master/lang/c%2B%2B/ test|examples, so I 
guess it is not implemented yet.

You could add an entry for basicSchemas 
(https://github.com/apache/avro/blob/1aa963c44d1b9da3dfcf74acb3eeed56439332a0/lang/c%2B%2B/test/SchemaTests.cc#L30)
 and if it fails then create an issue / PR.  

 

On Wed, Dec 22, 2021 at 10:13 AM Anton <anton...@yandex.ru 
<mailto:anton...@yandex.ru> > wrote:

Hello, I have problem with serialization of data having optional fields. When I 
pass null in corresponding field it works but when it is non-null then 
serialization fails.

Schema:

{

                "type": "record",

                "name": "schema",

                "namespace": "space",

                "fields": [

                               {

                                               "name": "username",

                                               "type": "string"

                               },

                               {

                                               "name": "data",

                                               "type": [

                                                               "null",

                                                               "string"

                                               ],

                                               "defaults": null

                               },

                               {

                                               "name": "timestamp",

                                               "type": "long"

                               }

                ]

}

 

Data that works:

{"username":"miguno","data":null,"timestamp": 1366150681 }

 

Data that fails:

{"username":"miguno","data":"test","timestamp": 1366150681 }

 

Should it work or I have some error in my schema? I didn’t find any active 
issues in jira so I guess the concept of optional fields should work just fine, 
also in C++.

 

The code is:

 

    std::unique_ptr<avro::InputStream> in = avro::memoryInputStream((const 
uint8_t*)&json[0], json.size()); // json is incoming data

    avro::ValidSchema validSchema;

    std::istringstream ins(schema); // schema is avro-schema

    try {

        avro::compileJsonSchema(ins, validSchema);

    }

    catch (const std::exception& e1) {

        std::string errstr = e1.what();

    }

    avro::DecoderPtr jd = avro::jsonDecoder(validSchema);

    avro::GenericDatum datum(validSchema);

    jd->init(*in);

    avro::decode(*jd, datum); //serialization with non-null data fails 
somewhere inside this step