[
https://issues.apache.org/jira/browse/AVRO-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Coutinho updated AVRO-2910:
---------------------------------
Attachment: test.avsc
test.avdl
> Random untion(__, null) data doesn't conform to schema
> ------------------------------------------------------
>
> Key: AVRO-2910
> URL: https://issues.apache.org/jira/browse/AVRO-2910
> Project: Apache Avro
> Issue Type: Bug
> Components: tools
> Reporter: Chris Coutinho
> Priority: Minor
> Attachments: sample.json, test.avdl, test.avsc
>
>
> The random Avro data generator tool doesn't generate data that conforms to a
> schema containing nullable fields.
> For example, the following test.avdl can be used to define a schema for
> single `test` record, where some of the fields are nullable based on `union`.
> There doesn't seem to be a difference between setting null first or last.
>
> {{@namespace("com.example")}}
> {{ protocol test {}}
> {{ record test}}
> {{{ string name; union \\{null, string}}}
> {{null_name1;}}
> {{ union \{string, null} null_name2;}}
> {{ int num;}}
> {{ union \{null, int} null_num1;}}
> {{ union \{int, null} null_num2;}}
> {{ }}}
> {{}}}
>
> Using the `idl2schemata` tool, the previous .avdl file generates the
> following (.avsc) schema
>
> {{{}}
> {{ "type" : "record",}}
> {{ "name" : "test",}}
> {{ "namespace" : "com.example",}}
> {{ "fields" : [}}
> {{{ "name" : "name", "type" : "string" }}}
> {{,}}
> {{{ "name" : "null_name1", "type" : [ "null", "string" ] }}}
> {{,}}
> {{{ "name" : "null_name2", "type" : [ "string", "null" ] }}}
> {{,}}
> {{{ "name" : "num", "type" : "int" }}}
> {{,}}
> {{{ "name" : "null_num1", "type" : [ "null", "int" ] }}}
> {{,}}
> {{{ "name" : "null_num2", "type" : [ "int", "null" ] }}}
> {{]}}
> {{ }}}
>
>
> Using the `random` tool and piping the result into `tojson`, you can see that
> the generated data doesn't conform to the schema:
>
>
> {{$ avro-tools random --count 10 --schema-file test.avsc - | avro-tools
> tojson --reader-schema-file test.avsc - > sample.json}}
>
> Note how the non-null random values are unnecessarily nested in the following
> examples, where the generated value is nested within an object with the data
> type as a key (e.g. {{"null_num2":\{"int":1517937645}` instead of
> `"null_num2":1517937645}})
>
> {{{"name":"xgpmamxstwwehhluowytabwleviffo","null_name1":null,"null_name2":null,"num":565397409,"null_num1":\\{"int":1893963068}}}
> {{,"null_num2":\{"int":1517937645}}}}
> {{{"name":"qexuommiixgjriduvqydexerym","null_name1":null,"null_name2":\\{"string":"oxflkduyjagrsrlnagnkxuwej"}}}
> {{,"num":1709296377,"null_num1":null,"null_num2":\{"int":1155695595}}}}
> {{{"name":"adjlcnnewythkqsufythqj","null_name1":null,"null_name2":\\{"string":"qttntcfnuxvetji"}}}
> {{,"num":-985375976,"null_num1":null,"null_num2":\{"int":1473838855}}}}
> {{{"name":"atvtwigrsgjsbqllinslarhayra","null_name1":null,"null_name2":null,"num":-829302451,"null_num1":null,"null_num2":null}
>
> \{"name":"mbshjkjmpoaotrrcmo","null_name1":null,"null_name2":\{"string":"fpoaply"}}}
> {{,"num":-528843282,"null_num1":null,"null_num2":null}}}
> {{{"name":"swnvwyhl","null_name1":\\{"string":"eppuugalnegxqlntgsjyjipynhmgsha"}}}
> {{,"null_name2":null,"num":1959434696,"null_num1":\{"int":401147221},"null_num2":\{"int":-1054848203}}}}
> {{{"name":"nxejciyeghtsrparo","null_name1":\\{"string":"pdnhnhh"}}}
> {{,"null_name2":null,"num":226278493,"null_num1":\{"int":-1921928686},"null_num2":null}}}
> {{{"name":"iqeoxaorhs","null_name1":null,"null_name2":\\{"string":"qjmjhy"}}}
> {{,"num":-1562871043,"null_num1":\{"int":819200163},"null_num2":null}}}
> {{{"name":"qumynkqmmvubsrwhdkutcwuupddqalmlwek","null_name1":null,"null_name2":\\{"string":"s"}}}
> {{,"num":-1023135104,"null_num1":null,"null_num2":\{"int":-1707061745}}}}
> {{{"name":"sajkrdfevvwokxpdrqhymjdvv","null_name1":null,"null_name2":\\{"string":"egiv"}}}
> {{,"num":389828169,"null_num1":\{"int":-511951083},"null_num2":null}}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
