[
https://issues.apache.org/jira/browse/CASSANALYTICS-204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jyothsna Konisa updated CASSANALYTICS-204:
------------------------------------------
Status: Ready to Commit (was: Review In Progress)
> CDC Avro schemas have invalid or colliding record names for UDTs and
> collections
> --------------------------------------------------------------------------------
>
> Key: CASSANALYTICS-204
> URL: https://issues.apache.org/jira/browse/CASSANALYTICS-204
> Project: Apache Cassandra Analytics
> Issue Type: Bug
> Components: CDC
> Reporter: Jyothsna Konisa
> Assignee: Jyothsna Konisa
> Priority: Normal
> Fix For: 0.5
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *Problem*
>
>
>
>
>
> The CDC Avro schema converter (CqlToAvroSchemaConverterImplementation)
> generates record names that are either invalid or not unique, for UDTs and
> for records nested inside collections.
>
>
>
> # Invalid names
>
>
> For lists and sets, the CQL type text is added to the record namespace, e.g.:
> cdc_demo.task.results.list<frozen<task_result>>.record_udt
>
> "<" and ">" are not allowed in Avro
> names. Avro 1.12+ rejects such schemas when parsing them
> (SchemaParseException: Namespace part "list<frozen<task_result>>" is
> invalid). Since consumers must parse the writer schema before decoding, no
> message of an affected table can be read by consumers or tools using Avro
> 1.12+ (e.g. Kafka UI). Avro 1.11 and older do not check namespace parts,
> which is why the issue is not visible with every consumer. CDC itself uses
> Avro 1.11.4; upgrading to 1.12 could break generating these schemas.
> *Affected:* UDTs or maps inside a list or set, e.g. list<frozen<udt>>,
> set<frozen<udt>>, list<frozen<map<text, text>>>.
>
>
>
>
> # Colliding names
> All UDT records are named record_udt, and all fields of a UDT (and the key
> and value of a map) share one namespace. So two different UDTs at the same
> level get the same full name, e.g. the fields of UDT person (home
> frozen<address>, mobile frozen<phone>) both become
> ...info.record_udt.record_udt.record_udt. When the schema is written as JSON
> (as registered in Schema Registry), Avro writes the second record as a
> reference to the first. The registered schema then describes the second field
> with the wrong fields, and consumers cannot decode it correctly.
>
> *Affected:* a UDT with fields of different UDT types, or a map whose key and
> value are different UDTs.
>
>
>
>
> *Fix*
>
>
>
>
>
> Name each record after its position in the table, so every name is valid
> and unique:
>
> - UDT records are named after the UDT, in the namespace of their position:
> ks.users.info.home.address (was
> ks.users.info.record_udt.record_udt.record_udt).
>
> - UDT fields add the field name to the namespace of their children.
>
>
> - List and set elements use the collection's namespace; the CQL type is no
> longer part of the name (it is still available in the cqlType property).
>
> - Records under a map key or value are namespaced by key/value when they
> contain a UDT, or when both key and value contain records.
>
> - Schemas without UDTs keep their current names (e.g. map entry records
> ks.t.m.array_map.array_map).
> ||Column type||Record||Before||After||
> |frozen<address>|UDT|ks.t.c.record_udt.record_udt|ks.t.c.address|
> |list<frozen<address>>|element
> UDT|ks.t.c.list<frozen<address>>.record_udt.record_udt
> (invalid)|ks.t.c.address|
> |list<frozen<map<text, text>>>|map entry|ks.t.c.list<frozen<map<text,
> text>>>.array_map.array_map (invalid)|ks.t.c.array_map.array_map|
> |frozen<person> (home frozen<address>, mobile frozen<phone>)|home /
> mobile|ks.t.c.record_udt.record_udt.record_udt for both
> (collision)|ks.t.c.home.address / ks.t.c.mobile.phone|
> |map<frozen<address>, frozen<phone>>|key / value
> UDT|ks.t.m.array_map.record_udt.record_udt for both
> (collision)|ks.t.m.array_map.key.address / ks.t.m.array_map.value.phone|
> |map<text, text>|map entry|ks.t.m.array_map.array_map|unchanged|
>
>
>
>
>
> Impact
>
>
>
>
>
> - UDTs are still identified by the record_udt logical type, so readers in
> Cassandra Analytics are unaffected.
>
> - The binary encoding of the data does not change, and the schema version
> UUID (derived from the CQL schema) does not change.
>
> - Record names change only for UDTs and records nested in lists/sets. UDT
> columns could not be published before CASSANALYTICS-203, so no data depends
> on the old names. With Schema Registry, a subject registered for a UDT table
> during earlier failed publish attempts may reject the new schema once
> (NAME_MISMATCH); deleting the subject or relaxing its compatibility resolves
> it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
