Jyothsna Konisa created CASSANALYTICS-204:
---------------------------------------------
Summary: CDC Avro schemas have invalid or colliding record names
for UDTs and collections
Key: CASSANALYTICS-204
URL: https://issues.apache.org/jira/browse/CASSANALYTICS-204
Project: Apache Cassandra Analytics
Issue Type: Bug
Reporter: Jyothsna Konisa
*Problem*
The CDC Avro schema converter (CqlToAvroSchemaConverterImplementation)
generates record names that are either invalid or not unique, for UDTs and for
records nested inside collections.
# 1. Invalid names
For lists and sets, the CQL type
text is added to the record namespace, e.g.:
cdc_demo.task.results.list<frozen<task_result>>.record_udt
"<" and ">" are not allowed in Avro
names. Avro 1.12+ rejects such schemas when parsing them (SchemaParseException:
Namespace part "list<frozen<task_result>>" is invalid). Since consumers must
parse the writer schema before decoding, no message of an affected table can be
read by consumers or tools using Avro 1.12+ (e.g. Kafka UI). Avro 1.11 and
older do not check namespace parts, which is why the
issue is not visible with every consumer. CDC itself uses Avro 1.11.4;
upgrading to 1.12 could break generating these schemas.
*Affected: *UDTs or maps
inside a list or set, e.g. list<frozen<udt>>, set<frozen<udt>>,
list<frozen<map<text, text>>>.
# 2. Colliding names
All UDT records are named
record_udt, and all fields of a UDT (and the key and value of a map) share one
namespace. So two different UDTs at the same level get the same full name, e.g.
the fields of UDT person (home frozen<address>, mobile frozen<phone>) both
become ...info.record_udt.record_udt.record_udt. When the schema is written as
JSON (as registered in Schema Registry), Avro writes the second record as a
reference to the first. The registered schema then describes the second field
with the wrong fields, and consumers cannot decode it correctly.
*Affected: *a UDT with fields of different UDT types, or a map whose key and
value are different UDTs.
*Fix *
Name each record after its position in the table, so every name is valid and
unique:
- UDT records are named after the UDT, in the namespace of their position:
ks.users.info.home.address (was
ks.users.info.record_udt.record_udt.record_udt).
- UDT fields add the field name to the namespace of their children.
- List and set elements use the collection's namespace; the CQL type is no
longer part of the name (it is still available in the cqlType property).
- Records under a map key or value are namespaced by key/value when they
contain a UDT, or when both key and value contain records.
- Schemas without UDTs keep their current names (e.g. map entry records
ks.t.m.array_map.array_map).
||Column type||Record||Before||After||
|frozen<address>|UDT|ks.t.c.record_udt.record_udt|ks.t.c.address|
|list<frozen<address>>|element
UDT|ks.t.c.list<frozen<address>>.record_udt.record_udt (invalid)|ks.t.c.address|
|list<frozen<map<text, text>>>|map entry|ks.t.c.list<frozen<map<text,
text>>>.array_map.array_map (invalid)|ks.t.c.array_map.array_map|
|frozen<person> (home frozen<address>, mobile frozen<phone>)|home /
mobile|ks.t.c.record_udt.record_udt.record_udt for both
(collision)|ks.t.c.home.address / ks.t.c.mobile.phone|
|map<frozen<address>, frozen<phone>>|key / value
UDT|ks.t.m.array_map.record_udt.record_udt for both
(collision)|ks.t.m.array_map.key.address / ks.t.m.array_map.value.phone|
|map<text, text>|map entry|ks.t.m.array_map.array_map|unchanged|
Impact
- UDTs are still identified by the record_udt logical type, so readers in
Cassandra Analytics are unaffected.
- The binary encoding of the data does not change, and the schema version UUID
(derived from the CQL schema) does not change.
- Record names change only for UDTs and records nested in lists/sets. UDT
columns could not be published before CASSANALYTICS-203, so no data depends on
the old names. With Schema Registry, a subject registered for a UDT table
during earlier failed publish attempts may reject the new schema once
(NAME_MISMATCH); deleting the subject or relaxing its compatibility resolves it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
