[
https://issues.apache.org/jira/browse/CASSANALYTICS-204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jyothsna Konisa updated CASSANALYTICS-204:
------------------------------------------
Description:
*Problem*
The CDC Avro schema converter (CqlToAvroSchemaConverterImplementation)
generates record names that are either invalid or not unique, for UDTs and for
records nested inside collections.
# Invalid names
For lists and sets, the CQL type text is added to the record namespace, e.g.:
cdc_demo.task.results.list<frozen<task_result>>.record_udt
"<" and ">" are not allowed in Avro
names. Avro 1.12+ rejects such schemas when parsing them (SchemaParseException:
Namespace part "list<frozen<task_result>>" is invalid). Since consumers must
parse the writer schema before decoding, no message of an affected table can be
read by consumers or tools using Avro 1.12+ (e.g. Kafka UI). Avro 1.11 and
older do not check namespace parts, which is why the issue is not visible with
every consumer. CDC itself uses Avro 1.11.4; upgrading to 1.12 could break
generating these schemas.
*Affected: *UDTs or maps inside a list or set, e.g. list<frozen<udt>>,
set<frozen<udt>>, list<frozen<map<text, text>>>.
# 2. Colliding names
All UDT records are named record_udt, and all fields of a UDT (and the key and
value of a map) share one namespace. So two different UDTs at the same level
get the same full name, e.g. the fields of UDT person (home frozen<address>,
mobile frozen<phone>) both become ...info.record_udt.record_udt.record_udt.
When the schema is written as JSON (as registered in Schema Registry), Avro
writes the second record as a reference to the first. The registered schema
then describes the second field with the wrong fields, and consumers cannot
decode it correctly.
*Affected: *a UDT with fields of different UDT types, or a map whose key and
value are different UDTs.
*Fix *
Name each record after its position in the table, so every name is valid and
unique:
- UDT records are named after the UDT, in the namespace of their position:
ks.users.info.home.address (was
ks.users.info.record_udt.record_udt.record_udt).
- UDT fields add the field name to the namespace of their children.
- List and set elements use the collection's namespace; the CQL type is no
longer part of the name (it is still available in the cqlType property).
- Records under a map key or value are namespaced by key/value when they
contain a UDT, or when both key and value contain records.
- Schemas without UDTs keep their current names (e.g. map entry records
ks.t.m.array_map.array_map).
||Column type||Record||Before||After||
|frozen<address>|UDT|ks.t.c.record_udt.record_udt|ks.t.c.address|
|list<frozen<address>>|element
UDT|ks.t.c.list<frozen<address>>.record_udt.record_udt (invalid)|ks.t.c.address|
|list<frozen<map<text, text>>>|map entry|ks.t.c.list<frozen<map<text,
text>>>.array_map.array_map (invalid)|ks.t.c.array_map.array_map|
|frozen<person> (home frozen<address>, mobile frozen<phone>)|home /
mobile|ks.t.c.record_udt.record_udt.record_udt for both
(collision)|ks.t.c.home.address / ks.t.c.mobile.phone|
|map<frozen<address>, frozen<phone>>|key / value
UDT|ks.t.m.array_map.record_udt.record_udt for both
(collision)|ks.t.m.array_map.key.address / ks.t.m.array_map.value.phone|
|map<text, text>|map entry|ks.t.m.array_map.array_map|unchanged|
Impact
- UDTs are still identified by the record_udt logical type, so readers in
Cassandra Analytics are unaffected.
- The binary encoding of the data does not change, and the schema version UUID
(derived from the CQL schema) does not change.
- Record names change only for UDTs and records nested in lists/sets. UDT
columns could not be published before CASSANALYTICS-203, so no data depends on
the old names. With Schema Registry, a subject registered for a UDT table
during earlier failed publish attempts may reject the new schema once
(NAME_MISMATCH); deleting the subject or relaxing its compatibility resolves it.
was:
*Problem*
The CDC Avro schema converter (CqlToAvroSchemaConverterImplementation)
generates record names that are either invalid or not unique, for UDTs and for
records nested inside collections.
# 1. Invalid names
For lists and sets, the CQL type
text is added to the record namespace, e.g.:
cdc_demo.task.results.list<frozen<task_result>>.record_udt
"<" and ">" are not allowed in Avro
names. Avro 1.12+ rejects such schemas when parsing them (SchemaParseException:
Namespace part "list<frozen<task_result>>" is invalid). Since consumers must
parse the writer schema before decoding, no message of an affected table can be
read by consumers or tools using Avro 1.12+ (e.g. Kafka UI). Avro 1.11 and
older do not check namespace parts, which is why the
issue is not visible with every consumer. CDC itself uses Avro 1.11.4;
upgrading to 1.12 could break generating these schemas.
*Affected: *UDTs or maps
inside a list or set, e.g. list<frozen<udt>>, set<frozen<udt>>,
list<frozen<map<text, text>>>.
# 2. Colliding names
All UDT records are named
record_udt, and all fields of a UDT (and the key and value of a map) share one
namespace. So two different UDTs at the same level get the same full name, e.g.
the fields of UDT person (home frozen<address>, mobile frozen<phone>) both
become ...info.record_udt.record_udt.record_udt. When the schema is written as
JSON (as registered in Schema Registry), Avro writes the second record as a
reference to the first. The registered schema then describes the second field
with the wrong fields, and consumers cannot decode it correctly.
*Affected: *a UDT with fields of different UDT types, or a map whose key and
value are different UDTs.
*Fix *
Name each record after its position in the table, so every name is valid and
unique:
- UDT records are named after the UDT, in the namespace of their position:
ks.users.info.home.address (was
ks.users.info.record_udt.record_udt.record_udt).
- UDT fields add the field name to the namespace of their children.
- List and set elements use the collection's namespace; the CQL type is no
longer part of the name (it is still available in the cqlType property).
- Records under a map key or value are namespaced by key/value when they
contain a UDT, or when both key and value contain records.
- Schemas without UDTs keep their current names (e.g. map entry records
ks.t.m.array_map.array_map).
||Column type||Record||Before||After||
|frozen<address>|UDT|ks.t.c.record_udt.record_udt|ks.t.c.address|
|list<frozen<address>>|element
UDT|ks.t.c.list<frozen<address>>.record_udt.record_udt (invalid)|ks.t.c.address|
|list<frozen<map<text, text>>>|map entry|ks.t.c.list<frozen<map<text,
text>>>.array_map.array_map (invalid)|ks.t.c.array_map.array_map|
|frozen<person> (home frozen<address>, mobile frozen<phone>)|home /
mobile|ks.t.c.record_udt.record_udt.record_udt for both
(collision)|ks.t.c.home.address / ks.t.c.mobile.phone|
|map<frozen<address>, frozen<phone>>|key / value
UDT|ks.t.m.array_map.record_udt.record_udt for both
(collision)|ks.t.m.array_map.key.address / ks.t.m.array_map.value.phone|
|map<text, text>|map entry|ks.t.m.array_map.array_map|unchanged|
Impact
- UDTs are still identified by the record_udt logical type, so readers in
Cassandra Analytics are unaffected.
- The binary encoding of the data does not change, and the schema version UUID
(derived from the CQL schema) does not change.
- Record names change only for UDTs and records nested in lists/sets. UDT
columns could not be published before CASSANALYTICS-203, so no data depends on
the old names. With Schema Registry, a subject registered for a UDT table
during earlier failed publish attempts may reject the new schema once
(NAME_MISMATCH); deleting the subject or relaxing its compatibility resolves it.
> CDC Avro schemas have invalid or colliding record names for UDTs and
> collections
> --------------------------------------------------------------------------------
>
> Key: CASSANALYTICS-204
> URL: https://issues.apache.org/jira/browse/CASSANALYTICS-204
> Project: Apache Cassandra Analytics
> Issue Type: Bug
> Reporter: Jyothsna Konisa
> Priority: Normal
>
> *Problem*
>
>
>
>
>
> The CDC Avro schema converter (CqlToAvroSchemaConverterImplementation)
> generates record names that are either invalid or not unique, for UDTs and
> for records nested inside collections.
>
>
>
> # Invalid names
>
>
> For lists and sets, the CQL type text is added to the record namespace, e.g.:
> cdc_demo.task.results.list<frozen<task_result>>.record_udt
>
> "<" and ">" are not allowed in Avro
> names. Avro 1.12+ rejects such schemas when parsing them
> (SchemaParseException: Namespace part "list<frozen<task_result>>" is
> invalid). Since consumers must parse the writer schema before decoding, no
> message of an affected table can be read by consumers or tools using Avro
> 1.12+ (e.g. Kafka UI). Avro 1.11 and older do not check namespace parts,
> which is why the issue is not visible with every consumer. CDC itself uses
> Avro 1.11.4; upgrading to 1.12 could break generating these schemas.
> *Affected: *UDTs or maps inside a list or set, e.g. list<frozen<udt>>,
> set<frozen<udt>>, list<frozen<map<text, text>>>.
>
>
>
>
> # 2. Colliding names
> All UDT records are named record_udt, and all fields of a UDT (and the key
> and value of a map) share one namespace. So two different UDTs at the same
> level get the same full name, e.g. the fields of UDT person (home
> frozen<address>, mobile frozen<phone>) both become
> ...info.record_udt.record_udt.record_udt. When the schema is written as JSON
> (as registered in Schema Registry), Avro writes the second record as a
> reference to the first. The registered schema then describes the second field
> with the wrong fields, and consumers cannot decode it correctly.
>
> *Affected: *a UDT with fields of different UDT types, or a map whose key and
> value are different UDTs.
>
>
>
>
> *Fix *
>
>
>
>
>
> Name each record after its position in the table, so every name is valid
> and unique:
>
> - UDT records are named after the UDT, in the namespace of their position:
> ks.users.info.home.address (was
> ks.users.info.record_udt.record_udt.record_udt).
>
> - UDT fields add the field name to the namespace of their children.
>
>
> - List and set elements use the collection's namespace; the CQL type is no
> longer part of the name (it is still available in the cqlType property).
>
> - Records under a map key or value are namespaced by key/value when they
> contain a UDT, or when both key and value contain records.
>
> - Schemas without UDTs keep their current names (e.g. map entry records
> ks.t.m.array_map.array_map).
> ||Column type||Record||Before||After||
> |frozen<address>|UDT|ks.t.c.record_udt.record_udt|ks.t.c.address|
> |list<frozen<address>>|element
> UDT|ks.t.c.list<frozen<address>>.record_udt.record_udt
> (invalid)|ks.t.c.address|
> |list<frozen<map<text, text>>>|map entry|ks.t.c.list<frozen<map<text,
> text>>>.array_map.array_map (invalid)|ks.t.c.array_map.array_map|
> |frozen<person> (home frozen<address>, mobile frozen<phone>)|home /
> mobile|ks.t.c.record_udt.record_udt.record_udt for both
> (collision)|ks.t.c.home.address / ks.t.c.mobile.phone|
> |map<frozen<address>, frozen<phone>>|key / value
> UDT|ks.t.m.array_map.record_udt.record_udt for both
> (collision)|ks.t.m.array_map.key.address / ks.t.m.array_map.value.phone|
> |map<text, text>|map entry|ks.t.m.array_map.array_map|unchanged|
>
>
>
>
>
> Impact
>
>
>
>
>
> - UDTs are still identified by the record_udt logical type, so readers in
> Cassandra Analytics are unaffected.
>
> - The binary encoding of the data does not change, and the schema version
> UUID (derived from the CQL schema) does not change.
>
> - Record names change only for UDTs and records nested in lists/sets. UDT
> columns could not be published before CASSANALYTICS-203, so no data depends
> on the old names. With Schema Registry, a subject registered for a UDT table
> during earlier failed publish attempts may reject the new schema once
> (NAME_MISMATCH); deleting the subject or relaxing its compatibility resolves
> it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
