Andreas Neumann created SPARK-58327:
---------------------------------------
Summary: Consider whether AutoCDC should reject null fields within
a multi-column (struct) sequencing value
Key: SPARK-58327
URL: https://issues.apache.org/jira/browse/SPARK-58327
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 5.0.0
Reporter: Andreas Neumann
This is an open question / investigation, not a decided fix -- the desired
behavior is not yet clear.
Background: ScdBatchValidator.validateMicrobatch rejects a microbatch row whose
sequencing expression evaluates to null
(AUTOCDC_MICROBATCH_VALIDATION.NULL_SEQUENCE: 'All rows must have a non-null
sequencing value.'). This guards the ordering key that reconciliation relies on.
Gap to consider: for a multi-column sequence expressed as a struct (e.g.
struct(seq1, seq2)), a Spark struct built from columns is non-null at the top
level even when one of its fields is null. So a row with, say, seq2 = null
inside an otherwise non-null struct passes the NULL_SEQUENCE check today -- the
validator only tests whether the whole sequencing value is null, not whether
any field within it is null. Such a row would then flow into reconciliation and
be used as (part of) the ordering key, where a null sub-field has undefined /
degenerate ordering behavior.
Open questions to resolve (no decision made yet):
- Should a null field within a struct sequencing value be rejected at all, or
is it acceptable/meaningful (e.g. treat null as a well-defined ordering
position)?
- If rejected, at what granularity -- any null field, or only leading/all
fields -- and with what error (extend
AUTOCDC_MICROBATCH_VALIDATION.NULL_SEQUENCE, or a new sub-condition)?
- Does the same concern apply to nested structs / arrays used as sequencing,
not just a flat top-level struct?
- What is the ordering semantics we actually want for null sub-fields (SQL
NULLS FIRST/LAST vs. reject), and is it consistent between SCD1 and SCD2?
Context: surfaced while adding schema tests for struct sequencing in
SPARK-58319 (a nullable struct field flows through to the __START_AT / __END_AT
column types). This ticket is only about the runtime validation of sequencing
*values*, separate from the schema-level nullability handled there. Decide the
intended behavior before implementing anything.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
