naveenp2708 commented on code in PR #57644:
URL: https://github.com/apache/spark/pull/57644#discussion_r3772872662
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/GraphValidations.scala:
##########
@@ -269,9 +271,13 @@ trait GraphValidations extends Logging {
)
t.specifiedSchema.foreach { ss =>
- // Check the inferred schema matches the specified schema. Used to
catch errors where the
- // inferred user-facing schema has columns that are not in the
specified one.
- if (inferredSchema != ss) {
+ // Check the specified schema matches the inferred schema once the
engine-owned reserved
+ // AUTO CDC metadata column(s) are set aside on both sides. The user
may omit them (the
+ // engine appends them at materialization) or declare them; comparing
both schemas with the
+ // reserved columns removed accepts either while still catching a
genuine mismatch in the
+ // remaining columns, and stays correct if more than one reserved
column is ever added.
+ if (AutoCdcMergeFlow.stripReservedFields(inferredSchema, resolver) !=
Review Comment:
Went with always materializing the engine-owned shape. DatasetManager strips
whatever reserved column the user declared and appends the inferred
engine-owned one, so the created table always matches what the MERGE writes.
While adding tests I also confirmed a conflicting-type declaration never
reaches materialization anyway, mergeSchemas can't merge a user-declared
__spark_autocdc_metadata of a different type against the inferred struct, so
it's rejected at inference with
USER_SPECIFIED_AND_INFERRED_SCHEMA_NOT_COMPATIBLE before MERGE time. Added a
negative test for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
