naveenp2708 commented on PR #57644: URL: https://github.com/apache/spark/pull/57644#issuecomment-5276703735
> Test coverage gap: the new tests cover data-only schemas, named vs implicit flows, wrong data columns, full metadata schemas, and SCD2 omission, but they do not cover the case where the AutoCDC flow’s effective case sensitivity differs from the Spark session. That is the important edge case for this change because the surrounding AutoCDC code derives resolver behavior from pipeline/flow SQL conf, not just the session. I would add coverage with session `spark.sql.caseSensitive=false` and pipeline/flow `spark.sql.caseSensitive=true` (and/or the reverse), plus a materialization-level test to verify the metadata column is appended or not appended under the effective resolver. I would also add a negative test where the user declares `__spark_autocdc_metadata` with the wrong type/nullability so validation catches it before MERGE time. Added coverage in a new AutoCdcReservedColumnMaterializationSuite. There's a materialization-level test with the session case-sensitive and the flow case-insensitive: the user declares the reserved column as __SPARK_AUTOCDC_METADATA and the created table ends up with exactly one engine-owned __spark_autocdc_metadata. I checked it has teeth, with the old session resolver it fails with AMBIGUOUS_REFERENCE on a duplicated metadata column, which is the case you flagged. Also added the wrong-type negative test, and the omit-and-append case lives there too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
