AnishMahto commented on PR #59311: URL: https://github.com/apache/spark/pull/59311#issuecomment-6160155030
> If it is the persisted target schema, could you state that requirement in the contract, as SCD2 does? It will use the persisted target schema, as is documented by the caller in the next PR: https://github.com/apache/spark/pull/59044. This is intentionally documented by the caller and not in `Scd2VersionMap`, because its not a fundamental requirement of the version map data structure/contract. `buildVersionMap` will build exactly from the schema provided at call time. Callers are responsible for canonicalizing schemas across runs if they want repeatable reads case-sensitivity/spellings might change. > Is the set-operation restriction on SCD1 targets intended? The restriction is known. If users want to do set operations they'll have to deselect the operational `__spark_autocdc_metadata` column first, for example by putting a view on top of it. In almost all cases users will want to do this anyway, since they'll likely want to compute the operations purely on the replication of their upstream data source (keys and data columns), not including the operational metadata. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
