dongjoon-hyun commented on PR #59895: URL: https://github.com/apache/spark/pull/59895#issuecomment-6499162921
Thank you for the review ([pullrequestreview-5691175149](https://github.com/apache/spark/pull/59895#pullrequestreview-5691175149)). **Delta-based `DELETE` on a metadata column that the operation does not read** You're right that it returned the correct result on master. As szehon-ho also pointed out, I fixed the root cause first in SPARK-60673 (#59900), which makes the rewrite read the metadata columns that the command references without writing them. It is merged now, and I rebased this PR on it. So this `DELETE` succeeds again, and the description no longer says that it fails. In 07d03a07f72, I added `DELETE ... WHERE _partition = 'hr' AND id < 1.5` to `GroupBasedNoMetadataDeleteFromTableSuite` and `DeltaBasedNoMetadataDeleteFromTableSuite` with the scan schema, write schema and write log checks. **CHAR/VARCHAR targets** I confirmed it at the current head with the in-memory row-level table with an extra `name VARCHAR(10)` column, for both group-based and delta-based tables: - An `UPDATE`, or a `MERGE` with `UPDATE` or `INSERT` actions, that references `_partition` fails with `MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION`, both in the first iteration and in a later one. - On master, the first-iteration forms fail in the same way. As you traced, the later-iteration `UPDATE` completes there, but it keeps the old `index` value and writes `index` and `_partition` as data columns. The later-iteration `MERGE` fails with a different error. - `DELETE` and a `MERGE` with only `DELETE` actions work in both forms, and with this PR their later-iteration forms no longer write the metadata columns as data columns. So CHAR/VARCHAR targets are not covered by this PR. It only makes the later-iteration forms fail like the first-iteration forms there, instead of writing stale metadata. The cause is the copy made by `cleanAttrMetadata` in `ResolveRowLevelCommandAssignments`, and the first-iteration form already fails on master, so I'd like to handle it in a separate JIRA issue. I stated the limitation in the PR description. **PR description** I updated the description for both suggestions. It narrows the reason for the accept-any-schema exclusion like the code comment, and it describes the `DELETE` above with SPARK-60673. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
