dongjoon-hyun commented on PR #59643:
URL: https://github.com/apache/spark/pull/59643#issuecomment-6297525123
Thank you for the review and approval, @peter-toth .
1. You're right. I updated the "user-facing change" section of the PR
description for `format("hive")`.
2. Yes, this is the same as before this PR. I'd like to keep this PR as it
is because these cases are rare, and the exact condition needs to compare the
location of the existing table with the new one. A wrong mismatch there would
make a normal overwrite of a managed table fail with `LOCATION_ALREADY_EXISTS`.
3. Good point. This can happen today, not only after a future rewrite. For
example, the experimental native Parquet writer of Apache DataFusion Comet
replaces `DataWritingCommandExec(InsertIntoHadoopFsRelationCommand)` with its
own operator, so the callback never runs. A loud failure after the write would
still leave the old table entry pointing at the new files. So,
`DataSource.writeAndRead` now drops the table before the write, as before this
PR, when the executed plan has no `InsertIntoHadoopFsRelationCommand` with the
tag. I added a test with a planner strategy that drops the tag.
4. Fixed.
5. Fixed in both `MetastoreDataSourcesSuite` and
`DataFrameReaderWriterSuite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
