morningman opened a new pull request, #69578:
URL: https://github.com/apache/doris/pull/69578
### What problem does this PR solve?
Issue Number: None
Related PR: #69571, #68142
Problem Summary:
**In short.** When a Paimon split's deletion vector file is missing, the
native readers report `NOT_FOUND`, and both file scanners take that for a
missing data file and skip the whole range. The query succeeds and silently
drops every surviving row of the data file. This PR reports a missing deletion
vector as an error, so the query fails instead. A missing data file is still
skipped as before.
**Background.** A Paimon split for a table with deletion vectors names a
data file and the deletion vector that marks its deleted rows. The native
readers open the deletion vector before they read a row: V1 in
`PaimonParquetReader` / `PaimonOrcReader::_init_deletion_vector`, V2 in
`TableReader::_parse_delete_predicates`, both through `DeletionVectorReader`.
`FileScanner` and `FileScannerV2` skip a range whose reader reports `NOT_FOUND`
while `ignore_not_found_file_in_external_table` is on (the default). That
setting exists for file listings that are cached by the FE and may name a file
that has since been removed. #68142 already exempted Iceberg ranges, whose
files all belong to a committed snapshot.
**The problem, and what it cost.** A deletion vector is never a stale
listing: it belongs to the snapshot that planned its data file. It can still be
missing, for example when Paimon expires the snapshot that wrote it while the
data file stays live under a newer one, or when it is looked up under a
partition name that is never found (#69571). The data file is still there, but
the scanners skip the whole range, so its surviving rows are dropped together
with the deleted ones. The BE logs `Skip missing file` with the path of the
data file, which exists. Paimon's own reader, the JNI reader and the Rust
reader fail on the same split instead.
**How this PR fixes it.** `DeletionVectorReader`, which opens every deletion
vector for V1 Paimon, V1 Iceberg and the V2 table reader, now reports a missing
file as `INTERNAL_ERROR` with the path of the deletion vector, both when it
opens the file and when it reads it. The scanners only skip `NOT_FOUND`, so the
query fails. A missing data file still reports `NOT_FOUND` and is skipped as
before, with or without a deletion vector. For Iceberg, which already failed on
any missing file (#68142), only the error code of a missing deletion vector
changes.
One boundary: if the data file is gone as well, the result depends on which
file the reader reaches first. FileScannerV2 parses the deletion vector while
it prepares the split, before it opens the data file, so such a split now
fails; in the local run below V1 Parquet failed too, while V1 ORC, which opens
the data file first, still skipped it.
**Results.** A local cluster with a `file://` Paimon catalog, two append
tables (Parquet and ORC, `deletion-vectors.enabled = true`), five rows each,
the first row deleted through a deletion vector the way a Spark `DELETE`
persists it:
| Files on storage | Reader | Before | After |
|---|---|---|---|
| data file and deletion vector | V1 / V2, Parquet / ORC | `2,3,4,5` |
`2,3,4,5` |
| data file only (deletion vector removed) | V1 / V2, Parquet / ORC | empty
result, query succeeds; `Skip missing file: path=<data file>` in be.INFO |
`[INTERNAL_ERROR]Deletion vector file is missing: path .../index/index-...-0,
failed to get file size ...: No such file or directory` |
| deletion vector only (data file removed) | V1 / V2, Parquet / ORC | empty
result | empty result (unchanged) |
| neither | V2, V1 Parquet | empty result | the deletion vector error above |
| neither | V1 ORC | empty result | empty result (unchanged) |
### Release note
A Paimon query now fails when a deletion vector file of its snapshot is
missing, instead of silently skipping the data file it applies to.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- `be/test/format/table/paimon_reader_test.cpp` (new): the V1
Parquet and ORC readers report `INTERNAL_ERROR` with the path for a missing
deletion vector file. These restore the V1 deletion vector tests of #65351 that
went away with `paimon_cpp_reader_test.cpp` in #67385.
- `be/test/format_v2/table/paimon_reader_test.cpp`: the V2 reader
fails the split for a missing deletion vector file, and a missing data file
with an existing deletion vector still reports `NOT_FOUND`.
- With the change reverted, the three missing-deletion-vector tests
fail and the data-file test still passes. The Iceberg helper test for a missing
deletion vector, `PaimonReaderTest`, the V2 Iceberg deletion vector tests and
`FileScanRangeUtilsTest` pass (51 tests).
- [x] Manual test (add detailed scripts or steps below)
- The cluster run in the table above, before and after replacing
`doris_be`.
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. A Paimon query whose deletion vector file is missing fails
instead of returning a result without the rows of that data file. A missing
data file is still skipped.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
