PDGGK opened a new pull request, #1006:
URL: https://github.com/apache/hudi-rs/pull/1006
## Description
Fixes #925.
On a table without the record index, one record key can be stored in several
partitions. A non-global Java index leaves a table like that, and `append` here
produces one too. Deleting such a key without naming a partition removes only
one copy. After appending `a` to `city=sf` and to `city=nyc` and `b` to
`city=sf`, hudi 0.6.0 from Python gives this on copy-on-write and on
merge-on-read:
```
delete("id = 'a'") num_deletes=1 left: ('a', 'nyc'), ('b', 'sf')
delete_keys(['a']) num_deletes=1 left: ('a', 'nyc'), ('b', 'sf')
delete("id != 'b'") num_deletes=2 left: ('b', 'sf')
```
The `delete_keys` docstring says a bare key is looked up in every partition,
and the last line shows the same rows do go away when the predicate does not
take the record key path.
Both delete paths resolve keys with `tag_location`, which returns one
location per key. Upsert needs that, because a record has to land in one place.
A delete then rewrites or logs only the file group of the copy that was picked.
`delete_keys` now asks the index for every stored copy a key names: all of
them for an empty partition path, and only those in that partition otherwise.
`SimpleIndex` already found every copy during its scan and then kept one, so
the scan moved into a helper used by both `tag_location` and the new lookup.
`tag_location` and upsert behave as before.
The record index is global and holds one location per key, so this change
does not alter that path. With default options, where the record index is on,
the same appends and `delete("id = 'a'")` on 0.6.0 leave `('a', 'sf')`. I left
that alone because finding the other copies would need the scan that index
exists to avoid, and the issue is about non-global indexes.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`test_delete_without_partition_removes_every_copy_of_the_key` stores a key
in two partitions and twice in one of them, on both table types, and deletes it
both through `delete("id = 'a'")` and through `delete_keys` with an empty
partition. On unmodified main it fails with `CopyOnWrite, by_filter=true`,
`left: 1`, `right: 3`. Reverting only the copy-on-write or only the
merge-on-read part makes it fail for that table type.
`crates/core/tests/table_write_tests.rs` passes (93 tests), as do the
`index::` and `write::rewrite` unit tests (21). `cargo fmt --all -- --check`
and `cargo clippy -p hudi-core --all-targets --no-default-features --no-deps --
-D warnings` are clean.
I ran these with `--no-default-features`, so the `spill-rocksdb` and
`datafusion` builds were not exercised, and the Python binding was only run
against the released 0.6.0. The test builds the duplicate key with `append`; I
did not run it on a table written by Java or read the result back with Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]