oglego opened a new pull request, #1013:
URL: https://github.com/apache/hudi-rs/pull/1013

   ## Description
   
   Closes #375
   
   User-provided options that contradicted what a table persists in 
`hoodie.properties` (e.g. `hoodie.table.type`) were silently overwritten by the 
persisted value. The table builder now returns an error when they conflict.
   
   I believe that I have implemented everything that we discussed within the 
ticket but if there is anything that I missed or that should be changed please 
let me know!
   
   This also picked up some whitespace fixes in 
hudi-rs/.github/workflows/ci.yml - feel free to throw that out if you want 
since it isn't technically related to this PR - this was grabbed when pushing 
to the repo to get through the checks that are ran!
   
   **How it works**
   
   - Every key in `hoodie.properties` is compared against the user options 
before merging. Keys that `HudiTableConfig` models are compared after parsing, 
so equivalent spellings are not flagged. Any other key, such as 
`hoodie.record.merge.mode`, is compared as a raw string.
   - Aliases are honored in both directions (e.g. 
`hoodie.table.ordering.fields` against a persisted 
`hoodie.table.precombine.field`).
   - All conflicts are reported together, sorted by key, through a new 
`ConfigError::ConflictingTableProperties`.
   - No conflict is raised for metadata-related configs 
(`MetadataTableEnabled`, `MetadataTablePartitions` and the unmodeled 
`hoodie.table.metadata.*` keys), `BasePath`, `Checksum`, `CreateSchema`, or an 
empty `DatabaseName`. The persisted value still wins for these, as before.
   - `build()` resolves on a clone of the resolver, so calling it twice on one 
builder compares against the original user options each time.
   - Background compaction reopens the table from its own resolved options, so 
that reopen skips the check through an internal 
`build_with_resolved_options()`. A later change to `hoodie.properties` can't 
fail it.
   - `HudiConfigValue` now derives `PartialEq` to compare parsed values.
   
   **Behavior change**
   
   Code that passes options contradicting the table used to have them silently 
dropped and now gets an error.
   
   **Cross-binding impact**
   
   The change adds a `ConfigError` variant and a `PartialEq` derive, both 
additive. Nothing in `crates/datafusion`, `python/`, `cpp/` or `crates/ffi` 
references `ConfigError`, and the Python binding converts `CoreError` into a 
`HudiCoreError` message string.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [ ] Manual tests
   - [ ] Details are described below
   
   Unit tests in `table/builder.rs` cover conflicts and equivalent spellings; 
values that fail to parse; exact comparison of string configs; aliases in both 
directions and a table that persists both a key and its alias; raw comparison 
of unmodeled keys; sorted multi-conflict reporting; and merge precedence. 
Builder-level tests against the V6 COW fixture cover a rejected conflict, a 
matching value (including building twice on one builder), and others.
   
   To see what a user gets on error, I ran 
`test_all_conflicts_are_reported_in_sorted_order` with a temporary `println!` 
of the error message (not included in this PR). The test passes two conflicting 
options, one modeled (`hoodie.table.type`) and one unmodeled 
(`hoodie.record.merge.mode`), and the output is:
   
   ```
   Config error: Conflicting user options with hoodie.properties: 
hoodie.record.merge.mode: user 'COMMIT_TIME_ORDERING', persisted 
'EVENT_TIME_ORDERING'; hoodie.table.type: user 'MERGE_ON_READ', persisted 
'COPY_ON_WRITE'
   ```
   
   Both conflicts are reported at once, sorted by key, each with the user value 
and the persisted value.
   
   Thanks in advance for the review I really appreciate it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to