xylaaaaa opened a new pull request, #66361:
URL: https://github.com/apache/doris/pull/66361
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
dbt-doris previously materialized every incremental batch and exposed Unique
Key upsert behavior under the standard `insert_overwrite` name. That made the
public strategy semantics differ from dbt Core and prevented native Doris
overwrite behavior.
This PR implements the dbt Core 1.12 incremental strategy contract for:
- `append` on Duplicate Key tables;
- `merge` as a full-row Merge-on-Write Unique Key upsert;
- `delete+insert`, using a single MOW upsert where equivalent and a staged
transactional `DELETE USING` plus `INSERT` path for Merge-on-Read targets;
- native whole-table, static-partition, and dynamic-partition
`insert_overwrite`.
It also adds early source/target key validation, duplicate-key protection,
`on_schema_change` handling with Doris schema-change polling, atomic
full-refresh table exchange, failed View-to-Table recovery, and cleanup of dbt
helper relations.
The adapter now targets dbt Core 1.12.x and Python 3.10 or newer.
`microbatch`, Doris 4.1 native `MERGE INTO`, partial merge columns, and
incremental predicates remain out of scope.
### Release note
Incremental strategy names now match dbt semantics. Projects that previously
used `insert_overwrite` for Doris Unique Key upserts must migrate to `merge`.
Native whole-table and partition `INSERT OVERWRITE` are now supported.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- `python -m pytest -q test/unit`: 61 passed on Python 3.12
- 20 Doris incremental functional tests passed on the original
implementation snapshot before this branch was rebased onto current master;
they were not rerun for this draft PR.
- flake8 passed for every Python file changed by this PR.
- `git diff --check origin/master...HEAD` passed.
- A full-tree flake8 invocation still reports the pre-existing
unused `pytest` import in `test/functional/adapter/test_basic.py`, which this
PR does not modify.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. `insert_overwrite` now performs a real overwrite; Unique Key
upsert uses `merge`.
- Does this need documentation?
- [ ] No.
- [x] Yes. Usage and migration notes are included in
`extension/dbt-doris/docs/incremental.zh-CN.md`.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
