Yukang-Lian opened a new pull request, #70286:
URL: https://github.com/apache/doris/pull/70286
### Background
In local (non-cloud) mode, a DELETE is pushed to every replica as a
`TPushReq` and handled by `PushHandler::_do_streaming_ingestion`, which writes
an empty rowset carrying the delete predicate. The rowset's schema is a copy of
the tablet's max-version schema with its columns rebuilt from
`request.columns_desc` (the FE's current schema). When a compaction merges
rowsets, it uses the schema with the highest `schema_version` among its inputs.
Then `TabletSchema::merge_dropped_columns` adds each delete rowset's columns
that are missing from that schema as dropped columns, and `CHECK`s that none of
them is a key column.
### What problem does this PR solve?
Issue Number: None
Related PR: #36083 (made the cloud delete task use `request.schema_version`;
the local path was not changed)
PR Type: Bugfix
Problem Summary:
The local push handler takes the delete rowset's columns from the FE but
keeps the tablet's schema version, which can be stale. After a key column is
appended to a DUPLICATE KEY table with a VARIANT column, a DELETE on that
column followed by a full compaction aborts every replica with `Check failed:
!src_col->is_key() <col> is key column, should not be dropped.` This PR sets
the delete rowset's schema version from `request.schema_version`, as the cloud
delete task already does, and adds a regression case. Cloud mode is not
affected.
### Details
- **Frequency:** Deterministic once the sequence below happens; found by a
randomized compaction/schema-change stress test.
- **Root cause:**
1. `ALTER TABLE ... ADD COLUMN <k> <type> KEY` appends a key column after
the existing keys. If the short key count does not change and the base tablet
has no delete predicates, the BE uses a linked schema change (`sc_sorting: 0,
sc_directly: 0`).
2. When the tablet schema has a VARIANT column,
`BaseBetaRowsetWriter::add_rowset` replaces the rowset schema with the linked
(old) rowset's schema to keep the variant subcolumns. All rowsets of the new
tablet therefore keep the old schema version and lack `<k>`.
3. The new tablet is created with schema version 0, and nothing raises its
max-version schema until a load arrives.
4. A DELETE that references `<k>` (e.g. `... AND <k> IS NULL`) is pushed
before any load. Its rowset has `<k>` in its columns (from `columns_desc`) but
the tablet's stale schema version, which is not higher than the linked rowsets'
version.
5. Full compaction picks a linked rowset's schema (without `<k>`) as the
target, `merge_dropped_columns` sees `<k>` from the delete rowset as a dropped
key column, and the `CHECK` aborts the BE. Every replica runs the same
compaction and crashes.
- **Why existing tests missed it:** It needs a VARIANT column, a linked
add-key schema change, and a delete on the new key before any other load. No
existing case combines them.
- **Fix:** When the columns are rebuilt from `columns_desc`, also set the
schema version from `request.schema_version` (only when it is set). The delete
rowset then has the FE's version and becomes the compaction target; older
rowsets are read with the missing key column filled by its default value, as
after any light schema change. The tablet's max-version schema is intentionally
left untouched (same as the cloud path): updating it from this rebuilt schema
would drop the variant's extracted subcolumns.
- **Other paths and branches:** The cloud delete task already sets the
version. The linked schema change keeping the old rowset schema for VARIANT
tables is left as is; with this fix it no longer leads to a crash.
### Testing
- New regression case
`schema_change_p0/test_delete_after_linked_add_key_variant`: DUPLICATE KEY
table with a VARIANT column, two loads, `ADD COLUMN k4 DATE KEY` (linked schema
change), `DELETE ... WHERE k1 = 1 AND k4 IS NULL`, full compaction, then
queries before/after compaction and after another load. Without the fix the
full compaction aborts the BE.
- The root cause was confirmed from BE logs of a failed stress run: the
tablet's schema change was linked, four delete pushes landed before any load,
and full compaction over all versions aborted with the `CHECK` above.
- The new case has not been run locally yet; it runs in CI.
### Release note
Fix a BE crash in compaction when a DELETE references a key column that was
just added to a table with a VARIANT column (local mode).
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Partially verified: the new regression case has not
been run locally; it runs in CI.
### AI Disclosure
- Tool:
- Generated:
- Human review:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
