yujun777 commented on issue #68767:
URL: https://github.com/apache/doris/issues/68767#issuecomment-6060338828
Thanks -- the correction is in the body now, and it splits the two things
that were conflated there:
- master does not reach the hook for an add at all
(`AddColumnOp#needChangeMTMVState` and `AddColumnsOp#needChangeMTMVState`
return false on master), so an add invalidates nothing there, and the interval
for an add arrives with #68646 rather than being widened by it;
- for the operations that do reach the hook, the interval pre-exists and is
the publication order itself: the schema is installed inside the table's write
lock and the hook runs after it is released. What #68646 changes is its width
-- from "one state change per dependent view" to "one query analysis per
dependent view".
On the reproduction: you are right that the snippet was schematic and not
valid SQL, and that the wrong result itself is not demonstrated here. What is
verified is the interval (the two lock scopes above), and that a name can move
at all: the same movement, in a shape that does not need a concurrent planner,
is demonstrated on #68646's `QUALIFY` thread -- a view created before that fix
keeps `... QUALIFY flag = 1` in the catalog, and after `DROP COLUMN flag` it
stays NORMAL and in sync holding no rows while the same query returns `(1,1)`.
What this interval adds is that the movement can be observed by a planner while
the view is still NORMAL: the view is not yet wrong for the query it was built
for, it is stale for the query the new schema makes of it.
A fixture for the deterministic test, which the interval needs and which the
scope-add regression does not cover:
```sql
CREATE TABLE t1 (id INT, flag INT) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id)
BUCKETS 1
PROPERTIES ("replication_num" = "1");
CREATE TABLE t2 (id INT) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "1");
INSERT INTO t1 VALUES (1, 1);
INSERT INTO t2 VALUES (1);
CREATE MATERIALIZED VIEW mv BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
AS SELECT o.id FROM t1 o WHERE EXISTS (SELECT 1 FROM t2 i WHERE i.id =
o.id AND flag = 1);
ALTER TABLE t2 ADD COLUMN flag INT DEFAULT 0; -- light: installed under
the write lock
```
`flag` inside the subquery is the outer table's before the add and `t2`'s
after it, so a query planned between the unlock and the invalidation binds the
new column and can be answered from a view whose rows were computed with the
outer one. The latch would sit between those two points -- after
`modifyTableLightSchemaChange` and before `MTMVService#alterTable` -- and the
plan built there is the interesting one with a cold cache
(`mtmv_cache_manage_num = 0`); the same SELECT with
`enable_materialized_view_rewrite = false` is the control for the rows.
On the mechanism notes: the existing index schema version is a good input
for the third option -- a view records the identity of the schema it refreshed
against and the rewrite requires it to still match -- since
`updateBaseIndexSchema` bumps it for a light change while leaving the data
visible versions alone, which is exactly the version that has to be compared.
And the observer side is why the first option (a transient barrier) is not
enough on its own: the interval there is a journal order rather than a lock
order, so the barrier would have to be replayable, or replaced by a comparison.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
