github-actions[bot] commented on code in PR #66307:
URL: https://github.com/apache/doris/pull/66307#discussion_r3851561447
##########
fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/event/MetastoreChangeDescriptor.java:
##########
@@ -104,16 +109,24 @@ private MetastoreChangeDescriptor(Op op, String dbName,
String tableName, String
/** A database-level change ({@code
REGISTER_/UNREGISTER_/RENAME_DATABASE}). */
public static MetastoreChangeDescriptor forDatabase(Op op, String dbName,
String dbNameAfter,
long eventId, long updateTime) {
- return new MetastoreChangeDescriptor(op, dbName, null, dbNameAfter,
null, null, updateTime, eventId);
+ return new MetastoreChangeDescriptor(
+ op, dbName, null, dbNameAfter, null, null, null, updateTime,
eventId);
}
/** A table-level change ({@code REGISTER_/UNREGISTER_/RENAME_TABLE},
{@code REFRESH_TABLE}). */
public static MetastoreChangeDescriptor forTable(Op op, String dbName,
String tableName,
String tableNameAfter, long eventId, long updateTime) {
- return new MetastoreChangeDescriptor(op, dbName, tableName, null,
tableNameAfter, null,
+ return new MetastoreChangeDescriptor(op, dbName, tableName, null,
tableNameAfter, null, null,
updateTime, eventId);
}
+ /** An in-place table refresh caused by ALTER TABLE, including columns
removed by the schema change. */
+ public static MetastoreChangeDescriptor forTableRefresh(String dbName,
String tableName,
Review Comment:
[P1] Version the new connector event SPI
`MetastoreChangeDescriptor` crosses the parent-first `ConnectorEventSource`
plugin boundary, but this public factory/accessor is added while both the FE
and connector jars still advertise API 6.0 and the frozen surface omits the
event types. A plugin built here is therefore accepted by an older 6.0 FE and
resolves that FE's old descriptor; the first ALTER event calling
`forTableRefresh` fails with `NoSuchMethodError`. That `Error` also bypasses
this driver's `Exception` catches after `isRunning` is set, wedging later
polling cycles. In the other direction, an old 6.0 plugin remains accepted and
emits the generic refresh descriptor, so removed-column cleanup is silently
skipped. Please add this event boundary to the frozen SPI surface, regenerate
the required baselines, bump/pin the connector API major, and cover both
version directions.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java:
##########
@@ -261,13 +275,16 @@ private void applyOne(PluginDrivenExternalCatalog
catalog, Connector connector,
break;
case UNREGISTER_DATABASE:
catalogMgr.unregisterExternalDatabaseFromEvent(before.localDbName, catalogName);
+ dropDatabaseConstraintsAndInvalidateMtmvs(catalogName,
before.localDbName);
Review Comment:
[P1] Make structural event cleanup replayable
These new database/table constraint drops and renames mutate only each FE's
memory, while `commitCursor` journals only the event high-water mark. A
follower can replay that upper bound before locally applying the descriptor and
then restart or be promoted; its local cursor resets to `-1`, and
first-sync/event-gap recovery performs only a cache refresh before advancing,
so it never reconstructs this transition. During a rolling upgrade, an old
follower deterministically applies the pre-change structural body and reaches
the same stale recovery state. A recreated name can then reactivate a false
PK/FK proof and enable join elimination. Please journal the master's transition
in cursor order or reconcile it during full refresh, and test follower
restart/failover plus an old-code structural apply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
