github-actions[bot] commented on code in PR #69806:
URL: https://github.com/apache/doris/pull/69806#discussion_r4420179484
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java:
##########
@@ -327,36 +327,40 @@ public void refreshTableInternal(ExternalDatabase db,
ExternalTable table, long
private void refreshTableInternal(ExternalDatabase db, ExternalTable
table, long updateTime,
boolean invalidateTableLookup) {
boolean invalidateDatabase = invalidateTableLookup &&
table.getCatalog().getLowerCaseTableNames() != 0;
- table.advanceMetadataRefreshGeneration();
- table.unsetObjectCreated();
- if (updateTime > 0) {
- table.setUpdateTime(updateTime);
- }
- // Connector metadata is the row-count source, so invalidate it before
engine and row-count caches.
+ table.beginMetadataRefresh();
Review Comment:
[P1] Extend the refresh fence to replacement objects during database-wide
invalidation. `refreshTableInternal` calls `beginMetadataRefresh()` only on the
old `ExternalTable`, then `invalidateTableLookup()` clears the whole
names/object cache before `endMetadataRefresh()`. A concurrent lookup can
publish a new table object in that interval; its `activeMetadataRefreshes` is
zero, so a delayed `ConnectorTableNotFoundException` can pass
`observeTableLookup` and evict the just-recreated object, ID route, and
connector caches while the wide refresh is still running. Carry a
database-level refresh epoch/count (or equivalent) into observations and add
this replacement-publication interleaving test.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenSysExternalTable.java:
##########
@@ -96,13 +96,17 @@ private static long generateSysTableId(long sourceTableId,
String sysName) {
@Override
public Optional<ConnectorTableHandle> resolveConnectorTableHandle(
ConnectorSession session, ConnectorMetadata metadata) {
- String dbName = db != null ? db.getRemoteName() : "";
- Optional<ConnectorTableHandle> baseHandle =
- metadata.getTableHandle(session, dbName,
sourceTable.getRemoteName());
+ MetadataRead read = observeConnectorMetadata(session, metadata);
+ Optional<ConnectorTableHandle> baseHandle =
sourceTable.resolveConnectorTableHandle(session, metadata);
if (!baseHandle.isPresent()) {
return Optional.empty();
}
- return metadata.getSysTableHandle(session, baseHandle.get(),
sysTableName);
+ return read.execute(() -> metadata.getSysTableHandle(session,
baseHandle.get(), sysTableName));
Review Comment:
[P1] Retain the source observation through pinned system-table schema
loading. The new wrapper at `getSysTableHandle` is not reused by
`schemaCacheValueAt`: after resolving the sys handle, it calls
`metadata.getTableSchema(..., connectorSnapshot)` directly. Iceberg turns a
drop in that load into `ConnectorTableNotFoundException`, but no source cleanup
runs, so the base table's ID/name and connector caches remain stale. Thread a
source `MetadataRead` through `schemaCacheValueAt` (and close the temporary
session) and add a concurrent-drop test for `getFullSchemaAt`/system-table time
travel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
