github-actions[bot] commented on code in PR #69806:
URL: https://github.com/apache/doris/pull/69806#discussion_r4450538170
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java:
##########
@@ -1823,7 +1899,7 @@ private long fetchRowCount(ConnectorSession session,
ConnectorMetadata metadata)
// feature flag because the listing can be a costly remote round-trip;
the connector self-samples,
// pins its classloader, and degrades to -1 rather than throwing.
if (GlobalVariable.enable_get_row_count_from_file_list) {
- long dataSize = metadata.estimateDataSizeByListingFiles(session,
handleOpt.get());
+ long dataSize = read.execute(() ->
metadata.estimateDataSizeByListingFiles(session, handleOpt.get()));
Review Comment:
[P2] Preserve guarded repair through Hive file-statistics reads
On an HMS that reports table absence from partition-name RPCs (for example,
this [pre-event-listener
setup](https://github.com/apache/hive/blob/rel/release-3.1.3/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L4160-L4170)),
warm the table cache, leave the statistics partition-name key cold, then drop
a partitioned table. With unknown HMS row/size stats and
`enable_get_row_count_from_file_list=true`, this new wrapper calls
`HiveConnectorMetadata.estimateDataSizeByListingFiles`, but `estimateDataSize`
catches the typed cause's `HmsClientException` wrapper as `RuntimeException`
and returns `-1`; `MetadataRead` never sees it. Sampled ANALYZE reaches the
same listing through `getChunkSizes -> listFileSizes`, whose FE call also lacks
a retained observation. Both leave stale names, object, numeric route and
connector caches after actual confirmed absence. Preserve the typed cause
through estimation and retain the observation around sample file-size
reads, while keeping ordinary statistics degradation, sampled-ANALYZE failure
behavior and scope closure. Add native absence/recreation controls for these
consumers; the existing `estimate` case tests streaming split estimation, not
this file-statistics path. These differ from the fixed display-listing and
Iceberg statistics cases.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java:
##########
@@ -148,17 +166,63 @@ public Optional<ConnectorTableHandle>
resolveConnectorTableHandle(
}
}
});
- String dbName = db != null ? db.getRemoteName() : "";
- // Some catalogs reject table requests for views with non-404 errors.
Classify views before
- // probing the table endpoint so snapshot loading can continue to view
expansion.
- Optional<ConnectorTableHandle> handle = supportsView() && isView()
- ? Optional.empty() : metadata.getTableHandle(session, dbName,
getRemoteName());
- // A view has no table handle. Check live view existence before
treating an empty handle as a
- // dropped table; the cached isView flag can outlive a remote view
deletion.
- if (!handle.isPresent() && !metadata.viewExists(session, dbName,
getRemoteName())) {
- observation.ifPresent(Runnable::run);
+ }
+
+ /** A synthetic system relation repairs the shared source identity rather
than its transient name. */
+ protected PluginDrivenExternalTable lookupRepairSource() {
+ return this;
+ }
+
+ /** Capture before resolving a handle, then retain through every provider
read made with that handle. */
+ public MetadataRead observeConnectorMetadata(ConnectorSession session,
ConnectorMetadata metadata) {
+ PluginDrivenExternalTable source = lookupRepairSource();
+ Connector connector = ((PluginDrivenExternalCatalog)
source.catalog).getConnector();
+ return new MetadataRead(source,
source.observeConnectorTableLookup(connector), session, metadata);
Review Comment:
[P2] Retain the observation through native scan partition reads
With hot FE/HMS table metadata but cold partition-name/derived-view caches,
drop a partitioned Hive table before an unfiltered scan finalizes. On HMS
setups with a pre-event listener, native listing reports source absence ([Hive
3.1.3
source](https://github.com/apache/hive/blob/rel/release-3.1.3/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L4160-L4170)).
`materializeLatest` returns a lightweight deferred pin, and
`getNameToPartitionItemsForScan` reaches `buildPartitionView`'s raw
`metadata.listPartitions` call; the eager `buildNameToPartitionItems` helper
also calls it directly. Neither retains this observation. The new typed Thrift
cause therefore aborts pruning/finalization while names, table object, numeric
route and connector caches remain stale; the guarded `planScan` call runs
later. Capture before handle resolution and retain through these
partition-builder provider calls, with native scan and delayed-recreation
coverage. This is a
separate consumer from the repaired MVCC materializer, `partition_values` and
SHOW/PARTITIONS paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
