924060929 commented on code in PR #70675:
URL: https://github.com/apache/doris/pull/70675#discussion_r4608399054
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java:
##########
@@ -209,22 +212,38 @@ Set<Integer> getOrLoadEqualityDeleteFieldIds(
* miss parses at most once.
*/
ManifestCacheValue getManifestCacheValue(ManifestFile manifest, Table
table) {
+ if (requiresStatementRead(manifest, table)) {
+ return loadManifestCacheValue(manifest, table, manifest.content(),
false);
+ }
IcebergManifestEntryKey key = IcebergManifestEntryKey.of(manifest);
- return entry.get(key, k -> loadManifestCacheValue(
+ // A plaintext manifest may reveal file keys only after loading;
isolate unclassified misses too.
+ return entry.getForLoadScope(key, new Object(), k ->
loadManifestCacheValue(
Review Comment:
Could we organize manifest loading around an explicit statement-owned
reader, with the catalog cache used only for results that are safe to share
across statements?
Each `new Object()` here creates a per-invocation load scope, so even
concurrent reads of the same manifest within one statement cannot coalesce.
`retainIf` correctly prevents publication of key-bearing values, but it does
not establish statement-level reuse. Table loading already has this distinction
through `IcebergStatementScope` and `IcebergTableCache`.
Suggested structure:
1. Create a manifest reader in the existing statement scope, bound to the
table lease, and use its stable identity to coalesce in-flight reads within
that statement. Keep different statements isolated.
2. Route synchronous, future, cached-only, data and delete reads through
that reader. Consult the catalog cache only when the current table/manifest
permits shared reuse.
3. After parsing, publish only shareable plaintext results through the
existing invalidation-guarded cache publication path. Keep key-bearing results
with the current consumers.
4. Keep prefetch/results bounded and remove completed in-flight
registrations; do not retain every consumed manifest until statement end.
Preserve active IO leases until the actual callable exits.
Please cover same-statement coalescing, cross-statement isolation,
subsequent plaintext cache hits, and bounded result retention. This is an
architecture/performance suggestion, not a claim that sensitive results are
currently inserted and removed afterward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]