924060929 commented on code in PR #70675:
URL: https://github.com/apache/doris/pull/70675#discussion_r4608399054


##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java:
##########
@@ -209,22 +212,38 @@ Set<Integer> getOrLoadEqualityDeleteFieldIds(
      * miss parses at most once.
      */
     ManifestCacheValue getManifestCacheValue(ManifestFile manifest, Table 
table) {
+        if (requiresStatementRead(manifest, table)) {
+            return loadManifestCacheValue(manifest, table, manifest.content(), 
false);
+        }
         IcebergManifestEntryKey key = IcebergManifestEntryKey.of(manifest);
-        return entry.get(key, k -> loadManifestCacheValue(
+        // A plaintext manifest may reveal file keys only after loading; 
isolate unclassified misses too.
+        return entry.getForLoadScope(key, new Object(), k -> 
loadManifestCacheValue(

Review Comment:
   Could we organize manifest loading around an explicit statement-owned 
reader, with the catalog cache used only for results that are safe to share 
across statements?
   
   Each `new Object()` here creates a per-invocation load scope, so even 
concurrent reads of the same manifest within one statement cannot coalesce. 
`retainIf` correctly prevents publication of key-bearing values, but it does 
not establish statement-level reuse. Table loading already has this distinction 
through `IcebergStatementScope` and `IcebergTableCache`.
   
   Suggested structure:
   1. Create a manifest reader in the existing statement scope, bound to the 
table lease, and use its stable identity to coalesce in-flight reads within 
that statement. Keep different statements isolated.
   2. Route synchronous, future, cached-only, data and delete reads through 
that reader. Consult the catalog cache only when the current table/manifest 
permits shared reuse.
   3. After parsing, publish only shareable plaintext results through the 
existing invalidation-guarded cache publication path. Keep key-bearing results 
with the current consumers.
   4. Keep prefetch/results bounded and remove completed in-flight 
registrations; do not retain every consumed manifest until statement end. 
Preserve active IO leases until the actual callable exits.
   
   Please cover same-statement coalescing, cross-statement isolation, 
subsequent plaintext cache hits, and bounded result retention. This is an 
architecture/performance suggestion, not a claim that sensitive results are 
currently inserted and removed afterward.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to