ArulJerald commented on PR #3991:
URL: https://github.com/apache/iceberg-python/pull/3991#issuecomment-5712328283
@Fokko you are right manifest files is immutable.
Why we have to cache manifest files: to avoid re-reading (re-parsing) the
same object multiple times.
Example:
when you create a table and append data the first time, it creates
ManifestFileA and a manifest list, ManifestList1 = [A]. Since manifest files
are immutable, the next append doesn't rewrite A — it creates a new
ManifestFileB and a new manifest list, ManifestList2 = [A, B]. Appending again
creates ManifestFileC and ManifestList3 = [A, B, C].
The catch:
it's not that "current data" never benefits from the cache and "historical
data" always does. What actually determines a cache hit is whether a given
manifest file (like A or B) was already parsed earlier in the same process —
via any manifest list, current or historical. Whichever query touches a
manifest first pays the parsing cost; every later query that references that
same manifest gets a cache hit, regardless of whether it's asking for current
or historical data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
