JingsongLi commented on code in PR #9547:
URL: https://github.com/apache/paimon/pull/9547#discussion_r3914489139
##########
paimon-python/pypaimon/read/reader/format_blob_reader.py:
##########
@@ -373,18 +389,10 @@ def _read_index(self) -> None:
if len(index_bytes) != index_length:
raise IOError("Invalid blob file: cannot read index")
- # Decompress blob lengths and compute offsets
- blob_lengths = DeltaVarintCompressor.decompress(index_bytes)
- blob_offsets = []
- offset = 0
- for length in blob_lengths:
- if length < 0:
- blob_offsets.append(-1)
- else:
- blob_offsets.append(offset)
- offset += length
- self.blob_lengths = blob_lengths
- self.blob_offsets = blob_offsets
+ # Readers own mutable lists; the cached immutable index is shared.
+ blob_lengths, blob_offsets = _decode_blob_index(index_bytes)
Review Comment:
Please use `file_path` as the cache key. Paimon BLOB files are immutable and
have unique paths, so the path already identifies the index. Keying the cache
by `index_bytes` forces every reader to read the complete index before it can
check the cache, hashes and compares a potentially large byte string, and
retains the compressed bytes alongside the decoded tuples. A path-keyed cache
can avoid all of these costs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
