JingsongLi commented on code in PR #9473:
URL: https://github.com/apache/paimon/pull/9473#discussion_r3889309588
##########
paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionSplitGenerator.java:
##########
@@ -79,6 +86,62 @@ public List<SplitGroup> splitForBatch(List<DataFileMeta>
input) {
.collect(Collectors.toList());
}
+ private List<List<List<DataFileMeta>>>
packWithUniqueSidecars(List<List<DataFileMeta>> ranges) {
+ List<List<List<DataFileMeta>>> packed = new ArrayList<>();
+ List<List<DataFileMeta>> current = new ArrayList<>();
+ Set<DataFileMeta> seenSidecars = Collections.newSetFromMap(new
IdentityHashMap<>());
+ long currentWeight = 0;
+
+ for (List<DataFileMeta> range : ranges) {
+ long weight = incrementalRangeWeight(range, seenSidecars);
+ if (!current.isEmpty() && currentWeight + weight >
targetSplitSize) {
Review Comment:
[P2] Keep an oversized shared sidecar from forcing one split per anchor.
When the first spanning sidecar already exceeds targetSplitSize, currentWeight
starts above the target. The next small anchor therefore flushes the bin,
resets seenSidecars, and counts the same oversized sidecar again; this repeats
for every anchor (100 ten-byte normal ranges plus one 100 KB spanning video
with a 500-byte target produce 100 splits). Oversized videos are expected
because VideoRollingFileWriter deliberately treats its target as soft to keep a
physical video intact. Please separate the unavoidable shared fixed cost from
marginal per-range weight so covered anchors can still coalesce.
##########
paimon-python/pypaimon/read/scanner/data_evolution_split_generator.py:
##########
@@ -112,6 +128,99 @@ def weight_func(file_list: List[DataFileMeta]) -> int:
return splits
+ def _pack_with_unique_sidecars(
+ self, groups: List[List[DataFileMeta]]
+ ) -> List[List[List[DataFileMeta]]]:
+ packed = []
+ current = []
+ current_weight = 0
+ seen_sidecars = set()
+
+ for group in groups:
+ weight = self._incremental_group_weight(group, seen_sidecars)
+ if current and current_weight + weight > self.target_split_size:
Review Comment:
[P2] Keep an oversized shared sidecar from forcing one split per anchor. If
the first spanning sidecar already exceeds target_split_size, current_weight
remains above the target. The next small anchor flushes the pack, resets
seen_sidecars, and counts the same oversized sidecar again, so 100 tiny anchors
plus one shared 100 KB video with a 500-byte target produce 100 splits.
Oversized videos are valid because a physical video group cannot be cut at the
configured target. Please separate this unavoidable shared fixed cost from
marginal per-group weight so its covered anchors can still be coalesced.
##########
paimon-python/pypaimon/read/reader/format_blob_reader.py:
##########
@@ -74,8 +76,18 @@ def __init__(self, file_io: FileIO, file_path: str,
read_fields: List[str],
if file_size is not None and file_size > 0
else file_io.get_file_size(file_path)
)
- self._input_stream = file_io.new_input_stream(file_path)
- self._read_index()
+ cached_video_meta = (
+ video_meta_cache.get(file_path)
+ if self._is_video and video_meta_cache is not None
Review Comment:
[P2] Cache ordinary BLOB indexes as well. This cache is gated on _is_video,
but .blob writers also roll independently from normal files and blob compaction
can emit one .blob covering many normal anchors. DataEvolutionSplitRead then
constructs one FormatBlobReader per anchor, so each takes the miss path and
reopens and decompresses the full delta-varint index. I reproduced three
Parquet anchors plus one 3,000-row .blob opening and parsing the same .blob
three times in one packed split. Please cache immutable unselected blob_lengths
and blob_offsets too, while keeping payload streams and row selections
reader-local.
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionSplitRead.java:
##########
@@ -1231,7 +1242,7 @@ public static List<List<DataFileMeta>>
mergeRangesAndSort(List<DataFileMeta> fil
// group by row id range
ToLongFunction<DataFileMeta> maxSeqF = DataFileMeta::maxSequenceNumber;
RangeHelper<DataFileMeta> rangeHelper = new
RangeHelper<>(DataFileMeta::nonNullRowIdRange);
- List<List<DataFileMeta>> result =
rangeHelper.mergeOverlappingRanges(files);
+ List<List<DataFileMeta>> result = groupByNormalFileRange(files,
Function.identity());
Review Comment:
[P2] Reuse sidecar metadata across anchor readers. groupByNormalFileRange
reattaches a spanning BLOB or video to every normal anchor, and createReader
builds a fresh union-reader supplier for every group. Each supplier reaches
BlobFileFormat or VideoFileFormat, which reopens the physical file and rebuilds
its complete BlobFileMeta or VideoFileMeta; one packed split with 100 anchors
therefore performs 100 remote opens and full index parses inside one task.
Please keep an immutable per-split metadata cache keyed by physical sidecar and
derive each anchor selection from it, or otherwise initialize each sidecar once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
