JingsongLi commented on code in PR #9473:
URL: https://github.com/apache/paimon/pull/9473#discussion_r3889529997
##########
paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionSplitGenerator.java:
##########
@@ -79,6 +102,73 @@ public List<SplitGroup> splitForBatch(List<DataFileMeta>
input) {
.collect(Collectors.toList());
}
+ private List<List<List<DataFileMeta>>> packWithUniqueSidecars(
+ List<List<DataFileMeta>> ranges, Set<DataFileMeta> sharedSidecars)
{
+ List<List<List<DataFileMeta>>> packed = new ArrayList<>();
+ List<List<DataFileMeta>> current = new ArrayList<>();
+ Set<DataFileMeta> seenSidecars = Collections.newSetFromMap(new
IdentityHashMap<>());
+ long currentWeight = 0;
+
+ for (List<DataFileMeta> range : ranges) {
+ long weight = incrementalRangeWeight(range, seenSidecars,
Collections.emptySet());
+ if (current.isEmpty() && weight > targetSplitSize) {
+ weight = incrementalRangeWeight(range, seenSidecars,
sharedSidecars);
Review Comment:
[P2] Close an oversized sidecar component before charging a new one
After an oversized shared sidecar is ignored here, currentWeight retains
only the tiny anchor cost. A later independent sidecar can therefore appear to
fit and be appended to the same split: a 200 MiB sidecar spanning ranges 0-1
followed by a 100 MiB sidecar spanning ranges 2-3 produces one roughly 300 MiB
task with a 128 MiB target. The boundary between these coverage components can
be split without rereading either file. Please keep coalescing ranges covered
by the ignored sidecar, but flush the saturated component before admitting a
newly encountered independent sidecar.
##########
paimon-python/pypaimon/read/scanner/data_evolution_split_generator.py:
##########
@@ -112,6 +128,125 @@ def weight_func(file_list: List[DataFileMeta]) -> int:
return splits
+ def _pack_with_unique_sidecars(
+ self, groups: List[List[DataFileMeta]]
+ ) -> List[List[List[DataFileMeta]]]:
+ sidecar_occurrences = defaultdict(int)
+ for group in groups:
+ for identity in {
+ id(file) for file in group if self._is_sidecar(file)
+ }:
+ sidecar_occurrences[identity] += 1
+ shared_sidecars = {
+ identity for identity, count in sidecar_occurrences.items()
+ if count > 1
+ }
+
+ packed = []
+ current = []
+ current_weight = 0
+ seen_sidecars = set()
+
+ for group in groups:
+ weight = self._incremental_group_weight(
+ group, seen_sidecars, set()
+ )
+ if not current and weight > self.target_split_size:
+ weight = self._incremental_group_weight(
Review Comment:
[P2] End the oversized-sidecar exemption at its coverage boundary
This removes the first oversized shared sidecar from current_weight for the
lifetime of the pack. Once that sidecar no longer covers subsequent anchors, an
unrelated sidecar can still appear to fit: with a 128 MiB target, a 200 MiB
sidecar over anchors 0-1 plus an independent 100 MiB sidecar over 2-3 becomes
one roughly 300 MiB split, although they can be separate tasks without
duplicating either file. Please track the ignored sidecar component and flush
before admitting an unrelated sidecar after its coverage ends.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
