leaves12138 commented on code in PR #9257:
URL: https://github.com/apache/paimon/pull/9257#discussion_r3793884803
##########
paimon-core/src/main/java/org/apache/paimon/operation/ManifestEntryRunMerge.java:
##########
@@ -234,11 +232,6 @@ private static ManifestEntryRunMergePlan discoverRuns(
ManifestFileMeta meta = section.get(manifestIndex);
Discovery.DiscoveredManifest manifest =
discovered.get(manifestIndex);
if (manifest.fragmented) {
- long entryCount = meta.numAddedFiles() +
meta.numDeletedFiles();
- inMemoryEntries += entryCount;
- if (inMemoryEntries > MAX_IN_MEMORY_FRAGMENTED_ENTRIES) {
- return null;
- }
sources.add(new
ManifestEntryRunMergePlan.Source.FragmentedManifestSpec(meta));
Review Comment:
Manifest files are currently rolled at about 8 MiB, and in practice almost
all of them stay within that size. Therefore a single fragmented manifest is
already small in normal production data, while MAX_STREAM_CURSORS bounds how
many such manifests are opened together. Do we really need a separate
entry-count guard here? Entry count is also not a very accurate proxy for
retained bytes, and an extra threshold can unnecessarily send otherwise
manageable sections back to external sort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
