github-actions[bot] commented on code in PR #66473:
URL: https://github.com/apache/doris/pull/66473#discussion_r3765680140
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -743,52 +745,147 @@ public List<org.apache.paimon.table.source.Split>
getPaimonSplitFromAPI() throws
if (PaimonScanParams.isPinnedEmptyScan(resolvedOptions)) {
return Collections.emptyList();
}
- Optional<Long> fileCreationTime =
PaimonScanParams.getPinnedFileCreationTime(resolvedOptions);
- if (fileCreationTime.isPresent()) {
- if (!(paimonTable instanceof FileStoreTable)) {
- throw new UserException("Paimon file-creation OPTIONS
require a data table.");
+ int[] projectedColumns = new int[0];
+ if
(!PaimonScanParams.getPinnedFileCreationTime(resolvedOptions).isPresent()) {
+ List<String> fieldNames =
paimonTable.rowType().getFieldNames();
+ projectedColumns = desc.getSlots().stream().mapToInt(
+ slot -> getFieldIndex(fieldNames,
slot.getColumn().getName()))
+ .toArray();
+ if (Arrays.stream(projectedColumns).anyMatch(index -> index <
0)) {
+ throw new UserException("Paimon scan schema does not
contain all bound Doris columns.");
}
- FileStoreTable fileStoreTable = (FileStoreTable) paimonTable;
- SnapshotReader snapshotReader =
fileStoreTable.newSnapshotReader()
- .withMode(ScanMode.ALL)
- .withSnapshot(Long.parseLong(
-
paimonTable.options().get(CoreOptions.SCAN_SNAPSHOT_ID.key())))
- .withManifestEntryFilter(entry ->
- entry.file().creationTimeEpochMillis() >=
fileCreationTime.get());
- preserveBatchScanFilters(fileStoreTable, snapshotReader);
- if (predicates != null) {
- predicates.forEach(snapshotReader::withFilter);
- }
- return snapshotReader.read().splits();
- }
- List<String> fieldNames = paimonTable.rowType().getFieldNames();
- int[] projected = desc.getSlots().stream().mapToInt(
- slot -> getFieldIndex(fieldNames,
slot.getColumn().getName()))
- .toArray();
- if (Arrays.stream(projected).anyMatch(index -> index < 0)) {
- throw new UserException("Paimon scan schema does not contain
all bound Doris columns.");
- }
- ReadBuilder readBuilder = paimonTable.newReadBuilder();
- TableScan scan = readBuilder.withFilter(predicates)
- .withProjection(projected)
- .newScan();
- PaimonMetricRegistry registry = new PaimonMetricRegistry();
- if (scan instanceof InnerTableScan) {
- scan = ((InnerTableScan) scan).withMetricRegistry(registry);
- }
- List<org.apache.paimon.table.source.Split> splits =
scan.plan().splits();
- PaimonScanMetricsReporter.report(source.getTargetTable(),
paimonTable.name(), registry);
- if (!registry.getAllGroups().isEmpty()) {
- registry.clear();
}
- return splits;
+ int[] projected = projectedColumns;
+ PaimonSplitTaskCacheKey cacheKey = createPaimonSplitTaskCacheKey(
+ relationSnapshot, paimonTable, resolvedOptions, projected);
+ return getOrLoadExternalScanTasks(cacheKey,
Review Comment:
[P2] Release Paimon tasks after conversion
This stores every fully materialized Paimon `DataSplit` in the statement
cache until execution ends, even after this node has converted the plan to
representations that no longer need those connector objects. In the native
path, `PaimonSplitCreator` builds independent file splits whose `paimonSplit`
field is null; JNI/count paths serialize the task into thrift before the local
split list is released. Before this change the `DataSplit`/`DataFileMeta` graph
could become collectible after conversion, but now the cache remains its owner
even when no alias reuses it, so statements with many Paimon files accumulate
that metadata through execution. Please bypass retention for plans consumed
into independent native splits, evict after the intended consumers finish, or
cache a compact converted representation, and cover a non-empty plan in the
lifetime test.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java:
##########
@@ -428,11 +447,10 @@ private List<HiveExternalMetaCache.HiveFileStatus>
selectFiles(List<FileCacheVal
long totalSize = 0;
for (FileCacheValue value : inputCacheValue) {
for (HiveExternalMetaCache.HiveFileStatus file : value.getFiles())
{
- file.setSplittable(value.isSplittable());
- file.setPartitionValues(value.getPartitionValues());
- file.setAcidInfo(value.getAcidInfo());
- fileList.add(file);
- totalSize += file.getLength();
+ HiveExternalMetaCache.HiveFileStatus sampledFile =
Review Comment:
[P2] Copy only the sampled Hive files
`selectFiles` now allocates a new `HiveFileStatus` for every listed file
before it knows which prefix the sample will consume. The returned
`ArrayList.subList` keeps the complete backing list reachable through
`splitAllFiles`, so even a tiny TABLESAMPLE on a large unpartitioned/non-batch
table creates one extra status wrapper per file (the previous code built only
an array of references). On very wide file sets this adds an avoidable O(all
files) allocation/GC spike. Please shuffle lightweight `(file, FileCacheValue)`
references and copy/decorate only the selected prefix, and add a
many-file/small-sample test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
