ColinLeeo commented on code in PR #19198:
URL: https://github.com/apache/iotdb/pull/19198#discussion_r4559032168
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/buffer/TimeSeriesMetadataCache.java:
##########
@@ -316,6 +317,216 @@ public TimeseriesMetadata get(
}
}
+ /**
+ * Load the TimeseriesMetadata of {@code allSensors} of a device, reading
the device in a single
+ * batch when the per-measurement cache does not already hold all of them.
This is used by
+ * wide-table scans (a device with many columns) to avoid walking the device
metadata index once
+ * per column / leaf node.
+ *
+ * <p>The returned map always holds copies of the cached (or freshly read)
metadata, so the caller
+ * may apply {@code setModified} and {@code setChunkMetadataLoader} without
corrupting cached
+ * entries — matching {@link #get}.
+ *
+ * @param filePath path of the tsfile
+ * @param tsFileID id of the tsfile
+ * @param device target device
+ * @param allSensors measurements the caller needs, including the time
column keyed by ""
+ * @param queryContext query context used for metrics and to detect external
tsfiles
+ * @param ignoreNotExists whether a device missing from a file that claims a
device-level time
+ * index should be skipped instead of reported as an inconsistency
+ * @return map from measurementId (including the time column, keyed by "")
to TimeseriesMetadata;
+ * entries whose series is empty (count == 0) are mapped to null
+ * @throws IOException if the metadata cannot be read from disk or the
device is unexpectedly
+ * absent
+ */
+ public Map<String, TimeseriesMetadata> getDeviceTimeSeriesMetadata(
+ String filePath,
+ TsFileID tsFileID,
+ IDeviceID device,
+ Set<String> allSensors,
+ QueryContext queryContext,
+ boolean ignoreNotExists)
+ throws IOException {
+ long startTime = System.nanoTime();
+ long loadBloomFilterTime = 0;
+ boolean externalTsFile = queryContext.isExternalTsFileScan();
+ LongConsumer timeSeriesMetadataIoSizeRecorder =
+
queryContext.getQueryStatistics().getLoadTimeSeriesMetadataActualIOSize()::addAndGet;
+ LongConsumer bloomFilterIoSizeRecorder =
+
queryContext.getQueryStatistics().getLoadBloomFilterActualIOSize()::addAndGet;
+ boolean cacheHit = true;
+
+ try {
+ if (CACHE_ENABLE && !externalTsFile) {
+ Map<String, TimeseriesMetadata> cached =
+ getCachedDeviceTimeSeriesMetadata(tsFileID, device, allSensors);
+ if (cached != null) {
+ return cached;
+ }
+ // serialize concurrent reads of the same device+file (same lock key
as get) so threads
+ // don't each deserialize the whole device and race on the shared
reader
+ synchronized (
+ devices.computeIfAbsent(device.toString() + SEPARATOR + filePath,
WeakReference::new)) {
Review Comment:
Kept the monitor spanning the device read intentionally: it is the same
per-(device,file) monitor as `get()`, and it is now only taken on a cache miss,
with one batch read per device/file. Noted as a trade-off rather than changed.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/buffer/TimeSeriesMetadataCache.java:
##########
@@ -316,6 +317,216 @@ public TimeseriesMetadata get(
}
}
+ /**
+ * Load the TimeseriesMetadata of {@code allSensors} of a device, reading
the device in a single
+ * batch when the per-measurement cache does not already hold all of them.
This is used by
+ * wide-table scans (a device with many columns) to avoid walking the device
metadata index once
+ * per column / leaf node.
+ *
+ * <p>The returned map always holds copies of the cached (or freshly read)
metadata, so the caller
+ * may apply {@code setModified} and {@code setChunkMetadataLoader} without
corrupting cached
+ * entries — matching {@link #get}.
+ *
+ * @param filePath path of the tsfile
+ * @param tsFileID id of the tsfile
+ * @param device target device
+ * @param allSensors measurements the caller needs, including the time
column keyed by ""
+ * @param queryContext query context used for metrics and to detect external
tsfiles
+ * @param ignoreNotExists whether a device missing from a file that claims a
device-level time
+ * index should be skipped instead of reported as an inconsistency
+ * @return map from measurementId (including the time column, keyed by "")
to TimeseriesMetadata;
+ * entries whose series is empty (count == 0) are mapped to null
+ * @throws IOException if the metadata cannot be read from disk or the
device is unexpectedly
+ * absent
+ */
+ public Map<String, TimeseriesMetadata> getDeviceTimeSeriesMetadata(
Review Comment:
Agreed this would be the cleaner long-term home. Moving the batch into
`get()`'s miss path needs a selective group-level reader (read only the leaf
groups overlapping `allSensors`), which does not exist yet; we kept the
separate path for this PR and recorded the selective reader as a follow-up.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/FileLoaderUtils.java:
##########
@@ -375,38 +412,108 @@ private static AbstractAlignedTimeSeriesMetadata
loadAlignedTimeSeriesMetadataFr
} else {
List<TimeseriesMetadata> valueTimeSeriesMetadataList =
new ArrayList<>(valueMeasurementList.size());
- // if all the queried aligned sensors does not exist, we will return
null
- boolean exist = false;
for (String measurement : valueMeasurementList) {
- TimeseriesMetadata valueColumn =
+ valueTimeSeriesMetadataList.add(
cache.get(
filePath,
new TimeSeriesMetadataCacheKey(resource.getTsFileID(),
deviceId, measurement),
allSensors,
- context.ignoreNotExistsDevice()
- || resource.getTimeIndexType() ==
ITimeIndex.FILE_TIME_INDEX_TYPE,
+ ignoreNotExists,
isDebug,
context,
- rootMeasurementMetadataIndexNodeOffset);
- exist = (exist || (valueColumn != null));
- valueTimeSeriesMetadataList.add(valueColumn);
- }
- if (!ignoreAllNullRows || exist) {
- alignedTimeSeriesMetadata =
- setModifications(
- resource,
- timeColumn,
- valueTimeSeriesMetadataList,
- alignedPath,
- context,
- globalTimeFilter,
- ignoreAllNullRows);
+ rootMeasurementMetadataIndexNodeOffset));
}
+ alignedTimeSeriesMetadata =
+ buildAlignedTimeSeriesMetadata(
+ resource,
+ alignedPath,
+ timeColumn,
+ valueTimeSeriesMetadataList,
+ context,
+ globalTimeFilter,
+ ignoreAllNullRows);
}
}
return alignedTimeSeriesMetadata;
}
+ static boolean shouldUseDeviceMetadataBatchScan(
+ int selectedColumnCount, int totalFieldColumnCount, boolean debug,
boolean offsetAddressed) {
+ // DEBUG queries keep the per-column cache's diagnostic logs;
offset-addressed external
+ // tsfiles cannot be resolved through the table metadata index; a
single-column device gains
+ // nothing from a batch read (one leaf either way)
+ return !debug
+ && !offsetAddressed
+ && totalFieldColumnCount > 1
Review Comment:
Added the fan-out requirement: the gate now requires `totalFieldColumnCount
> max_degree_of_index_node`, so devices that fit in one leaf group stay on the
per-column path.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/series/AlignedSeriesScanBatchTest.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.read.reader.series;
+
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.AlignedFullPath;
+import
org.apache.iotdb.db.queryengine.execution.operator.source.AlignedSeriesScanUtil;
+import
org.apache.iotdb.db.queryengine.plan.planner.plan.parameter.SeriesScanOptions;
+import org.apache.iotdb.db.queryengine.plan.statement.component.Ordering;
+import org.apache.iotdb.db.storageengine.buffer.TimeSeriesMetadataCache;
+import org.apache.iotdb.db.storageengine.dataregion.read.QueryDataSource;
+import org.apache.iotdb.db.utils.EnvironmentUtils;
+
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+
+/** Verifies the wide-table batch metadata scan (see FileLoaderUtils) reads
data correctly. */
+public class AlignedSeriesScanBatchTest extends AbstractAlignedSeriesScanTest {
+
+ @Test
+ public void testBatchScanReadsAlignedData() throws IOException,
IllegalPathException {
+ AlignedFullPath scanPath =
+ new AlignedFullPath(
+ TEST_DEVICE,
+ Arrays.asList("s1", "s2", "s3"),
Review Comment:
Added a real 700-column batch-vs-per-column comparison (statistics and
chunk-metadata counts, including an absent column) and rewrote
`AlignedSeriesScanBatchTest` to write a 300-column device spanning two leaf
groups and assert the end-to-end scan takes the batch path. A V4/table-model IT
is still missing and noted.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]