ColinLeeo commented on code in PR #19198:
URL: https://github.com/apache/iotdb/pull/19198#discussion_r4559033375


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/FileLoaderUtils.java:
##########
@@ -72,6 +73,12 @@ public class FileLoaderUtils {
   private static final SeriesScanCostMetricSet SERIES_SCAN_COST_METRIC_SET =
       SeriesScanCostMetricSet.getInstance();
 
+  /**
+   * Fraction of a table's field columns that a scan must select for it to use 
the batch
+   * device-metadata read instead of the per-column path.
+   */
+  private static final double DEVICE_METADATA_BATCH_SCAN_RATIO = 0.6;

Review Comment:
   Updated the PR description to match the code. The 0.6 threshold stays 
hard-coded for now (the config was intentionally removed); benchmark numbers on 
a wide device are not included in this change and are noted as follow-up. The 
re-read case from the cache comment is fixed.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/buffer/TimeSeriesMetadataCache.java:
##########
@@ -316,6 +317,216 @@ public TimeseriesMetadata get(
     }
   }
 
+  /**
+   * Load the TimeseriesMetadata of {@code allSensors} of a device, reading 
the device in a single
+   * batch when the per-measurement cache does not already hold all of them. 
This is used by
+   * wide-table scans (a device with many columns) to avoid walking the device 
metadata index once
+   * per column / leaf node.
+   *
+   * <p>The returned map always holds copies of the cached (or freshly read) 
metadata, so the caller
+   * may apply {@code setModified} and {@code setChunkMetadataLoader} without 
corrupting cached
+   * entries — matching {@link #get}.
+   *
+   * @param filePath path of the tsfile
+   * @param tsFileID id of the tsfile
+   * @param device target device
+   * @param allSensors measurements the caller needs, including the time 
column keyed by ""
+   * @param queryContext query context used for metrics and to detect external 
tsfiles
+   * @param ignoreNotExists whether a device missing from a file that claims a 
device-level time
+   *     index should be skipped instead of reported as an inconsistency
+   * @return map from measurementId (including the time column, keyed by "") 
to TimeseriesMetadata;
+   *     entries whose series is empty (count == 0) are mapped to null
+   * @throws IOException if the metadata cannot be read from disk or the 
device is unexpectedly
+   *     absent
+   */
+  public Map<String, TimeseriesMetadata> getDeviceTimeSeriesMetadata(
+      String filePath,
+      TsFileID tsFileID,
+      IDeviceID device,
+      Set<String> allSensors,
+      QueryContext queryContext,
+      boolean ignoreNotExists)
+      throws IOException {
+    long startTime = System.nanoTime();
+    long loadBloomFilterTime = 0;
+    boolean externalTsFile = queryContext.isExternalTsFileScan();
+    LongConsumer timeSeriesMetadataIoSizeRecorder =
+        
queryContext.getQueryStatistics().getLoadTimeSeriesMetadataActualIOSize()::addAndGet;
+    LongConsumer bloomFilterIoSizeRecorder =
+        
queryContext.getQueryStatistics().getLoadBloomFilterActualIOSize()::addAndGet;
+    boolean cacheHit = true;
+
+    try {
+      if (CACHE_ENABLE && !externalTsFile) {
+        Map<String, TimeseriesMetadata> cached =
+            getCachedDeviceTimeSeriesMetadata(tsFileID, device, allSensors);
+        if (cached != null) {
+          return cached;
+        }
+        // serialize concurrent reads of the same device+file (same lock key 
as get) so threads
+        // don't each deserialize the whole device and race on the shared 
reader
+        synchronized (
+            devices.computeIfAbsent(device.toString() + SEPARATOR + filePath, 
WeakReference::new)) {
+          cached = getCachedDeviceTimeSeriesMetadata(tsFileID, device, 
allSensors);
+          if (cached != null) {
+            return cached;
+          }
+          cacheHit = false;
+          long bloomFilterStartTime = System.nanoTime();
+          boolean devicePossiblyPresent =
+              isDevicePossiblyPresent(
+                  filePath, tsFileID, device, queryContext, 
bloomFilterIoSizeRecorder);
+          loadBloomFilterTime = System.nanoTime() - bloomFilterStartTime;
+          if (!devicePossiblyPresent) {
+            return Collections.emptyMap();
+          }
+          TsFileSequenceReader reader =
+              FileReaderManager.getInstance()
+                  .get(filePath, tsFileID, true, 
timeSeriesMetadataIoSizeRecorder, false);
+          return readDeviceTimeSeriesMetadata(
+              reader, tsFileID, device, allSensors, true, ignoreNotExists);
+        }
+      }
+      cacheHit = false;
+      long bloomFilterStartTime = System.nanoTime();
+      boolean devicePossiblyPresent =
+          isDevicePossiblyPresent(
+              filePath, tsFileID, device, queryContext, 
bloomFilterIoSizeRecorder);
+      loadBloomFilterTime = System.nanoTime() - bloomFilterStartTime;
+      if (!devicePossiblyPresent) {
+        return Collections.emptyMap();
+      }
+      TsFileSequenceReader reader =
+          FileReaderManager.getInstance()
+              .get(filePath, tsFileID, true, timeSeriesMetadataIoSizeRecorder, 
externalTsFile);
+      return readDeviceTimeSeriesMetadata(
+          reader, tsFileID, device, allSensors, false, ignoreNotExists);
+    } finally {
+      
queryContext.getQueryStatistics().getLoadBloomFilterTime().getAndAdd(loadBloomFilterTime);
+      if (cacheHit) {
+        queryContext
+            .getQueryStatistics()
+            .getLoadTimeSeriesMetadataFromCacheCount()
+            .incrementAndGet();
+        queryContext
+            .getQueryStatistics()
+            .getLoadTimeSeriesMetadataFromCacheTime()
+            .getAndAdd(System.nanoTime() - startTime - loadBloomFilterTime);
+      } else {
+        queryContext
+            .getQueryStatistics()
+            .getLoadTimeSeriesMetadataFromDiskCount()
+            .incrementAndGet();
+        queryContext
+            .getQueryStatistics()
+            .getLoadTimeSeriesMetadataFromDiskTime()
+            .getAndAdd(System.nanoTime() - startTime - loadBloomFilterTime);
+      }
+    }
+  }
+
+  /**
+   * The time column (keyed by "") is written for every aligned device, so its 
bloom filter entry is
+   * a cheap per-file presence probe. A null filter cannot rule the device out.
+   */
+  private boolean isDevicePossiblyPresent(

Review Comment:
   Extracted `loadBloomFilter`, `bloomFilterMayContain`, 
`collectCachedMetadata` and `recordLoadStatistics`, and reused the statistics 
recorder in both `get()` and the batch path, so the bloom-filter and statistics 
blocks are no longer duplicated.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/buffer/TimeSeriesMetadataCacheTest.java:
##########
@@ -280,4 +288,164 @@ public void testIgnoreNotExistsResultIsNotCached() throws 
Exception {
           .setMayCacheNonExistSeries(mayCacheNonExistSeries);
     }
   }
+
+  @Test
+  public void testGetDeviceTimeSeriesMetadata() throws Exception {
+    TimeSeriesMetadataCache cache = TimeSeriesMetadataCache.getInstance();
+    BloomFilterCache bloomFilterCache = BloomFilterCache.getInstance();
+    FileReaderManager fileReaderManager = FileReaderManager.getInstance();
+    TsFileID tsFileID = new TsFileID();
+    cache.clear();
+    bloomFilterCache.clear();
+    try {
+      String filePath = "target/batch-device.tsfile";
+      IDeviceID deviceID = Factory.DEFAULT_FACTORY.create("root.device");
+      Set<String> allSensors = new HashSet<>(Arrays.asList("", "s1", "s2"));
+
+      Statistics<?> timeStatistics = 
Statistics.getStatsByType(TSDataType.INT64);
+      timeStatistics.setCount(1);
+      timeStatistics.setStartTime(0);
+      timeStatistics.setEndTime(10);
+      TimeseriesMetadata timeColumn =
+          new TimeseriesMetadata(
+              (byte) 0, 0, "", TSDataType.INT64, timeStatistics, new 
PublicBAOS());
+
+      Statistics<?> s1Statistics = Statistics.getStatsByType(TSDataType.INT32);
+      s1Statistics.setCount(1);
+      s1Statistics.setStartTime(0);
+      s1Statistics.setEndTime(10);
+      TimeseriesMetadata s1 =
+          new TimeseriesMetadata(
+              (byte) 0, 0, "s1", TSDataType.INT32, s1Statistics, new 
PublicBAOS());
+
+      // an empty series (count == 0) is mapped to null and cached as the null 
placeholder
+      TimeseriesMetadata emptyS2 =
+          new TimeseriesMetadata(
+              (byte) 0,
+              0,
+              "s2",
+              TSDataType.INT32,
+              Statistics.getStatsByType(TSDataType.INT32),
+              new PublicBAOS());
+
+      List<TimeseriesMetadata> deviceMetadata = new ArrayList<>();
+      deviceMetadata.add(timeColumn);
+      deviceMetadata.add(s1);
+      deviceMetadata.add(emptyS2);
+      // the batch read must carry chunk metadata, otherwise the downstream 
chunk loader NPEs
+      s1.setChunkMetadataList(Collections.singletonList(new ChunkMetadata()));
+
+      TsFileSequenceReader reader = Mockito.mock(TsFileSequenceReader.class);
+      fileReaderManager.setReaderForTest(tsFileID, true, reader);
+      
Mockito.when(reader.getDeviceTimeseriesMetadata(deviceID)).thenReturn(deviceMetadata);
+
+      Map<String, TimeseriesMetadata> result =
+          cache.getDeviceTimeSeriesMetadata(
+              filePath, tsFileID, deviceID, allSensors, new 
QueryContext(false, false), false);
+
+      assertNotNull(result.get(""));
+      assertEquals("", result.get("").getMeasurementId());
+      assertNotNull(result.get("s1"));
+      assertEquals("s1", result.get("s1").getMeasurementId());
+      // chunk metadata must flow through to the result (and the cached copy) 
unchanged
+      assertNotNull(result.get("s1").getChunkMetadataList());
+      assertNull(result.get("s2"));
+
+      // a second lookup of cached measurements must be served without 
touching the reader again
+      Set<String> cachedSensors = new HashSet<>(Arrays.asList("", "s1"));
+      Map<String, TimeseriesMetadata> cachedResult =
+          cache.getDeviceTimeSeriesMetadata(
+              filePath, tsFileID, deviceID, cachedSensors, new 
QueryContext(false, false), false);
+      assertNotNull(cachedResult.get("s1"));
+      Mockito.verify(reader, 
Mockito.times(1)).getDeviceTimeseriesMetadata(deviceID);
+
+      // the batch read also populated the per-measurement cache
+      TimeseriesMetadata cached =
+          cache.get(
+              filePath,
+              new TimeSeriesMetadataCacheKey(tsFileID, deviceID, "s1"),
+              Collections.singleton("s1"),
+              true,
+              false,
+              new QueryContext(false, false));
+      assertNotNull(cached);
+      assertEquals("s1", cached.getMeasurementId());
+    } finally {
+      cache.clear();
+      bloomFilterCache.clear();
+      fileReaderManager.closeFileAndRemoveReader(tsFileID);
+    }
+  }
+
+  @Test
+  public void testAbsentDeviceSkippedWhenBloomFilterExcludesIt() throws 
Exception {
+    TimeSeriesMetadataCache cache = TimeSeriesMetadataCache.getInstance();
+    BloomFilterCache bloomFilterCache = BloomFilterCache.getInstance();
+    FileReaderManager fileReaderManager = FileReaderManager.getInstance();
+    TsFileID tsFileID = new TsFileID();
+    cache.clear();
+    bloomFilterCache.clear();
+    try {
+      String filePath = "target/batch-absent-device-bloom.tsfile";
+      IDeviceID deviceID = Factory.DEFAULT_FACTORY.create("root.absent");
+      Set<String> allSensors = Collections.singleton("");
+
+      TsFileSequenceReader reader = Mockito.mock(TsFileSequenceReader.class);
+      fileReaderManager.setReaderForTest(tsFileID, true, reader);
+      // the bloom filter proves the device is missing, so the device must not 
be read at all
+      Mockito.when(reader.readBloomFilter(Mockito.any(LongConsumer.class)))
+          .thenReturn(BloomFilter.getEmptyBloomFilter(0.05, 1));
+
+      // an absent device yields an empty map, which 
loadAlignedTimeSeriesMetadataBatch then
+      // distinguishes from "present but empty time column"
+      assertTrue(
+          cache
+              .getDeviceTimeSeriesMetadata(
+                  filePath, tsFileID, deviceID, allSensors, new 
QueryContext(false, false), false)
+              .isEmpty());
+      Mockito.verify(reader, 
Mockito.never()).getDeviceTimeseriesMetadata(deviceID);
+    } finally {
+      cache.clear();
+      bloomFilterCache.clear();
+      fileReaderManager.closeFileAndRemoveReader(tsFileID);
+    }
+  }
+
+  @Test
+  public void testAbsentDeviceThrowsWhenBloomFilterCannotExcludeIt() throws 
Exception {
+    TimeSeriesMetadataCache cache = TimeSeriesMetadataCache.getInstance();
+    BloomFilterCache bloomFilterCache = BloomFilterCache.getInstance();
+    FileReaderManager fileReaderManager = FileReaderManager.getInstance();
+    TsFileID tsFileID = new TsFileID();
+    cache.clear();
+    bloomFilterCache.clear();
+    try {
+      String filePath = "target/batch-absent-device-throw.tsfile";
+      IDeviceID deviceID = Factory.DEFAULT_FACTORY.create("root.absent");
+      Set<String> allSensors = Collections.singleton("");
+
+      TsFileSequenceReader reader = Mockito.mock(TsFileSequenceReader.class);
+      fileReaderManager.setReaderForTest(tsFileID, true, reader);
+      Mockito.when(reader.getDeviceTimeseriesMetadata(deviceID))
+          .thenReturn(Collections.emptyList());
+      BloomFilter bloomFilter = BloomFilter.getEmptyBloomFilter(0.05, 1);
+      bloomFilter.add(deviceID.toString() + ".");
+      
Mockito.when(reader.readBloomFilter(Mockito.any(LongConsumer.class))).thenReturn(bloomFilter);
+
+      assertThrows(
+          IOException.class,
+          () ->
+              cache.getDeviceTimeSeriesMetadata(
+                  filePath, tsFileID, deviceID, allSensors, new 
QueryContext(false, false), false));
+
+      assertTrue(
+          cache
+              .getDeviceTimeSeriesMetadata(
+                  filePath, tsFileID, deviceID, allSensors, new 
QueryContext(false, false), true)
+              .isEmpty());
+    } finally {

Review Comment:
   Added `bloomFilterCache.clear()` to the `finally` block.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/FileLoaderUtils.java:
##########
@@ -72,6 +73,12 @@ public class FileLoaderUtils {
   private static final SeriesScanCostMetricSet SERIES_SCAN_COST_METRIC_SET =
       SeriesScanCostMetricSet.getInstance();
 
+  /**
+   * Fraction of a table's field columns that a scan must select for it to use 
the batch
+   * device-metadata read instead of the per-column path.
+   */
+  private static final double DEVICE_METADATA_BATCH_SCAN_RATIO = 0.6;

Review Comment:
   Agreed; a selective reader that visits only the needed leaf groups is the 
right direction. This PR keeps the device-level read and the ratio gate; with 
the new per-measurement bloom probe and partial cache reuse, absent columns no 
longer force a device read, and the selective reader is noted as a follow-up.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to