ColinLeeo opened a new issue, #1081:
URL: https://github.com/apache/tsfile/issues/1081

   ## Summary
   
   The Python dataset layer (`TsFileDataFrame` / `DatasetRuntime`) caches 
resolved metadata: a bounded `PreparedSeriesCache` (default 4096 entries), a 
`ReaderSessionPool`, and descriptor caches. Plain users of the low-level 
`TsFileReader.query_table()` get none of it: every call resolves the device 
through the tag filter, re-reads and deserializes the device metadata node, and 
builds a new SSI/`ChunkReader`.
   
   For repeated point queries this per-call metadata rebuild dominates the 
runtime, and the same prepared-series machinery that the dataset layer already 
uses would remove it.
   
   ## Measurements
   
   323 regular series × 105,120 points (5-minute interval, float32 PLAIN, 
TS_2DIFF time, uncompressed); 2048-point windows; warm page cache; one CPU; 
`threads=1`; medians.
   
   | path | ms/call |
   |---|---:|
   | `TsFileReader.query_table(...)`, same series repeated | 0.316 |
   | `TsFileReader.query_table(...)`, cycling 323 series | 0.440 |
   | `query_prepared` on an already-prepared series (`limit` set) | 0.046 |
   | `query_prepared` without `limit` | 0.079 |
   | `TsFileDataFrame.loc[...]` (dataset path, uses the prepared cache) | 0.197 
|
   
   The cache already exists and is effective; it is only reachable through the 
dataset layer.
   
   ## Current behavior
   
   - `query_table()` → `tsfile_reader_query_table_batch_c` → 
`TsFileReader::query()` → `TableQueryExecutor::query()`. The executor itself is 
reused when `batch_size` is unchanged, but each call rebuilds the device task 
iterator and the result set.
   - `SingleDeviceTsBlockReader::init_internal()` calls 
`get_timeseries_indexes()`, whose `load_device_index_entry()` path does not 
consult the C++ `device_node_cache_` (reported separately in #1080).
   - The dataset-layer caches (`PreparedSeriesCache`, `ReaderSessionPool`, 
descriptor caches) are not used by `TsFileReader`.
   - The only public way to reuse a resolved series today is 
`TsFileReader.prepare_series(locator)` + `query_prepared(...)`, but the 
11-field locator is produced by `MappedDatasetIndex`, i.e. it requires 
`TsFileDataFrame(use_index=True)` and a `.tsidx` sidecar. The plain reader API 
cannot produce one.
   
   ## Proposal
   
   1. Add a logical-coordinate prepared-query entry point on `TsFileReader`, 
e.g.:
   
      ```python
      q = reader.prepare_table_query("series", tag_eq("series_id", "123"), 
["value"])
      for start, end, n in windows:
          with q.query(start_time=start, end_time=end, limit=n) as rs:
              ...
      ```
   
      It resolves the device and the timeseries index once and holds the 
resulting prepared series; subsequent queries skip tag lookup and index 
deserialization.
   
   2. Alternatively, keep `query_table()` unchanged but add an opt-in, bounded 
per-reader LRU keyed by (table, resolved tag values, column set) that reuses 
the resolved metadata. It must be documented as retaining memory per distinct 
series and must invalidate on file generation/fingerprint change.
   
   3. Apply the same mechanism to `query_table_by_row()`.
   
   4. Consider exposing the dataset layer's existing `PreparedSeriesCache` to 
plain readers instead of a second cache: it is already bounded, single-flight, 
and tracks active leases.
   
   ## Acceptance criteria
   
   - Repeated `query_table()`-equivalent calls on the same (table, tags, 
columns) must not re-read the device index node.
   - Results identical to today's `query_table()`; no change to query semantics 
or ordering.
   - Cache size configurable/disableable; memory accounted for in the 
results/RSS.
   
   ## Notes
   
   - Related: #1080 (per-call metadata rebuild in the C++ point-query path, and 
prepared-path defects such as `PREPARED_BATCH_ROWS = 65536`).
   - Measurements come from a LOTSA-derived workload; the benchmark script can 
be shared.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to