ColinLeeo opened a new issue, #1080:
URL: https://github.com/apache/tsfile/issues/1080

   ## Summary
   
   Applications that load training data issue one time-range query per window. 
On that access pattern the public table query API costs a **fixed ~0.30 ms per 
call**, almost independent of the number of rows returned: 64 windows (2048 
points each) take 20.6–23.1 ms, while a single prepared-series query for the 
same window takes 0.046 ms.
   
   `prepare_series` + `query_prepared` already avoid most of the per-call cost, 
but the prepared path is (a) only reachable through locators produced by the 
Python dataset index, and (b) has several defects that limit how much it helps. 
This issue records what we measured and the concrete fixes that would make 
prepared series the normal fast path.
   
   ## Workload
   
   - 323 regular series × 105,120 points, 5-minute interval, one table, one 
string TAG (`series_id`).
   - Values: float32 **PLAIN**; time: **TS_2DIFF**; no compression; page size 
10,000 points.
   - One window = 2048 rows; 64 windows per batch; warm page cache; one CPU; 
`threads=1`; medians over repeated runs.
   - AMD EPYC 9J14, Python 3.11.18, tsfile 2.5.0 wheel plus locally built 
`develop` (identical CMake flags).
   
   ## Measurements (ms per single-window call)
   
   | API path | ms/window |
   |---|---:|
   | `query_table`, `batch_size = window length` | 0.32–0.36 |
   | `query_prepared`, value column, time range, `limit = length` | **0.046** |
   | `query_prepared`, value column, time range, no `limit` | 0.079 |
   | `query_prepared`, `valid` column only | 0.076 |
   | two single-column `query_prepared` calls (value + valid) | 0.121 |
   | `query_prepared_multi`, two columns, time range | 0.26–0.27 |
   | `query_prepared_multi`, two columns, `offset/limit` | **1.79** |
   | prepared-handle acquire | 0.004 |
   
   Per 64-window batch: 20.6–23.1 ms with `query_table`, ~4–5 ms with prepared 
queries.
   
   The cost is not proportional to rows: a 0-row query still costs 0.325 ms, a 
10,000-row query 0.55 ms, and a full 105,120-row query 0.92 ms (8.8 ns/row).
   
   ## Where the time goes per `query_table` call
   
   `query_table()` itself is lazy (~0.008 ms). The first `read_arrow_batch()` 
performs all of the work:
   
   1. The table-level device index leaf node is read and deserialized to apply 
the tag filter.
   2. `SingleDeviceTsBlockReader::init_internal()` calls 
`TsFileIOReader::get_timeseries_indexes()`, which re-reads and deserializes the 
device metadata node via `load_device_index_entry()` and does **not** consult 
`device_node_cache_`, even though `alloc_multi_ssi()` later uses the cached 
node. Measured: repeated queries on the same series (cache hot) cost 0.316 ms 
vs 0.440 ms when cycling 323 series — the cache only saves ~28%.
   3. A new `TsFileSeriesScanIterator` + `ChunkReader` is constructed and 
destroyed per query.
   4. `ChunkReader::get_next_page()` walks page headers from the start of the 
chunk, reading `DEFAULT_READ_SIZE = 4096` bytes per header; a 2048-point window 
triggers ~14 such reads (~57 KiB) even when the filter covers only the first 
page. There is no early stop once the page range is past the filter.
   5. The intersecting page is decoded in full (10,000 points for a 2048-point 
window).
   6. The `TsBlock` is allocated per query with `max_row_count = batch_size` 
(e.g. 65,536 → ~0.8 MB); this alone costs 0.079 vs 0.040 ms.
   
   The profile of the per-query work is dominated by allocator traffic and 
metadata construction; actual decoding (`multi_DECODE_TV_BATCH` + TS_2DIFF + 
time filter) is only ~6.5%.
   
   ## Prepared path defects
   
   - `PreparedSeriesTsBlockReader` hard-codes `PREPARED_BATCH_ROWS = 65536`, so 
without an explicit `limit` every prepared query preallocates a 65,536-row 
block. Passing `limit` halves the call cost (0.079 → 0.046 ms).
   - Every prepared call still constructs a new SSI/`ChunkReader`; 
`PreparedSeries` only caches the resolved index.
   - `query_prepared_multi(..., offset, limit)` degrades to 1.79 ms/call, ~40× 
a single-column prepared call and ~7× the two-column time-range variant.
   - Two single-column prepared calls (0.121 ms) are ~2.2× faster than one 
`query_prepared_multi` (0.27 ms) for two columns.
   - The prepared locator is only produced by the Python dataset index 
(`TsFileDataFrame(use_index=True)`), so users not using that layer cannot call 
`prepare_series` by (table, tag values, field).
   
   ## Proposed changes
   
   1. Make `get_timeseries_indexes()` reuse the parsed device node from 
`device_node_cache_`, or drop the redundant call in `init_internal` since 
`alloc_multi_ssi()` already resolves the same node.
   2. Stop scanning page headers once the filter range is exhausted, and/or use 
a page index to seek.
   3. Push the row offset down to page granularity so a 2048-point window does 
not decode a 10,000-point page.
   4. In the prepared path: honor the requested row count in 
`PREPARED_BATCH_ROWS`, reuse the SSI/`ChunkReader` across calls on the same 
prepared series, and expose a public `prepare(table, tags, field)` entry point.
   5. Fix `query_prepared_multi` with `offset/limit` and investigate why two 
columns are slower than two single-column calls.
   
   ## Notes
   
   - PR #1072 (wide-table reads) does not change this workload: with the same 
build configuration it executes the same number of instructions as `develop` 
(8.824G vs 8.825G for 1500 queries), because this dataset uses PLAIN values (no 
RLE) and one measurement per device (no per-measurement index fan-out).
   - Separately, we measured current `develop` at ~8.8G instructions vs 7.2G 
for 2.5.0 on the same 1500-query loop (both locally built with identical CMake 
flags), i.e. ~21% more instructions per query. We have not bisected this yet; 
it may deserve its own issue.
   - The benchmark script and the dataset generator parameters can be shared if 
useful.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to