ColinLeeo opened a new issue, #1140:
URL: https://github.com/apache/tsfile/issues/1140
### Background
The C++ table read path is organized around one device at a time:
```
TableQueryExecutor
-> DeviceOrderedTsBlockReader (single-threaded, pulls one device)
-> DeviceTaskIterator (walks the device metadata index)
-> SingleDeviceTsBlockReader (per device: builds one TsBlock)
-> TsFileSeriesScanIterator -> ChunkReader / AlignedChunkReader
```
The only concurrency in the read path today is aligned multi-column page
decode, submitted to the process-wide thread pool and gated on `num_cols > 1 &&
parallel_read_enabled_`
([cpp/src/reader/tsfile_series_scan_iterator.cc:473](https://github.com/apache/tsfile/blob/develop/cpp/src/reader/tsfile_series_scan_iterator.cc#L473)).
A narrow query over many devices runs entirely on one thread: index I/O and
decode are serialized per device.
On our internal wide-table benchmark (about 4,500 devices, monthly files), a
narrow 5-column, one-month scan takes roughly 1.4 s on a single thread after
the index/decode work already merged. What is left is serialized index I/O and
single-threaded decode, which is what this issue targets.
Additional constraints verified in the code:
- `DeviceOrderedTsBlockReader` carries
`remaining_offset_`/`remaining_limit_` and decrements them sequentially as
devices complete
([cpp/src/reader/block/device_ordered_tsblock_reader.h:43](https://github.com/apache/tsfile/blob/develop/cpp/src/reader/block/device_ordered_tsblock_reader.h#L43)).
- TIME-ordered output is currently rejected with `E_UNSUPPORTED_ORDER`
([cpp/src/reader/table_query_executor.cc:103](https://github.com/apache/tsfile/blob/develop/cpp/src/reader/table_query_executor.cc#L103)).
- The current block is mutable and cursor-bearing, so a block cannot safely
be handed from a producer thread to the consumer while decode continues; that
is why this work depends on the Arrow-native block issue.
### Proposal
Stop treating "one device" as the unit of work.
- A task becomes a `(file, device range, time range)` unit, sized by
estimated bytes rather than by device boundaries.
- A bounded pool of workers produces blocks into a merge buffer; the
existing pull interface (`TsBlockReader::has_next/next`, `ResultSet`) stays
unchanged, so no consumer has to be rewritten.
- Device ordering is preserved by sequence number. TIME ordering becomes a
lazy k-way merge over per-device block streams rather than a physical
row-by-row merge.
- OFFSET/LIMIT need cancellation semantics: an atomic budget plus
cancellation of not-yet-started tasks, replacing the current sequential
`remaining_offset_`/`remaining_limit_` accounting.
- Cross-device parallelism should be opt-in via configuration (like
`parallel_read_enabled_`) initially.
This also addresses the latency-overlap problem for remote storage (#1138):
prefetching the next chunk(s) on the worker pool hides object-storage round
trips behind decode.
### Dependencies
Blocked on the Arrow-native block work (P0-P2): immutable, reference-counted
blocks are required to hand data between producer and consumer safely. The
scheduler (P3 in the mailing-list phasing) can start designing in parallel but
cannot land first.
### Acceptance criteria
- Multi-device scan scales with cores: e.g. >= 2x throughput on a 4-core
machine for the wide-table benchmark above, reported with thread count and
hardware.
- Device-order output is bit-identical to the sequential path; OFFSET/LIMIT
results identical for all combinations of offset/limit, including `limit == 0`
and limits that cut a device mid-block.
- LIMIT short-circuits: a query with `LIMIT 10` over 4,500 devices must not
scan every device.
- Thread-safety under TSan; no regression for single-device queries or when
the feature is disabled.
- Local-file and remote (seekable) sources both exercise the new path.
### Open questions
1. Is TIME-ordered output needed by downstream consumers, or is device order
sufficient? Supporting it changes the merge design.
2. Should cross-device parallelism be opt-in initially, or on by default
once stable?
3. For Java, is there interest in a parallel scan design now, or should that
be a separate discussion later?
Related: Arrow-native block issue (dependency); #1138 object-storage read
latency; #1080 point-query metadata rebuild; #1114/#1115 SPRINTZ decode cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]