ColinLeeo opened a new issue, #1139:
URL: https://github.com/apache/tsfile/issues/1139
### Background
The C++ table read path is organized around one device at a time:
```
TableQueryExecutor
-> DeviceOrderedTsBlockReader (single-threaded, pulls one device)
-> DeviceTaskIterator (walks the device metadata index)
-> SingleDeviceTsBlockReader (per device: builds one TsBlock)
-> TsFileSeriesScanIterator -> ChunkReader / AlignedChunkReader
```
On export, `cwrapper` converts each block to the Arrow C Data Interface by
hand (`TsBlockToArrowStruct`,
[cpp/src/cwrapper/tsfile_cwrapper.cc:55](https://github.com/apache/tsfile/blob/develop/cpp/src/cwrapper/tsfile_cwrapper.cc#L55),
exposed as `tsfile_result_set_get_next_tsblock_as_arrow`,
[:1029](https://github.com/apache/tsfile/blob/develop/cpp/src/cwrapper/tsfile_cwrapper.cc#L1029)).
That conversion allocates new buffers and copies every column. `TsBlock`
itself never crosses the C ABI: Python and Go both consume the Arrow export.
`common::TsBlock` has three properties that constrain any redesign:
1. The iteration cursor lives inside the block: `RowIterator`/`ColIterator`
advance a mutable offset stored in each `Vector`
([cpp/src/common/tsblock/tsblock.h:285](https://github.com/apache/tsfile/blob/develop/cpp/src/common/tsblock/tsblock.h#L285),
[:350](https://github.com/apache/tsfile/blob/develop/cpp/src/common/tsblock/tsblock.h#L350)).
Two consumers cannot read the same block concurrently, and a scan reuses a
single output block, so the consumer must finish before the next `next()` call.
2. Nulls are dense: a null row occupies no slot in the value buffer, only a
bit in the validity `BitMap`.
3. Variable-length values are stored as a 4-byte length prefix followed by
the bytes, inline in the value buffer
([cpp/src/common/tsblock/vector/variable_length_vector.h:30](https://github.com/apache/tsfile/blob/develop/cpp/src/common/tsblock/vector/variable_length_vector.h#L30)).
Properties 1 and the copy-on-export are exactly what blocks cross-device
parallelism (see the companion cross-device scan issue): a parallel producer
pipeline needs immutable, reference-counted blocks it can hand to a consumer
while continuing to work.
### Proposal
Make the internal read block Arrow-native, with no Arrow C++ dependency and
no C++11 requirement change:
- Introduce a small `common::Block` that owns an `ArrowArray`/`ArrowSchema`
pair (the same layout the existing exporter already produces), held by
`shared_ptr`, with typed column views.
- `TsFileSeriesScanIterator` and the chunk readers produce `Block`; the
table readers produce `Block`; `cwrapper` transfers buffer ownership to the
caller instead of converting.
- Keep `TsBlock` as an assembly shim only where it is still needed
(tree-view QDS readers and unit tests); it becomes an internal compatibility
path, not the primary representation.
- Java is explicitly out of scope for the first phase; its
`TsBlock`/`Column` classes are a public read API and need a separate
coexistence/deprecation plan.
Suggested phasing (from the mailing-list proposal):
- **P0** `common::Block` + block pool + TsBlock assembler + tests. No
behavior change; independently testable.
- **P1** `TsFileSeriesScanIterator` and chunk readers produce `Block`,
including the existing direct-decode path.
- **P2** Table readers, `ResultSet` row accessors, zero-copy C ABI export.
- **P3** (in the companion issue) cross-device scheduler and ordered merge
on top of P0-P2.
- **P4** Tree-view migration, or keep the assembler long term.
Python and Go need no API change: they keep using the same C functions,
which simply stop copying.
### Known costs and risks
- Dense nulls become slot-aligned. Fixed-width columns keep a placeholder
slot for null rows, so a sparse column can take more memory than today. Measure
the null-slot ratio on representative data before committing to this part, or
keep dense storage internally and align only at the export boundary.
- A block exported to pyarrow is owned by the consumer and may be held
indefinitely, so block recycling must be reference-count based.
- Row-oriented accessors (`ResultSet::next()`, `get_*_at()`) must read the
Arrow buffers directly; assembling a `TsBlock` there would add a full block
copy to the Python row path.
### Acceptance criteria
- Export path performs zero column-buffer copies (verified with an
allocation/copy counter or ASan-friendly instrumentation).
- Python and Go APIs unchanged; existing Arrow round-trip and cross-language
tests pass.
- Wide-table benchmark: narrow 5-column scan not slower than today; document
the memory delta from slot-aligned nulls.
- C++ ASan/LSan clean; no file-format change.
### Open questions
1. Does anyone depend on the current `TsBlock` C++ API directly, outside
this repository? That determines whether the assembler must stay a stable
interface.
2. Is the memory cost of slot-aligned nulls acceptable for sparse
measurements?
3. For Java, is there interest in a parallel/Arrow-native read design now,
or is that a separate discussion?
Related: cross-device parallel scan issue; #1138 (object-storage read
latency) benefits directly once blocks are immutable and export stops copying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]