Hi all,

I would like to share recent work on integrating the TsFile C++ library with
LabVIEW.


The implementation is available here:


https://github.com/Young-Leo/tsfile/tree/ly/labview-bulk-io-perf


The goal is to provide a stable C ABI that can be called from LabVIEW through
the Call Library Function Node (CLFN), without exposing the C++ ABI or C++
object lifetimes directly.


The current implementation includes:


- Schema and writer lifecycle APIs.
- Bulk INT32, FLOAT, and DOUBLE writes.
- Row-major/interleaved input support:


  data[row * column_count + column]


- Column-major/channel-contiguous input support:


  data[column * row_count + row]


- Reusable Tablets for continuous acquisition.
- Bulk numeric read APIs.
- Explicit flush and synchronous close.
- A two-phase asynchronous close API for file rotation.
- CMake integration, tests, and LabVIEW usage documentation.


The row-major path fills Tablet column buffers directly in one fused loop.
The column-major path copies each channel as a contiguous array. Both paths
avoid the previous full-size transpose buffer.


The intended LabVIEW streaming flow is:


  create schema
    -> add columns
    -> open writer
    -> repeatedly write acquisition blocks
    -> optional flush
    -> close or asynchronous close


I also compared the write performance with NI TDMS using the same
nine-channel DOUBLE input:


- 60,000 rows per block
- 9 channels
- 25 blocks
- 13,500,000 channel values in total
- Uncompressed value columns
- Full-path timing from file creation until close returned


The median full-path throughput was:


- NI TDMS:             14.78 M values/s
- TsFile row-major:    13.72 M values/s
- TsFile column-major: 13.50 M values/s


The target acquisition workload is 9 channels at 60 kHz, or approximately
0.54 M values/s. The measured TsFile throughput is about 25 times the target
workload.


During optimization, TsFile full-path throughput increased from
8.58 M values/s to 13.77 M values/s, while median flush time decreased from
approximately 0.622 seconds to 0.066 seconds.


The main optimizations were:


- Direct row-major Tablet filling.
- Column-major APIs for channel-contiguous input.
- Reusable Tablet buffers.
- Bulk bitmap handling.
- 64 KiB Chunk pages.
- Direct writing of encoded Chunk pages, avoiding an additional file-sized
  intermediate copy.


The branch has been rebased onto the latest develop branch. Feedback and
suggestions on the LabVIEW integration and API design are welcome.


Best regards,
Leo

Reply via email to