Hello community,

I would like to discuss a possible approach to supporting Go in Apache TsFile.


The community previously discussed Go support in [Issue 
#73](https://github.com/apache/tsfile/issues/73). There was once a pure-Go 
implementation,
but it is no longer maintained and is incompatible with the current TsFile 
format. As the C++ implementation and C wrapper have become more mature,
we can now consider providing Go bindings without maintaining another 
independent implementation of the TsFile format:


```text
Go API → cgo → C API → C++ TsFile Core
```


This approach would allow Go applications to read and write current TsFile 
files while sharing the format compatibility, bug fixes, and performance 
improvements of the C++ module.


The initial Go binding design could have the following characteristics:


- expose idiomatic Go Reader, Writer, Schema, Tablet, and ResultSet APIs;
- keep cgo and `unsafe` code inside an internal native package;
- use opaque handles for C/C++ objects without exposing their internal layouts 
to Go users;
- provide explicit `Close` methods with clearly defined memory ownership across 
the language boundary;
- map C API error codes to Go errors;
- provide both a custom Tablet API and an Apache Arrow API;
- support batch data exchange through the Arrow C Data Interface;
- reuse the existing `_tsfile_*` interfaces and extend the low-level 
capabilities needed by multiple language bindings when necessary.


For performance, the primary read and write paths should use batch-oriented, 
columnar interfaces. Crossing the cgo boundary once per cell could introduce 
significant overhead, so Tablet, Arrow RecordBatch, or another batch 
representation should be used as the main data exchange unit, requiring only 
one or a small number of cgo calls per batch.


Row-oriented APIs could still be provided for convenience, but they could 
buffer data in Go and write it as a Tablet instead of calling the C API once 
for every value.


Batch data exchange can currently use the Arrow C Data Interface path. If the 
layout and lifetime of the existing TsBlock representation can be exposed to Go 
safely and reliably, a TsBlock batch path can also be used. Both paths have 
costs: the Arrow path requires data conversion and copying between TsBlock, 
Arrow, and Tablet, while directly exposing TsBlock increases the complexity of 
cross-language memory ownership and lifetime management.


The initial scope could include:


- Go module and package structure;
- table-model Reader and Writer;
- Schema and Tablet APIs;
- custom Tablet batch reads and writes;
- Apache Arrow batch reads and writes;
- common scalar types, null values, strings, and binary values;
- deterministic resource cleanup;
- compatibility tests against files produced by the Java and C++ 
implementations;
- benchmarks comparing native C++, batched Go bindings, and row-oriented Go 
access;
- support for major platforms, including Linux, macOS, Windows, amd64, and 
arm64.


Tree-model support could be added incrementally after the first-stage design 
and implementation become stable.


I would appreciate the community's feedback on the following questions:


1. Which existing `_tsfile_*` interfaces can be used directly as general 
binding APIs, and which ones need to be extended or adjusted?
2. Which compatibility and performance metrics should be used as acceptance 
criteria for the Go bindings?


More background and the initial proposal are available in [Issue 
#924](https://github.com/apache/tsfile/issues/924).


Feedback on the overall direction, initial scope, and low-level API design 
would be very welcome.


Thank you!

Reply via email to