Hi everyone,

I would like to discuss creating a maintained performance and resource 
reference for Apache TsFile, together with a companion command-line tool 
tentatively named `tsfile-benchmark`.


The community has already accumulated benchmark programs, test results, and 
performance investigations. However, we do not yet have one public, repeatable 
reference that users can consult when they ask practical questions such as:


- How much memory does TsFile use for this workload?
- Does continuous writing remain stable over a long period?
- What changes when the application runs with limited memory?
- What read performance can I expect on my own edge device?


Today, these questions often lead to an ad hoc measurement for an individual 
user. Those results can be useful, but they are difficult for other users to 
find, reproduce, or compare across versions and devices. Similar questions 
therefore require similar measurements again and again.


My proposal is to consolidate and extend the work we already have, rather than 
replace it:


1. **A Performance and Resource Reference.** A versioned, community-maintained 
document published through the TsFile documentation after community review. It 
would describe measured behavior under clearly stated conditions, with the 
workloads, environments, configurations, raw results, and reproduction scripts 
available to readers.
2. **`tsfile-benchmark`.** An installable command-line tool that lets users run 
standard TsFile benchmarks on their own machines or edge devices and save the 
result locally in a readable and machine-readable format.


The two efforts are complementary. The reference would provide curated results 
from controlled environments; `tsfile-benchmark` would let a user characterize 
the device and files they actually use. Where practical, they should share 
workload definitions and result formats, but a local result should be 
interpreted in the context of its own device and configuration rather than as a 
universal score.


For the reference, I suggest that the initial focus be the questions users 
currently find hardest to answer:


- peak and steady-state memory use during writes;
- throughput and memory stability during long-running writes;
- behavior when the application is subject to practical memory constraints;
- the trade-offs among write throughput, flush behavior, and resulting file 
size; and
- read throughput and latency on a small set of reference devices.


For `tsfile-benchmark`, the initial scope should include both read and write 
benchmarks, together with the resource measurements needed to interpret them. 
Read workloads can include scans, time-range queries, point or small-range 
lookups, and representative multi-series queries. Write workloads can cover 
sustained writes, bounded-memory configurations, flush behavior, throughput, 
memory use, and resulting file size. The tool should record enough context to 
make a local run meaningful, including TsFile and tool versions, workload 
parameters, file or dataset identity, operating system, CPU architecture, 
memory, storage, and relevant configuration. A user should be able to keep the 
resulting JSON or text summary with a support question or use it to compare 
versions on the same device.


To keep the first iteration focused, I suggest we start from the C++ core and 
existing benchmark work, rather than try to cover every binding and platform at 
once. Java and Python behavior, additional architectures, and further workloads 
can follow once we agree on a useful initial baseline.


Before proposing a detailed design, I would like to collect feedback on four 
questions:


1. Which deployment scenarios should the first reference and tool prioritize, 
especially for edge deployments?
2. Which questions and metrics matter most to users—for example, peak memory, 
memory after close, long-running write stability, read throughput or latency, 
flush behavior, and file size?
3. Which device classes, storage media, memory conditions, workloads, or 
datasets would be representative enough to include first?
4. Are community members willing to share existing benchmark programs, 
representative data, reference hardware, or methodology review?


Please reply in this thread with your feedback. After collecting the 
discussion, I will summarize the feedback and propose a small initial scope, 
metric definitions, and a reproducible methodology for a follow-up discussion. 
Any project-maintained tool or published reference would proceed through the 
normal community review process.


Thank you for your feedback.


Best regards,  
Colin

Reply via email to