Hi all,

I would like to synchronize with the Apache TsFile community on the current
DuckDB extension work and an external discussion that I have opened with
the DuckDB community.

The discussion is available here:

https://github.com/duckdb/duckdb/discussions/24932

The proposal explores a roadmap for eventually making TsFile an externally
maintained DuckDB Core extension. This is not a request for immediate
promotion or integration. The purpose is to confirm whether DuckDB
maintainers consider TsFile suitable for this model and to understand the
technical, governance, adoption, and maintenance requirements.

Motivation

DuckDB provides first-class support for several general-purpose analytical
and lakehouse formats, including Parquet, JSON, Iceberg, Delta, Lance, and
DuckLake. However, its Core extension ecosystem does not currently include
an on-disk file format designed specifically for time-series data.

TsFile could fill this gap. Its organization and indexes around time,
devices, measurements, tags, and fields, together with time-series-specific
encoding and compression, allow existing TsFile data to be queried without
first converting it to Parquet or loading it into a separate database
system.

Direct integration would also allow users to join TsFile measurements with
Parquet, CSV, relational, spatial, and other data sources through DuckDB
SQL.

Current status

The TsFile extension is already available through DuckDB Community
Extensions:

https://duckdb.org/community_extensions/extensions/tsfile

The current version supports table-model reading, projection pushdown, time
predicate pushdown, primary TsFile data types, NULL handling, and native
builds for Linux, macOS, and Windows.

Work is also underway on:

   -

   COPY ... TO ... (FORMAT tsfile) support;
   -

   TAG predicate pushdown;
   -

   round-trip read/write tests;
   -

   expanded SQL documentation.

The current implementation is still at an early stage. Tree-model support,
multi-file scans, automatic discovery, replacement scans, remote filesystem
integration, parallelism, broader predicate pushdown, and additional
compatibility and robustness testing remain to be completed.

Proposed Apache repository

As previously discussed in the TsFile community, we plan to establish a
dedicated repository:

https://github.com/apache/tsfile-duckdb

The extension was initially developed and submitted to DuckDB Community
Extensions from a personal repository in order to validate the integration.
The proposed Apache repository would become the canonical location for its
long-term development and maintenance.

This would ensure that the extension is:

   -

   maintained by the Apache TsFile community rather than an individual
   account;
   -

   developed through public Apache review and contribution processes;
   -

   supported by multiple maintainers;
   -

   built only from upstream Apache TsFile dependencies;
   -

   continuously tested against DuckDB releases and development versions;
   -

   governed independently from any single company.

Under the proposed DuckDB Core model, the source would remain in the Apache
repository and the Apache TsFile community would remain responsible for
implementation, compatibility updates, issue handling, and user support.
DuckDB's infrastructure would only build, sign, and distribute revisions
accepted into its Core extension repository.

Feedback requested

I would appreciate feedback from the TsFile community on the following
points:

   1.

   Do we agree that apache/tsfile-duckdb should become the canonical
   repository for this integration?
   2.

   Who would be interested in contributing to or maintaining the repository?
   3.

   What functionality should we consider mandatory before requesting DuckDB
   Core integration?
   4.

   How should we prioritize table-model writing, tree-model support,
   multi-file scans, remote storage, parallel scanning, replacement scans, and
   predicate pushdown?
   5.

   Are there any concerns about the proposed ownership, dependency,
   release, or compatibility model?
   6.

   Does anyone have additional use cases, production requirements, or
   adoption evidence that should be included in the DuckDB discussion?

Comments related to DuckDB's integration requirements are also welcome
directly in the GitHub Discussion.

Best regards,
Yuan Tian

Reply via email to