Hi Chris,


Thanks for the suggestion. I also agree that, whenever possible, integrations 
should live in the target project, where they are more likely to receive 
attention and feedback from its maintainers.


I still think it is worth working toward having the DuckDB project accept and 
maintain TsFile support directly, and I plan to ask the DuckDB community about 
this by email. However, given DuckDB's current division of responsibilities, 
this seems unlikely in the near term. The main data formats currently 
maintained directly by the DuckDB team include built-in CSV support and the 
Parquet, JSON, Avro, and Excel Core Extensions. The team also maintains 
table-format integrations such as Delta, Iceberg, and DuckLake. For Community 
Extensions, the source code generally remains maintained by external 
contributors, while DuckDB provides centralized building, testing, signing, and 
distribution.


DuckDB Core Extensions list: 
<https://duckdb.org/docs/current/core_extensions/overview>


The TsFile extension has now been accepted into DuckDB Community Extensions:


- Extension source: <https://github.com/ColinLeeo/tsfile-duckdb>
- Merged Community Extensions PR: 
<https://github.com/duckdb/community-extensions/pull/2435>


There is also a specific CI limitation. A DuckDB Community Extension descriptor 
can specify only a GitHub repository and a Git reference, not a subdirectory 
within the repository. After checking out the repository, the CI expects the 
extension to be buildable from the repository root using the DuckDB Extension 
Template structure. It therefore cannot currently point directly to a `duckdb/` 
subdirectory in a larger integrations repository.


I agree that we can follow an approach similar to `plc4x-extras`. As TsFile 
gains integrations with more third-party systems, we can use a single 
integrations repository to maintain the related code instead of creating and 
releasing a separate repository for every target project.


I also looked further into Apache Druid. Its model is different from DuckDB's: 
community extension code can be contributed directly to the 
`extensions-contrib` directory in the `apache/druid` repository and maintained 
by community contributors. Druid also supports external extensions distributed 
as Maven/JAR plugins. I have already sent a discussion email about TsFile 
integration to the Druid developer community and am currently waiting for their 
feedback.


Based on these differences, I currently prefer the following principles:


1. Create a shared Apache TsFile integrations repository. Its name can be 
discussed later, with options such as `tsfile-extras` or `tsfile-integrations`. 
It would host integration code that target projects will not, or do not yet, 
accept. Individual integrations could be organized as separate directories or 
submodules as needed.
2. When a target project is willing to accept the integration code, contribute 
it to that project's repository whenever possible.
3. When a target project requires extension code to remain externally 
maintained, the Apache TsFile community can continue to maintain it.


This would keep integrations close to their target communities while reducing 
the maintenance and release overhead for Apache TsFile itself.


These are my current thoughts. Further discussion and suggestions are very 
welcome.


Best regards,


Colin Lee













At 2026-08-07 16:16:23, "Christofer Dutz" <[email protected]> wrote:
>Hi all,
>
>I would instead recommend to actually add it to the duckdb repo, if that’s 
>possible.
>The reason is, that usually this way the plugin receives more attention by the 
>folks that the integration is for. Also do the people very knowledgeable about 
>the target platform usually come back with interesting feedback, which usually 
>doesn’t happen if it’s in a completely disconnected repo.
>
>With PLC4X we also try to develop plugins in the PLC4X project, but move them 
>over to the target platforms as soon as possible.
>
>Also with really separate repos for every target, the beaurocratic effort for 
>the TsFile project increases.
>
>That’s why for example we have all integration modules in PLC4X in a 
>plc4x-extras repo so we don’t have to release 100 individual reos (Each one 
>requiring the full staging, reviewing, voting, releasing process)
>
>Just my opinion …
>
>Chris
>
>
>Von: ColinLee <[email protected]>
>Datum: Freitag, 7. August 2026 um 05:48
>An: [email protected] <[email protected]>
>Betreff: Re:Re: [DISCUSS] Integrate Apache TsFile with DuckDB
>
>Hi Yuan,
>
>
>Thanks for the suggestion. I agree with creating the repository as:
>
>
>https://github.com/apache/tsfile-duckdb
>
>
>The name clearly reflects the integration between Apache TsFile and DuckDB.
>I can prepare the initial code and documentation for the new repository.
>
>
>Best regards,
>
>
>Colin Lee
>
>
>
>
>
>
>
>At 2026-08-07 11:28:41, "Yuan Tian" <[email protected]> wrote:
>>Hi Colin,
>>
>>I think we can add a new repo under apache github org:
>>https://github.com/apache, and named it as
>>https://github.com/apache/tsfile-duckdb
>>
>>What do you think?
>>
>>
>>Best regards,
>>--------------------
>>Yuan Tian
>>
>>On Fri, Aug 7, 2026 at 11:25 AM ColinLee <[email protected]> wrote:
>>
>>> Hi all,
>>>
>>>
>>> I have created an experimental DuckDB extension to query Apache TsFile
>>> table-model files directly:
>>>
>>>
>>> https://github.com/ColinLeeo/tsfile-duckdb
>>>
>>>
>>> The extension exposes a table function such as:
>>>
>>>
>>> SELECT time, timeseries_id, value
>>> FROM read_tsfile('/data/measurements.tsfile', 'greend_1')
>>> WHERE time BETWEEN 100000 AND 200000;
>>>
>>>
>>> The current implementation includes:
>>>
>>>
>>> 1. Integration of the TsFile C++ reader with DuckDB's table function API.
>>> 2. Static builds of TsFile and its compression dependencies.
>>> 3. TsFile-to-DuckDB type mapping and streaming result batches.
>>> 4. Projection pushdown and time predicate pushdown.
>>> 5. Build verification on Linux, macOS, and Windows through DuckDB Community
>>>    Extensions CI.
>>>
>>>
>>> The DuckDB Community Extensions PR is available here:
>>>
>>>
>>> https://github.com/duckdb/community-extensions/pull/2435
>>>
>>>
>>> Based on this initial implementation and validation, I would like to
>>> discuss
>>> bringing this work into a repository under the Apache TsFile organization.
>>> The current repository is still hosted under my personal GitHub account:
>>>
>>>
>>> https://github.com/ColinLeeo/tsfile-duckdb
>>>
>>>
>>> If the community agrees with this direction, we can move the DuckDB
>>> extension
>>> to an Apache-owned repository in a future step.
>>>
>>>
>>> The current prototype supports one local TsFile and one table per query.
>>> Directory scans, automatic table discovery, multi-file scans, and
>>> DuckDB-WASM support are not included yet.
>>>
>>>
>>> Reviews and suggestions are welcome.
>>>
>>>
>>> Best regards,
>>>
>>>
>>> Colin Lee

Reply via email to