Sounds good :) Tsfile-extras would match the principle of iotdb which also has an iotdb-extras repo for exactly this sort of thing. I guess „integrations“ would probably limit what should go in there … in plc4x and I think in iotdb we chose extras as also a lot of examples went in there.
And sure .. let’s start there and try and move it to the duckdb repo in parallel … I assume an extras/integrations repo won’t loose it’s usefulness. Chris Von: ColinLee <[email protected]> Datum: Freitag, 14. August 2026 um 10:37 An: [email protected] <[email protected]> Betreff: Re:AW: Re:Re: [DISCUSS] Integrate Apache TsFile with DuckDB Hi Chris, Thanks for the suggestion. I also agree that, whenever possible, integrations should live in the target project, where they are more likely to receive attention and feedback from its maintainers. I still think it is worth working toward having the DuckDB project accept and maintain TsFile support directly, and I plan to ask the DuckDB community about this by email. However, given DuckDB's current division of responsibilities, this seems unlikely in the near term. The main data formats currently maintained directly by the DuckDB team include built-in CSV support and the Parquet, JSON, Avro, and Excel Core Extensions. The team also maintains table-format integrations such as Delta, Iceberg, and DuckLake. For Community Extensions, the source code generally remains maintained by external contributors, while DuckDB provides centralized building, testing, signing, and distribution. DuckDB Core Extensions list: <https://duckdb.org/docs/current/core_extensions/overview> The TsFile extension has now been accepted into DuckDB Community Extensions: - Extension source: <https://github.com/ColinLeeo/tsfile-duckdb> - Merged Community Extensions PR: <https://github.com/duckdb/community-extensions/pull/2435> There is also a specific CI limitation. A DuckDB Community Extension descriptor can specify only a GitHub repository and a Git reference, not a subdirectory within the repository. After checking out the repository, the CI expects the extension to be buildable from the repository root using the DuckDB Extension Template structure. It therefore cannot currently point directly to a `duckdb/` subdirectory in a larger integrations repository. I agree that we can follow an approach similar to `plc4x-extras`. As TsFile gains integrations with more third-party systems, we can use a single integrations repository to maintain the related code instead of creating and releasing a separate repository for every target project. I also looked further into Apache Druid. Its model is different from DuckDB's: community extension code can be contributed directly to the `extensions-contrib` directory in the `apache/druid` repository and maintained by community contributors. Druid also supports external extensions distributed as Maven/JAR plugins. I have already sent a discussion email about TsFile integration to the Druid developer community and am currently waiting for their feedback. Based on these differences, I currently prefer the following principles: 1. Create a shared Apache TsFile integrations repository. Its name can be discussed later, with options such as `tsfile-extras` or `tsfile-integrations`. It would host integration code that target projects will not, or do not yet, accept. Individual integrations could be organized as separate directories or submodules as needed. 2. When a target project is willing to accept the integration code, contribute it to that project's repository whenever possible. 3. When a target project requires extension code to remain externally maintained, the Apache TsFile community can continue to maintain it. This would keep integrations close to their target communities while reducing the maintenance and release overhead for Apache TsFile itself. These are my current thoughts. Further discussion and suggestions are very welcome. Best regards, Colin Lee At 2026-08-07 16:16:23, "Christofer Dutz" <[email protected]> wrote: >Hi all, > >I would instead recommend to actually add it to the duckdb repo, if that’s >possible. >The reason is, that usually this way the plugin receives more attention by the >folks that the integration is for. Also do the people very knowledgeable about >the target platform usually come back with interesting feedback, which usually >doesn’t happen if it’s in a completely disconnected repo. > >With PLC4X we also try to develop plugins in the PLC4X project, but move them >over to the target platforms as soon as possible. > >Also with really separate repos for every target, the beaurocratic effort for >the TsFile project increases. > >That’s why for example we have all integration modules in PLC4X in a >plc4x-extras repo so we don’t have to release 100 individual reos (Each one >requiring the full staging, reviewing, voting, releasing process) > >Just my opinion … > >Chris > > >Von: ColinLee <[email protected]> >Datum: Freitag, 7. August 2026 um 05:48 >An: [email protected] <[email protected]> >Betreff: Re:Re: [DISCUSS] Integrate Apache TsFile with DuckDB > >Hi Yuan, > > >Thanks for the suggestion. I agree with creating the repository as: > > >https://github.com/apache/tsfile-duckdb > > >The name clearly reflects the integration between Apache TsFile and DuckDB. >I can prepare the initial code and documentation for the new repository. > > >Best regards, > > >Colin Lee > > > > > > > >At 2026-08-07 11:28:41, "Yuan Tian" <[email protected]> wrote: >>Hi Colin, >> >>I think we can add a new repo under apache github org: >>https://github.com/apache, and named it as >>https://github.com/apache/tsfile-duckdb >> >>What do you think? >> >> >>Best regards, >>-------------------- >>Yuan Tian >> >>On Fri, Aug 7, 2026 at 11:25 AM ColinLee <[email protected]> wrote: >> >>> Hi all, >>> >>> >>> I have created an experimental DuckDB extension to query Apache TsFile >>> table-model files directly: >>> >>> >>> https://github.com/ColinLeeo/tsfile-duckdb >>> >>> >>> The extension exposes a table function such as: >>> >>> >>> SELECT time, timeseries_id, value >>> FROM read_tsfile('/data/measurements.tsfile', 'greend_1') >>> WHERE time BETWEEN 100000 AND 200000; >>> >>> >>> The current implementation includes: >>> >>> >>> 1. Integration of the TsFile C++ reader with DuckDB's table function API. >>> 2. Static builds of TsFile and its compression dependencies. >>> 3. TsFile-to-DuckDB type mapping and streaming result batches. >>> 4. Projection pushdown and time predicate pushdown. >>> 5. Build verification on Linux, macOS, and Windows through DuckDB Community >>> Extensions CI. >>> >>> >>> The DuckDB Community Extensions PR is available here: >>> >>> >>> https://github.com/duckdb/community-extensions/pull/2435 >>> >>> >>> Based on this initial implementation and validation, I would like to >>> discuss >>> bringing this work into a repository under the Apache TsFile organization. >>> The current repository is still hosted under my personal GitHub account: >>> >>> >>> https://github.com/ColinLeeo/tsfile-duckdb >>> >>> >>> If the community agrees with this direction, we can move the DuckDB >>> extension >>> to an Apache-owned repository in a future step. >>> >>> >>> The current prototype supports one local TsFile and one table per query. >>> Directory scans, automatic table discovery, multi-file scans, and >>> DuckDB-WASM support are not included yet. >>> >>> >>> Reviews and suggestions are welcome. >>> >>> >>> Best regards, >>> >>> >>> Colin Lee
