Hi Hongzhi, Haonan, Chris, and all,
Thank you for the discussion. I agree that Apache Arrow C++ provides a good reference for managing third-party dependencies. We should not treat copying complete third-party source trees into the TsFile repository as the default approach. Instead, the dependency source should be selected according to the build and distribution scenario. TsFile C++ mainly may needs to support the following two types of scenarios: 1. Native packages for Debian, Fedora, Homebrew, and other package management systems. These builds generally need to use system-provided libraries, while the package manager handles dependency installation, version management, and security updates. 2. Regular source builds, Windows, embedded systems, and offline environments. These scenarios require better control over dependency versions and should minimize the environment preparation required from users. Based on these scenarios, I propose introducing the following CMake option: TSFILE_DEPENDENCY_SOURCE=AUTO|SYSTEM|BUNDLED The modes would be defined as follows: - AUTO: The default mode. Prefer compatible system dependencies and fall back to bundled dependencies when they are unavailable. - SYSTEM: Use only dependencies provided by the system or package manager. The build should fail with a clear error if a dependency is missing or incompatible. This mode is mainly intended for native Debian, Fedora, and Homebrew package builds. - BUNDLED: Let the TsFile build system manage pinned dependency versions. This mode is mainly intended for regular source builds, portable binary packages, Windows, embedded systems, and offline environments. BUNDLED does not mean that complete third-party source trees must be maintained in the TsFile repository. Similar to Arrow, we can centrally record dependency versions, source archive URLs, and SHA256 checksums, and then build from pinned upstream source archives. To support offline builds, we can also provide a dependency pre-download mechanism so that users can prepare the required archives in advance and build from a local cache without network access. If a small amount of third-party source code genuinely needs to be kept directly in the TsFile repository, I suggest treating it as an exception and requiring the following conditions: - The code is small and stable. - Its license permits redistribution and integration. - Its upstream source and version or commit are documented. - Any source trimming and local modifications are documented. - The required license files are preserved, and LICENSE and NOTICE are updated where necessary. - The inclusion is reviewed by the community. As the next step, we can inventory the existing dependencies under cpp/third_party, including antlr4-cpp-runtime, snappy, lz4, lzokay, simde, and zlib. For each dependency, we should determine: - Whether it can use a system package. - Which system library versions are supported. - Whether a bundled fallback is required. - Whether keeping its source code in the repository is necessary. - Whether its source and license information are complete. - Whether any local patches need to be maintained. CI should also validate the SYSTEM, BUNDLED, and AUTO modes, as well as offline builds using pre-downloaded dependencies. I believe this approach can balance native package requirements, cross-platform and offline builds, license compliance, and long-term maintenance costs. If the community agrees with this direction, I will refine the proposal based on further feedback and then start a formal vote. The implementation will be submitted and reviewed incrementally through separate pull requests. Further comments are welcome. Thanks, Colin At 2026-07-10 21:30:01, "Christofer Dutz" <[email protected]> wrote: >Hi Haonan, > >Thanks for this information … I was just about to say: If you copy stuff in >the repo, you must list it in the LICENSES file together with which license >it’s under and bundle the license text along with the rest. > >If it’s really just a hand full of files from a stable repo with a license >that allows integration, and pulling in the full third party repo is total >overkill, then I’d agree to integrate these. If it’s just „simpler“, I’d >prefer the other approaches. > >Chris > >Von: Haonan Hou <[email protected]> >Datum: Donnerstag, 9. Juli 2026 um 10:51 >An: [email protected] <[email protected]> >Betreff: Re: Re:AW: Proposal for Managing Third-Party Source Code in the C++ >Module > >Hi, > >I looked a bit at how Apache Arrow C++ handles third-party dependencies, and I >think it may be a useful reference for this discussion. > >Arrow does not simply put all third-party source code directly into the main >repository. Instead, its CMake build supports several dependency resolution >modes through `ARROW_DEPENDENCY_SOURCE`, such as `AUTO`, `SYSTEM`, `BUNDLED`, >`CONDA`, `VCPKG`, and `BREW`. > >In particular, the `BUNDLED` mode usually means that Arrow builds the >dependency from a pinned source archive during the build, rather than >maintaining the full upstream source tree in the repository. The dependency >versions, source URLs, and SHA256 checksums are recorded in >`cpp/thirdparty/versions.txt`. Arrow also provides >`cpp/thirdparty/download_dependencies.sh` to pre-download these archives for >offline builds. > >At the same time, Arrow does keep a small number of vendored source files >in-tree under paths such as `cpp/src/arrow/vendored/`. These cases are >explicitly documented in `LICENSE.txt`, including the upstream source and >license information. > >So Arrow seems to use a mixed approach: > >- Prefer system or environment-provided dependencies when suitable. >- Allow bundled builds from pinned source archives for portability and >reproducibility. >- Support offline builds by pre-downloading dependency archives. >- Keep only selected vendored source code in-tree, with clear license >documentation. >- Avoid treating in-repository third-party source copies as the default >dependency management model. > >For TsFile C++, maybe we can consider a similar compromise. Instead of always >copying third-party source code into `cpp/third_party`, we could introduce a >dependency policy like: > >- Use system dependencies where reasonable. >- Provide a bundled build option for important C++ dependencies, using pinned >versions, URLs, and checksums. >- Support offline builds by allowing users to pre-download dependency archives. >- Only vendor source code directly in the repository for small, stable, >carefully reviewed cases. >- For any in-tree vendored code, require clear documentation of origin, >version or commit, license, trimming scope, and local modifications. > >This may address the portability and build stability concerns while also >reducing legal, licensing, and long-term maintenance risks. > >References: > >- Arrow C++ dependency management: > > https://arrow.apache.org/docs/developers/cpp/building.html#build-dependency-management >- Arrow bundled dependency versions: > https://github.com/apache/arrow/blob/main/cpp/thirdparty/versions.txt >- Arrow license documentation for vendored/included code: > https://github.com/apache/arrow/blob/main/LICENSE.txt > >Best, >Haonan > >On 2026/07/09 07:42:52 ColinLee wrote: >> Hi Chris, >> >> >> Thanks for raising this. Your concerns make sense, especially around >> long-term >> maintenance, redistribution rights, and the risk of treating vendored source >> as >> a default dependency-management pattern. >> >> >> Let me clarify the scope of what I had in mind. The third-party code I was >> looking at is mainly for encoding, decoding, and compression support. These >> implementations are generally mature and stable, and the corresponding file >> formats or compression formats do not change frequently. >> >> >> I also agree that copying third-party code into the repository should not >> become >> a general dependency-management approach. For dependencies that are actively >> evolving, security-sensitive, or likely to require frequent upstream >> synchronization, we should discuss the appropriate integration approach >> case by case before introducing them, rather than defaulting to vendoring the >> source code. >> >> >> My intent was only to explore whether a small set of stable codec and >> compression implementations might need a different treatment for C++ build >> portability and format compatibility. I look forward to further discussion on >> the legal, licensing, maintenance, and build-portability aspects so that we >> can >> choose the right approach together. >> >> >> Best, >> Colin >> >> >> >> >> >> >> >> >> >> >> >> >> >> At 2026-07-09 14:42:30, "Christofer Dutz" <[email protected]> wrote: >> >Hi Colin, >> > >> >Are you proposing to copy the code of another project into the repo here >> >and to also release that alongside the project? >> > >> >If that’s the case I would strongly suggest to not do that. >> > >> >In general, this would be a hostile fork of the source project and you end >> >up having to manually sync it. >> > >> >The way more pressing problem is that we generally only have code in our >> >repositories and releases that we have the rights to share. We wouldn’t >> >have this from the original project. >> > >> >I don’t know the specifics of the project you want to bring in, but I think >> >copying it here will cause more legal, license and trademark issues than it >> >will solve build issues. >> > >> >Chris >> > >> > >> >Von: ColinLee <[email protected]> >> >Datum: Donnerstag, 9. Juli 2026 um 05:04 >> >An: dev <[email protected]> >> >Betreff: Proposal for Managing Third-Party Source Code in the C++ Module >> > >> > >> >Hi all, >> > >> >To improve build stability across different platforms and usage scenarios, >> >I propose that we manage necessary third-party libraries by maintaining >> >their source code directly in the repository. >> > >> >## Why We Vendor Source Code >> > >> >The C++ module needs to support Linux, macOS, MSVC, embedded environments, >> >and other build environments. If we rely on system libraries or download >> >dependencies during the build, it becomes harder to control library >> >versions, compiler options, and network availability. >> > >> >By keeping the source code under `cpp/third_party`, we can better control >> >build details such as PIC, static linking, MSVC runtime settings, and >> >source trimming. It also helps reduce the extra dependency installation >> >burden for users of `libtsfile`. >> > >> >## Initial Import and Commit Structure >> > >> >Third-party source code should be placed under >> >`cpp/third_party/<third-party-dir>/`. We should only keep the source subset >> >that is actually needed, and preserve the corresponding license files. >> > >> >I also suggest adding a README to record the source origin, version or >> >upstream commit, trimming scope, and license information. >> > >> >Suggested commit structure: >> > >> >- First commit: import the third-party source code, license files, and >> >README. >> >- Second commit: integrate it with TsFile logic and local CMake glue. >> > >> >## Future Changes >> > >> >In general, we should avoid modifying third-party source code directly. >> >Prefer CMake options, wrappers, or adapters when possible. >> > >> >If third-party source code must be modified, the change should be kept >> >separate from TsFile logic as much as possible. Alternatively, the >> >corresponding commit should include a patch that tracks the local >> >modification against upstream. >> > >> >Future commits should also remain reasonably independent: >> > >> >- Third-party version upgrade: separate commit. >> >- Third-party source modification or patch: separate commit when possible, >> >or include the patch in the corresponding commit. >> > >> >This keeps third-party code origins clear, licenses traceable, and local >> >changes reviewable. It also makes future upgrades or rollbacks easier. >> > >> >Thanks. >> > >> >colin >>
