Hi all,
Following up on the 0.2.0-incubating roadmap thread, where the repo
merge came through as the strongest priority signal, I am opening this
as its own [DISCUSS] as promised.
Per the consensus on that thread, 0.2.0 development continues on
fluss-rust in parallel and is not blocked by this FIP.
I do not currently have edit rights on the FLUSS Confluence space, so
I am posting the full draft inline. Once discussion converges, a
committer can paste the agreed text into a FIP-XX page. Please treat
the text below as a working draft; I am happy to revise it as feedback
comes in.
===============================================================
FIP-XX: Consolidate apache/fluss-rust into apache/fluss
Discussion thread: this thread
Vote thread: TBD
Release: Fluss 1.0 (target)
Status: Under Discussion
Motivation
=============
Fluss source code currently lives in two ASF repos.
1.1 Current layout today
apache/fluss
├─ fluss-server/
├─ fluss-client/
├─ fluss-rpc/
│ └─ src/main/proto/*.proto
├─ fluss-flink/, fluss-spark/, ...
├─ website/
│ └─ fluss.apache.org
└─ .github/workflows/
apache/fluss-rust
├─ crates/fluss/
│ └─ src/proto/*.proto
│ hand-ported subset
├─ bindings/python/
├─ bindings/cpp/
├─ bindings/elixir/
├─ website/
│ └─ clients.fluss.apache.org
└─ .github/workflows/
fluss-rust was incubated in its own repo so the Rust core could move
quickly. With 1.0 GA on the horizon, the costs of the split now exceed
the benefit.
As discussed on the 0.2.0 thread, 1.0 should represent the complete
form of the project: all language SDKs and unified release
coordination from a single repository, to signal maturity to users. A
fragmented multi-repo layout at GA undermines that message.
1.2 Concrete costs today
Protobuf schema is duplicated and can diverge. The source of truth is
fluss-rpc/src/main/proto/FlussApi.proto (package fluss).
fluss-rust carries a hand-maintained subset at
crates/fluss/src/proto/fluss_api.proto (package proto).
The subset exists only because the two repos cannot share files;
nothing enforces consistency.
Server version is pinned manually in two files:
crates/fluss-test-cluster/test-images.env and
bindings/elixir/test/support/cluster.ex, both pointing today to
apache/fluss:0.9.0-incubating, with no automation.
Docs are split across two subdomains. fluss.apache.org/docs/apis/
*-client.md pages are stubs that link out to
clients.fluss.apache.org/user-guide//... This means two
Docusaurus builds and two deployment pipelines.
Infrastructure is duplicated: 8 CI workflows in each repo,
duplicated LICENSE / NOTICE / DISCLAIMER files at repo root, and
two release trains.
Cross-repo changes cost two PRs, two reviews, and a Docker image cut
per wire-protocol change.
1.3 ASF peer evidence
ASF peer evidence points the other way: mature ASF projects
overwhelmingly colocate language bindings with core.
Apache Thrift -> apache/thrift/lib/{java,py,cpp,rs,go,rb,...}
Apache Avro -> apache/avro/lang/{java,py,c,rust,ruby,...}
Apache Beam -> apache/beam/sdks/{java,python,go,typescript}
PyFlink -> apache/flink/flink-python/
PySpark -> apache/spark/python/
Apache Arrow is the visible counter-example: arrow-rs, arrow-go,
arrow-java, and arrow-js live in separate repos. Apache Avro also
externalized its Go binding (apache/avro-go) on a similar trajectory.
In both cases, the split happened after the format stabilized and
after the specific language community had grown large enough to
self-sustain.
Fluss is at neither milestone. The wire protocol is still evolving
aggressively pre-1.0, and the Rust, Python, C++, and Elixir bindings
are all co-maintained by the same contributors. Splitting now incurs
coordination overhead that Arrow and Avro only accepted at maturity,
without yet having their enabling conditions.
The Flink connector-externalization retrospective is sometimes cited,
but does not really apply: connectors are N external-system
integrations maintained by independent communities, not N language
wrappers over one wire protocol.
Public interfaces
====================
None. Published artifacts remain unchanged:
Maven Central: org.apache.fluss:fluss-*
crates.io: fluss-rs
PyPI: pyfluss
git: Elixir fluss (planned hex.pm publication)
CMake / Bazel: fluss-cpp
Proposed changes
===================
Jark outlined the merge as five workstreams: code migration, CI
unification, documentation restructuring, issue migration, and
contributor workflow changes. Sections 3 and 4 address each.
3.1 Code migration - target layout
apache/fluss/
├─ fluss-server/ (Java, unchanged)
├─ fluss-client/ (Java, unchanged)
├─ fluss-rpc/ (single .proto source; see 3.4)
│ └─ src/main/proto/FlussApi.proto
├─ fluss-flink/, fluss-spark/, ... (Java modules, unchanged)
├─ fluss-rust/ (new: today's apache/fluss-rust tree)
│ ├─ Cargo.toml, Cargo.lock
│ ├─ crates/{fluss, examples, fluss-test-cluster}/
│ ├─ bindings/{python, cpp, elixir}/
│ ├─ justfile, MODULE.bazel, rust-toolchain.toml,
│ │ deny.toml, rustfmt.toml
│ └─ scripts/
├─ website/ (unified Docusaurus; see 3.3)
└─ .github/workflows/ (unified with path filters; see 3.2)
fluss-rust mirrors sibling naming such as fluss-server/ and
fluss-flink/. Bindings stay nested because they share a Cargo
workspace, rust-toolchain.toml, deny.toml, and workspace-level
dependency resolution with the Rust core.
3.2 CI unification
A single .github/workflows/ directory. Path filters are a hard
requirement:
Java CI: pom.xml, fluss-*/** (excluding fluss-rust/), tools/**
Rust core CI: fluss-rust/Cargo.*, fluss-rust/crates/**
OR fluss-rpc/src/main/proto/**
Binding CI: fluss-rust/bindings//**
OR fluss-rpc/src/main/proto/**
Docs build: website/**
Existing release workflows move over unchanged. Their triggers are
tag-based and already disjoint from each other.
3.3 Documentation restructuring
fluss/website/docs/
├─ intro.md
├─ quickstart/
├─ concepts/
├─ clients/ (new section)
│ ├─ support-matrix.md
│ ├─ java/
│ ├─ rust/
│ ├─ python/
│ └─ cpp/
├─ engine-flink/, engine-spark/, ...
└─ ...
clients.fluss.apache.org is retired. Redirects from old paths use
@docusaurus/plugin-client-redirects, with mappings generated by
walking the two sidebars.ts files. The
fluss-rust/.github/workflows/deploy_documentation.yml workflow is
removed; deployment converges on the existing fluss/website ->
apache/fluss-website dispatch.
3.4 Protobuf single source of truth
Today, crates/fluss/src/proto/fluss_api.proto exists as a
hand-maintained subset of the Java-side canonical proto solely because
the two repos could not share files. The merge removes that
constraint, and this FIP removes the subset.
After the structural merge:
Delete fluss-rust/crates/fluss/src/proto/fluss_api.proto.
Point fluss-rust/crates/fluss/build.rs at the canonical proto:
config.compile_protos(
&["../../../fluss-rpc/src/main/proto/FlussApi.proto"],
&["../../../fluss-rpc/src/main/proto"],
)?;
Update the .bytes() prefixes in build.rs from ".proto." to
".fluss." because the canonical proto declares package fluss.
Update lib.rs include! from "/proto.rs" to "/fluss.rs", since
prost names the generated file after the proto package.
The vendored copy exists only to satisfy crates.io packaging rules;
the monorepo build reads the canonical proto directly. For crates.io
publishing, vendor the canonical .proto into the crate at
cargo-publish time via scripts/release.sh, with build.rs falling
back to the in-crate copy when the monorepo path is absent.
All existing crate::proto::TypeName call sites compile unchanged,
because the pub mod proto { ... } wrapper in lib.rs preserves the
import path.
After this, adopting a new server RPC becomes a single-PR change: the
message lands in the canonical proto and Rust types appear
automatically. Nothing remains to drift against, so no CI drift-gate
is needed. Section 6 covers why a drift-gate alone would be the wrong
answer.
3.5 Release cadence
Java server and Rust client keep their current independent SemVer
cadences until 1.0 GA. From 1.0 onward, both trees cut a jointly
numbered X.Y.Z release from a single tag; patch releases can still
ship independently where scope is binding-only or server-only.
Binding-specific SemVer continues where public APIs diverge, for
example if a pyfluss major bump is driven by PyO3 ABI changes.
Open to revision: the post-1.0 release model is listed in section 8
and likely deserves its own [DISCUSS] once this FIP lands.
Compatibility, deprecation, and migration plan
=================================================
4.1 Phases
[0 Agreement] -> [1 Dry run] -> [2 Merge window] -> [3 Archive] -> [4 Cleanup]
DISCUSS + VOTE
fork dry run
follow-up PRs
redirects
INFRA ticket
Agreement. This FIP's DISCUSS + VOTE. INFRA ticket filed and
resolved before VOTE is called, so the migration plan is confirmed
viable rather than assumed.
Dry run on a fork: verify CI, path filters, docs build, redirects,
and git log --follow across the boundary.
Merge window. Short freeze on apache/fluss-rust; structural merge
commit into apache/fluss. Protobuf (3.4) and docs (3.3)
consolidation land as separate follow-up PRs for reviewability.
Archive. INFRA archives apache/fluss-rust read-only; issues, PRs,
commits, and tags remain as permalinks. Subdomain redirects go
live. A pinned README on the archived repo points to apache/fluss,
since GitHub does not forward content-moves between repos
automatically.
Cleanup. Consolidate root-level duplicates such as LICENSE /
NOTICE / DISCLAIMER and contributor docs.
4.2 History preservation
On a clone of apache/fluss-rust:
git filter-repo --to-subdirectory-filter fluss-rust
In apache/fluss:
git remote add rust ../fluss-rust-rewritten
git fetch rust
git merge --allow-unrelated-histories --no-ff rust/main
This preserves git log --follow and git blame --follow on every
rewritten file. git subtree breaks log --follow on files inside the
subtree; git filter-branch is deprecated.
4.3 Issue migration
Closed issues and PRs remain at their apache/fluss-rust archive URLs
as permalinks. Open issues at merge time are re-filed in apache/fluss
with cross-links, scripted via gh CLI. Open PRs are reopened by their
authors against the new path; GitHub does not support cross-repo PR
transfer.
4.4 Contributor workflow
Contributors re-clone and re-open in-flight PRs against the new path.
Downstream artifact consumers see no change. Docs visitors follow
redirects.
4.5 Timeline
There is no dependency on a specific release. Target completion is
before 1.0 GA, per the fluss-rust roadmap 0.2.0 DISCUSS consensus.
Within that bound, the community can pick the window that minimizes
rebase pain, most likely after 0.2.0-incubating is cut, to avoid
forcing in-flight roadmap work to rebase across the merge.
Test plan
============
In the merged repo, CI runs the full Java + Rust + all-binding
matrices on every PR.
Path-filter correctness is explicitly tested: a Rust-only PR must
not trigger Java jobs, and vice versa.
Protobuf: a change to fluss-rpc/src/main/proto/FlussApi.proto alone
produces a green Rust build with no Rust-side edit, confirming the
single-source claim.
Docs: every fluss-rust/website/docs/user-guide/... page renders
under fluss.apache.org/docs/clients/...; sampled old URLs redirect.
History: git log --follow on a sample migrated file shows full
pre-merge history.
Rejected alternatives
========================
Status quo. The section 1 costs recur every release cycle and grow
with each new binding.
Drift-gate instead of full proto consolidation. Keep the Rust
subset and add CI to flag divergence on messages present in both
files. Rejected because it catches only half the drift problem,
preserves a hand-maintained file whose only reason to exist was the
cross-repo constraint the merge removes, and adds custom comparison
logic to maintain.
Polyglot-repo federation, Iceberg-style. This works for Iceberg
because bindings have independent maintainer communities and the
table-format spec is stable enough that protocol-evolution coupling
is low. Neither holds for Fluss pre-1.0: bindings are co-maintained
by the same team, and the wire protocol is still evolving
aggressively. For Fluss in the run-up to 1.0, locking bindings to
server evolution is a feature, not a bug.
Apache Arrow precedent (arrow-rs split, 2021).
https://arrow.apache.org/blog/2021/05/04/rust-dev-workflow/
Section 1 covers the structural frame: Arrow split after
stabilization and after each binding had grown an autonomous
community. Arrow's specific 2021 rationale also inverts for Fluss.
Git submodules. Standard UX problems and does not solve the docs
split.
Further splitting into per-binding repos. This multiplies every cost
listed in section 1.
Flatten bindings to the top level of the merged repo. This breaks
the shared Cargo workspace. Keeping them under fluss-rust/
preserves it; rearrangement remains an option later.
Risks
========
CI blast radius if path filters are misconfigured. Mitigated by
sections 3.2 and 5.
In-flight PRs stall during the freeze. Mitigated by maintainers
proactively reopening high-value PRs.
Open questions
=================
Directory name for the Rust tree. Proposal: fluss-rust/
(mirrors fluss-server/ and fluss-flink/ sibling naming).
Alternatives: rust/, clients/rust/.
Docs section name. Proposal: docs/clients/ to unify client and
binding docs under one heading, while keeping docs/engine-*/ for
pipeline integrations. Alternative: keep docs/apis/ to reduce link
churn during migration.
Post-1.0 release model. Section 3.5 proposes synchronized X.Y.Z
across Java and Rust with independent patches. Alternative:
Arrow-style coordinated-but-independent versioning. This likely
deserves its own [DISCUSS] once this FIP lands.
Looking forward to feedback.
Best,
Anton
