Pandas886 opened a new issue, #8700:
URL: https://github.com/apache/paimon/issues/8700
### Search before asking
- [x] I searched the existing issues and found nothing similar.
### Motivation
Databend is an open-source analytical database built in Rust. It is currently
adding native support for Paimon catalogs and tables through Apache Paimon's
Rust implementation, `paimon-rust`.
Apache Paimon already provides mature integrations with Flink and Spark. The
development of paimon-rust creates an additional opportunity to bring Paimon
into Rust-native query engines and data infrastructure.
Databend can provide a real adoption and validation environment for this
direction:
- Paimon provides Databend with an open table-format foundation for realtime
lakehouse workloads.
- Databend provides Paimon with Rust-native distributed analytical and write
workloads.
- Databend can provide paimon-rust with compatibility feedback from real
distributed workloads.
- The integration can encourage Paimon adoption and interoperability across
the Rust and Arrow data infrastructure ecosystems.
- Users gain more open and interoperable analytical engine choices for their
lakehouse architectures.
The proposed Databend integration currently focuses on:
- Native integration based on Apache paimon-rust
- Filesystem and REST catalogs
- Distributed table scans
- Paimon system tables
- Distributed writes for supported append and primary-key tables
Databend implementation:
https://github.com/databendlabs/databend/pull/20132
### Solution
Once the Databend implementation is stable and its supported version and
capability boundaries are clear, we plan to add a Databend page to Apache
Paimon's ecosystem documentation.
Proposed file:
`docs/docs/ecosystem/databend.md`
The page would cover:
- Supported Databend and Paimon/paimon-rust versions
- Filesystem and REST Catalog configuration
- Verified read, write, and system-table capabilities
- SQL examples
- Verified data-type compatibility
- Compatibility notes where necessary
- Links to the official Databend documentation and implementation
The Databend community also plans to add corresponding Paimon documentation.
The two pages will link to each other and provide users with consistent,
verifiable integration information.
The documented capabilities and compatibility boundaries will be based on the
final merged and tested Databend implementation.
### Anything else?
This proposal aims not only to add a documentation entry, but also to
establish
a public collaboration point for the two communities around:
- Compatibility feedback from real paimon-rust workloads
- Distributed reads and writes
- Primary-key table interoperability
- REST Catalog interoperability
- The Rust and Arrow data ecosystems
- Joint examples and lakehouse best practices
Apache Paimon provides the open table-format foundation for realtime
lakehouses, paimon-rust brings that foundation into the Rust data ecosystem,
and Databend applies it to Rust-native distributed analytical and write
workloads.
Databend tracking issue:
https://github.com/databendlabs/databend/issues/20162
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
