Summary of Apache Fluss Community Call 29th May:
Meeting recording: https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo Meeting Notes: Community sync focused on incubation graduation adjustments, volunteer co-hosting, deletion vectors and union read (Iceberg/Paimon) progress and conference plans, FlusRust merge and roadmap, Spark optimizer, Append Column proposal, and Kafka compatibility scope. Issues & Tickets: 1. Incubation Graduation Guidelines and PMC List Update Context: - New ASF incubation graduation guidelines updated ~6 months ago require active contributions from all PMC nominees. Status: - Mentors and community were unaware; received challenges from incubator regarding PMC list compliance. Concerns, Challenges, and Blockers - Need to validate PMC nominees' activity; may delay graduation. - Coordination overhead to revise PMC list on private list. Ideas and Explanations - Discuss and refresh PMC roster based on recent contribution activity. - Proceed with other graduation requirements in parallel. Decisions and Next Steps - Action: Start private list discussion to refine PMC list and add active members (Owner: Jark; Mentor Chair looped). - Continue graduation process post-list update. Current Objective: - Align PMC list with new guidelines to unblock graduation and 1.0 timing. 2. Community Call Co-host Roster Context: - Co-hosting burden previously concentrated; Mehul drafted how-to guide. Status: - Multiple volunteers offered to rotate hosting. Concerns, Challenges, and Blockers - Single point of failure when primary host unavailable. Ideas and Explanations - Maintain a public roster; pick alternates as needed. Decisions and Next Steps - Action: Start a public mailing-list thread to collect volunteers and schedule months (Owner: 蒋晓峰/Niklas; Mehul to initiate if needed). Current Objective: - Establish a rotating co-host roster for resilience. 3. Deletion Vectors, Union Read: Iceberg/Paimon Progress and Demos Context: - Design posted last month; focus on deletion vectors and union read. POC efforts on Paimon; interest in Iceberg path. Status: - Paimon POC nearly ready; complexity higher than expected. - Plan to demo at Flink Forward Shenzhen. - Community sharing blogs/examples (Iceberg vs Paimon feeds). Concerns, Challenges, and Blockers - Complexity of Paimon compaction/erase semantics. - Time/ bandwidth to cover both Paimon and Iceberg before events. Ideas and Explanations - Target Iceberg union read alongside Flus to leverage market interest (e.g., Redpanda feature parity) and openness vs paywalled offerings. - Explore DuckDB extension for analytics consumption beyond Flink SQL. Decisions and Next Steps - Action: Coordinate Yuxia + Mehul + Muhammet to push Flus+Iceberg union read for Flus Forward; ensure demo readiness (Owner: Yuxia). - Action: Start Flink Dev discussion after refining design this month (Owner: Yuxia). - George to share dual-ingest blog/resources in community chat. Current Objective: - Deliver working union read (Iceberg first) and Paimon POC/demo; refine design and socialize on mailing lists. 4. Iceberg Table Maintenance and Gaps Context: - Giannis exploring improvements (orphan file deletion, maintenance service). Status: - Open question on whether maintenance should live as a service vs user-managed jobs. Concerns, Challenges, and Blockers - Unclear ownership model for maintenance orchestration. - Need prioritized list of small, impactful tasks. Ideas and Explanations - Compare approaches (Gravitino, Apache Amoro). - Start thread to enumerate needs; Giannis to tackle smaller items. Decisions and Next Steps - Action: Open design/prioritization thread for Iceberg maintenance gaps (Owner: Giannis). Current Objective: - Define and prioritize Iceberg maintenance enhancements aligned with Flus usage. 5. Catalogs, Access Control, and Cross-Engine Constraints (Snowflake) Context: - Discussion on catalog limitations (e.g., Snowflake external catalog disallows deletes) impacting interoperability. Status: - Raised as architectural risk; not yet an active proposal. Concerns, Challenges, and Blockers - Catalog feature disparity blocks deletes/equality deletes. - Fragmented auth across Flus, storage, and table formats. Ideas and Explanations - Explore unified auth token and delegated credentials across layers. - Monitor catalog capabilities as central to analyst workflows. Decisions and Next Steps - Spin out mailing list/side meetings for deeper exploration (Owners: interested parties). Current Objective: - Document constraints and potential integration patterns; avoid scope creep in core agenda. Fluss Rust Merge and Roadmap Context: - Consensus to merge FlusRust into main Flus repo and align releases. Status: - PR ready; CI passing; unified protobufs. Progress on complex types, limit/prefix scans, metrics, schema evolution for log tables. Python/C++ lagging. Concerns, Challenges, and Blockers - Bindings (Python/C++) lack parity. - Coordination on gateway work vs core gaps. Ideas and Explanations - Limit 1.0 Rust scope to core roadmap items; bindings follow later. - Stream interface to ease DataFusion integration. Decisions and Next Steps - Action: Review and merge FlusRust PR (Owners: reviewers/core). - Action: Create issues for stream interface and DataFusion provider alignment (Owner: Anton). - Action: Coordinate with Leonard/Hongshun/Junbo on Flus Gateway + Rust client roadmap; public Slack channel created (Owner: Leonard; Anton to plan issues). - Note: 1.0 release delayed pending graduation; FlusRust considered milestone for 1.0. Current Objective: - Complete merge; close core Rust gaps; plan gateway integration path; align release cadence with Flus 1.0. Spark Union Read and Optimizer Context: - 1.0 roadmap item covering union read and scan/pushdown optimizations. Status: - Union read implemented (except streaming); scan optimizations in progress (limits, partition/column pruning; pushdown/join interfaces under umbrella issues). Concerns, Challenges, and Blockers - Streaming union read lower priority; polish needed across features. Ideas and Explanations - Deliver a cohesive set: scan optimizations + union read + pushdown interfaces for 1.0. Decisions and Next Steps - Ongoing: Yang/Jark/Anton to continue collaboration and reviews. Current Objective: - Ship Spark optimizer features and union read as key 1.0 deliverables. Append Column Proposal (Local Enrichment) Context: - Keith's FIP proposes append-only column files on immutable log tables to support enrichment (e.g., embeddings) without duplicating data. Status: - Discussion thread open; early feedback likens approach to LANs append columns and aligns with future feature engineering needs; Iceberg exploring similar. Concerns, Challenges, and Blockers - Need concrete real-world use cases to validate scope and design. - Cross-feature implications (replication, durability, tiering). Ideas and Explanations - Use cases: inline vector embeddings on log tables; LANs tiering without duplicating tables. - Collect scenarios in FIP comments; consider recurring focused meetings if interest is high. Decisions and Next Steps - Action: Community to review doc and add concrete use cases; Keith to facilitate follow-ups (Owner: All; Keith). - Allow time due to many concurrent proposals. Current Objective: - Refine design with use-case evidence and subsystem considerations; build consensus for a robust implementation. Kafka Compatibility vs Migration Tool Context: - Two approaches debated via FIPs: full Kafka API compatibility vs migration-focused tooling. Status: - Ben authored broad-scope designs; community leaning to narrow initial scope; agreement to move to dev list for broader input. Concerns, Challenges, and Blockers - Large scope risks slow progress. - Messaging: Flus is not "another Kafka"; avoid confusion. Ideas and Explanations - Stage implementation: start with producer API to ingest via Kafka clients/log proxies; expand to consumer/admin later. - Clarify terminology (compatibility vs migration) in proposals. Decisions and Next Steps - Action: Define bare-minimum initial scope (producer-focused) and proceed with dev-list proposal (Owners: Yang with Ben; Giannis to encourage scoping). - Action: Full design sketched for major APIs; phased delivery plan documented (Owner: Jark/Ben). Current Objective: - Reach community agreement on initial scope and terminology; begin incremental implementation toward broader compatibility. Other & Incidental Topics: 1. Volunteer Energy and Load Sharing Community appreciated increased co-host volunteers; aim to avoid single-person burden; jokes aside, roster will distribute responsibilities. 2. Event Targets and Messaging Plan demos/talks at Flink Forward Shenzhen, Flus Forward, Lakehouse Days, and Community Over Code; Iceberg union read and DuckDB integration highlighted as high-impact narratives. 3. Public Channels for Coordination Created public Slack channel "Flus Rust Gateway" to coordinate Rust client and gateway work; open invite for contributors. Action Items - Start mailing-list thread re: next month's community call co-host - WATCH (5 secs): https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=350.9999 - Share Iceberg vs Paimon blog links in community chat - WATCH (5 secs): https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=679.9999 - Start mailing-list thread re: Iceberg table maintenance gaps/priorities - WATCH (5 secs): https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=967.9999 - Start mailing-list thread re: deletion vectors/Iceberg union read; then refine design doc - WATCH (5 secs): https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=3127.9999 - Create GitHub issues for FlusRust stream interface + Gateway; then coordinate w/ Leonard, Hongshun, Junbo - WATCH (5 secs): https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=3149.9999 Best Regards, Nicholas Jiang
