Recording: https://fathom.video/share/T2WYJvZv1XX4sNzBsuDLSDzxMQM-DTew
Metting Notes: Kafka Compatibility Plugin - Ben demoed working Kafka-compatible API layer as Fluss plugin (not proxy); 1:1 wire compatibility + optional live schema conversion to typed Fluss tables. - POC passes much of Kafka test suite; supports SASL, compacted/log topic mapping (compacted→PK, log→log tables). - Gaps: transactions, some metadata, Scram; Java 11 limits async/perf. - Approach: codegen from JSON protocol specs (Tanzu/Redpanda style); Anton validated Elixir macro path. - Ben to publish design doc first, then staged PRs; FIP required before landing. 0.9.1 Patch Release - Code freeze starting; bug fixes + major Helm chart improvements backported (credits: Muhammet, Anton, Lorenzo). - Giannis = Release Manager (first-time); Yuxia supporting. Merge Fluss Rust into Main Repo - Consensus to co-locate for aligned release cycles, component reuse, simpler CI. - Anton to own schema evolution support for KV tables; unify Protobuf schemas across Rust/Java. - Post-merge: accelerate Rust-based Fluss Gateway using DataFusion. Iceberg Deletion Vectors & Union-Read - Yuxia has POC + Chinese design doc (inspired by Mooncake/Moonlink); English doc in progress. - Deletion vectors (position deletes) likely land first; union-read phased later. - Key risk: external compaction tools won't update mapping indices → inconsistency; proposing time-service-managed mapping. - Muhammet offered to help review/contribute. AI-Native Direction (Vectors & Multimodal) - Framing split into three tracks: (a) vector/multimodal data model, (b) real-time context engine, (c) agent skills/MCP. - Priority: add vector column type (currently stored as bytes); minimal Lance integration; use Fluss + Flink/Ray/Daft for real-time enrichment pipelines feeding downstream vector DBs. - Fluss is NOT positioning as vector search engine — real-time multimodal context pipeline instead. - Lorenzo to draft unified design doc (use cases + retrieval patterns + roadmap). - Mehul share multimodal findings with Ray/Daft Benchmarks & Performance - Anton building binding-overhead benchmarks: Rust core baseline, C++ ~97%, Elixir log tables surprisingly strong; issues found in C++ writer + Elixir Explorer conversions. - Gaps: broker-level + E2E metrics; multi-broker overhead unknown. - Plan: leverage OpenMessaging Benchmark + Yahoo KV benchmark; extend to Fluss bindings/brokers. - Jark + contributors to define community benchmark plan; publish reproducible numbers with methodology. Current Objective Ship stable 0.9.1; land Kafka-compat plugin via FIP; complete Rust merge and advance Iceberg union-read; define vector datatype as first concrete step toward AI-native positioning. Best Regards, Mehul Batra
