Hi, Yang Zhang Thanks for driving this FIP. Left minor comments: 1: New configuration options Do we really need introduce/expose these configuration options? According to my experience, the deault value always be fine, and hardly users will want to change it. So, I tend to not expose/introduce them unless we have a strong reason to expose them.
2: About code reuse In the Module Structure section, several proposed classes (e.g., TieringSplitGenerator, TieringCoordinator, TieringCommitter) already have counterparts in the existing Flink tiering module (fluss-flink-common). Rather than duplicating this logic into a new Spark-specific module, I'd recommend extracting the engine-agnostic components into a shared module (e.g.,fluss-lake-tiering-common or moving them down to fluss-client). Best regards, Yuxia ----- 原始邮件 ----- 发件人: "Yang Zhang" <[email protected]> 收件人: "dev" <[email protected]> 发送时间: 星期三, 2026年 5 月 06日 下午 3:15:26 主题: [SPAM][DISCUSS] Proposal: Add Spark as Tiering Compute Engine Hi everyone, I'd like to propose adding Spark as a supported compute engine for Fluss's data tiering service. Below is the motivation, key design decisions, and future roadmap. Motivation Spark is the most widely adopted compute engine in data platforms, supported across all major cloud platforms (Databricks, EMR, Dataproc) and used extensively in production environments. Adding Spark tiering brings these key benefits: - Broader accessibility: Users with existing Spark infrastructure can enable tiering without introducing additional compute frameworks, lowering the barrier to adoption. - Natural fit: Tiering is inherently a periodic batch operation (heartbeat -> poll -> read -> write -> commit), which maps cleanly to Spark's RDD-based batch parallelism. Each tiering round processes a snapshot or log segment in parallel across buckets, making it a good match for Spark's execution model. - Ecosystem completeness: Fluss 0.9 already supports Spark catalog, read, and write. Fluss 1.0 plans Union Read for Spark. Adding tiering completes the full lifecycle from a single Spark cluster. Key Highlights - Driver-Loop Architecture: A long-running driver loop orchestrates heartbeat -> split generation -> RDD job -> commit -> report. - Compatibility: Pure additive feature, no changes to existing APIs or Flink tiering, all 3 lake plugins (Paimon, Iceberg, Lance) work unmodified. Future Work - Multi-table concurrent tiering: Currently one table per loop iteration; parallel tiering will improve throughput and reduce latency. - Spark DataSource V2 integration: Replace the low-level RDD pattern with Spark's DataSource V2 or custom operators for better scheduler integration, metrics, and fault tolerance. - Decouple from datalake format: A single tiering service should tier tables to different lake formats simultaneously, rather than being bound to one format per instance. You can find the full proposal document here: https://cwiki.apache.org/confluence/display/FLUSS/FIP-42%3A+Add+Spark+as+Tiering+Compute+Engine You can find the PoC pull request here: https://github.com/apache/fluss/pull/3099 I'd appreciate your feedback on the design direction. Looking forward to the discussion! Best regards, Yang Zhang
