KevinyhZou commented on issue #13256: URL: https://github.com/apache/gluten/issues/13256#issuecomment-6137189666
## Proposed ClickHouse backend design for reading Paimon tables This describes the implementation in the current local Gluten branch. It is work in progress and has not landed on `main`. ### Planning and offload - The Paimon Spark connector remains responsible for table metadata, snapshot selection, partition pruning, and planning `DataSplit`s. - `CHPaimonComponent` recognizes Paimon's `BatchScanExec` and replaces supported scans with `PaimonScanTransformer`. Unsupported scans fall back to the Spark/Paimon reader. - The transformer supports two read paths: | Path | Configuration | Scope | | --- | --- | --- | | Raw files | `spark.gluten.sql.columnar.backend.ch.paimon.native.read.enabled=false` (default) | Non-primary-key Parquet/ORC tables whose `DataSplit.convertToRawFiles()` succeeds; deletion vectors are excluded. | | Paimon C++ SDK | `spark.gluten.sql.columnar.backend.ch.paimon.native.read.enabled=true` | Supported non-primary-key tables; primary-key tables additionally require `spark.gluten.sql.columnar.backend.ch.paimon.native.pk.read.enabled=true`. | For the raw-file path, the transformer converts each split into Parquet/ORC file paths, offsets, lengths, and partition values. ClickHouse's existing file readers consume them. Primary-key tables and tables with deletion vectors remain on the Spark/Paimon reader in this mode. For the SDK path, the transformer serializes each Paimon `DataSplit` and passes its bytes and the table path through Substrait's `ReadRel.LocalFiles.FileOrFiles.PaimonReadOptions`. The ClickHouse backend deserializes the split, creates a `paimon::ReadContext` and `TableRead`, applies the requested column names, and reads batches using `paimon::BatchReader`. Arrow C Data batches are converted to ClickHouse chunks. A read-only libhdfs3 filesystem adapter handles `hdfs://` tables. The primary-key switch is separate because correct merge/delete semantics depend on the Paimon C++ reader. ### Build and deployment - Native Paimon support is optional: build `libch.so` with `-DGLUTEN_ENABLE_NATIVE_PAIMON=ON`. The current bundled build uses `paimon-cpp` v0.3.0 (`GLUTEN_BUNDLE_PAIMON=ON`) and builds it against the ClickHouse C++ runtime and Arrow 16.1.0. - Deploy the matching `libch.so` and `libpaimon.so` together. Build the ClickHouse native backend before the Maven package, and include the Maven `paimon` profile. - For HDFS tables, configure `spark.gluten.sql.columnar.backend.ch.runtime_config.hdfs.libhdfs3_conf` on executors. `spark.gluten.sql.columnar.backend.ch.runtime_config.paimon.read_batch_size` optionally adjusts the SDK batch size. ### Validation and remaining work `ClickHousePaimonSuite` covers raw Parquet/ORC reads, partition columns, primary-key fallback, and opt-in native reads of non-primary-key and primary-key tables, including `COUNT` with a partition filter. Native SDK tests are gated by `GLUTEN_TEST_NATIVE_PAIMON=1`; they require a build containing the SDK. Broader compatibility and stability testing is still needed for primary-key merge/delete cases and HDFS workloads before enabling native reads by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
