I think discussion on the doc has settled if you want to move it forward to a 
VOTE Tornike?

On 2026-10-25 13:28 UTC Tornike Gurgenidze wrote:
> Hi all,
> 
> I'd like to start a discussion on a SPIP to add a built-in ADBC (Arrow
> Database Connectivity) data source to Spark, alongside the existing JDBC
> one.
> 
> - SPIP doc:
> https://docs.google.com/document/d/1hewLmAc8yFCY4PYepE02Yz8iACCAWkRUZuhzHyyMqBY
> - JIRA: https://issues.apache.org/jira/browse/SPARK-60255
> - Earlier discussion: https://github.com/apache/spark/issues/54603
> - Prototype: https://github.com/tokoko/spark-adbc
> 
> Motivation
> 
> JDBC returns data one row at a time, and Spark reads each value with a
> separate getXxx call. Many analytical databases store and send data in
> columns. Reading them through JDBC means converting columns to rows in the
> driver, reading each value one by one in Spark, and then converting back to
> columns again if Comet or Gluten is used. ADBC drivers return Arrow record
> batches instead. Spark can wrap those as ColumnarBatches through
> ArrowColumnVector, almost without copying. On the prototype, reads from
> columnar databases are 3-10x faster than JDBC, and the gap grows on wider
> tables.
> 
> What's proposed
> 
> - A new optional "adbc" format. There are no public API changes, and the
> JDBC source is unchanged. ADBC complements JDBC; it doesn't replace it.
> - Pushdown of column pruning, filters, limit, Top-N and aggregates
> (COUNT/SUM/MIN/MAX/AVG).
> - Two kinds of partitioning that can be combined:
>   - JDBC-style range partitioning
> (partitionColumn/lowerBound/upperBound/numPartitions).
>   - Driver partitioning through ADBC's executePartitioned, where each
> descriptor the driver returns becomes one Spark task. Flight SQL servers
> already split results this way.
> - Appending writes through ADBC bulk ingest, one transaction per task where
> the driver supports it. This is the same at-least-once guarantee as the
> JDBC writer.
> - No per-database dialect classes in Spark. The driver reports the few
> syntax choices pushdown needs through GetInfo/SqlInfo, and filter values
> are sent as bound Arrow parameters, not literals. As a second path,
> Substrait plans can be handed to drivers that translate them to SQL
> themselves.
> 
> Out of scope for the first version: atomic job-level writes and Structured
> Streaming.
> 
> Upstream dependencies. Two Arrow PRs would improve this. Neither blocks a
> first version:
> 
> - apache/arrow#49796: SqlInfo codes for limit/offset syntax, NULLS
> ordering, LIKE escaping and parameter markers.
> - apache/arrow-adbc#4317: a partitioned bulk ingest API. It maps directly
> onto Spark's write commit protocol, which would make writes atomic across
> the whole job.
> 
> The prototype currently runs a shared test suite against 13 engines:
> PostgreSQL, MySQL, MSSQL, DuckDB, ClickHouse, Trino, Presto, DataFusion,
> SingleStore, Exasol, Oracle, Flight SQL (GizmoSQL) and Spark Connect.
> 
> Longer-term direction
> 
> The first goal is narrow: a faster alternative to JDBC for databases that
> ship ADBC drivers. Longer term, I think ADBC could also become a general
> way for non-JVM sources to integrate with Spark. ADBC already covers
> catalog metadata (GetObjects, GetTableSchema), driver-planned partitioned
> reads, and bulk ingest, with partitioned ingest proposed in
> apache/arrow-adbc#4317. Substrait pushdown would let a driver accept more
> than SQL. Together, these could let catalogs and table formats with an ADBC
> driver plug into Spark, for example through a TableCatalog backed by
> GetObjects, without a Spark-specific connector. To try the idea out, I've
> put together a small proof of concept: an ADBC driver for Iceberg built on
> iceberg-rust (https://github.com/tokoko/adbc-iceberg). It's early and
> experimental so far though.
> 
> Thanks to everyone who gave early feedback on the issue. Holden has offered
> to shepherd the SPIP. Comments are welcome here or on the doc.
> 
> Thanks,
> Tornike
> 

---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]

Reply via email to