Or we could hold off until after the first community sync end of November
if you wanted to bring it up there?

On Mon, Nov 16, 2026 at 2:36 PM Holden Karau via dev <[email protected]>
wrote:

> I think discussion on the doc has settled if you want to move it forward
> to a VOTE Tornike?
>
> On 2026-10-25 13:28 UTC Tornike Gurgenidze wrote:
> > Hi all,
> >
> > I'd like to start a discussion on a SPIP to add a built-in ADBC (Arrow
> > Database Connectivity) data source to Spark, alongside the existing JDBC
> > one.
> >
> > - SPIP doc:
> >
> https://docs.google.com/document/d/1hewLmAc8yFCY4PYepE02Yz8iACCAWkRUZuhzHyyMqBY
> > - JIRA: https://issues.apache.org/jira/browse/SPARK-60255
> > - Earlier discussion: https://github.com/apache/spark/issues/54603
> > - Prototype: https://github.com/tokoko/spark-adbc
> >
> > Motivation
> >
> > JDBC returns data one row at a time, and Spark reads each value with a
> > separate getXxx call. Many analytical databases store and send data in
> > columns. Reading them through JDBC means converting columns to rows in
> the
> > driver, reading each value one by one in Spark, and then converting back
> to
> > columns again if Comet or Gluten is used. ADBC drivers return Arrow
> record
> > batches instead. Spark can wrap those as ColumnarBatches through
> > ArrowColumnVector, almost without copying. On the prototype, reads from
> > columnar databases are 3-10x faster than JDBC, and the gap grows on wider
> > tables.
> >
> > What's proposed
> >
> > - A new optional "adbc" format. There are no public API changes, and the
> > JDBC source is unchanged. ADBC complements JDBC; it doesn't replace it.
> > - Pushdown of column pruning, filters, limit, Top-N and aggregates
> > (COUNT/SUM/MIN/MAX/AVG).
> > - Two kinds of partitioning that can be combined:
> >   - JDBC-style range partitioning
> > (partitionColumn/lowerBound/upperBound/numPartitions).
> >   - Driver partitioning through ADBC's executePartitioned, where each
> > descriptor the driver returns becomes one Spark task. Flight SQL servers
> > already split results this way.
> > - Appending writes through ADBC bulk ingest, one transaction per task
> where
> > the driver supports it. This is the same at-least-once guarantee as the
> > JDBC writer.
> > - No per-database dialect classes in Spark. The driver reports the few
> > syntax choices pushdown needs through GetInfo/SqlInfo, and filter values
> > are sent as bound Arrow parameters, not literals. As a second path,
> > Substrait plans can be handed to drivers that translate them to SQL
> > themselves.
> >
> > Out of scope for the first version: atomic job-level writes and
> Structured
> > Streaming.
> >
> > Upstream dependencies. Two Arrow PRs would improve this. Neither blocks a
> > first version:
> >
> > - apache/arrow#49796: SqlInfo codes for limit/offset syntax, NULLS
> > ordering, LIKE escaping and parameter markers.
> > - apache/arrow-adbc#4317: a partitioned bulk ingest API. It maps directly
> > onto Spark's write commit protocol, which would make writes atomic across
> > the whole job.
> >
> > The prototype currently runs a shared test suite against 13 engines:
> > PostgreSQL, MySQL, MSSQL, DuckDB, ClickHouse, Trino, Presto, DataFusion,
> > SingleStore, Exasol, Oracle, Flight SQL (GizmoSQL) and Spark Connect.
> >
> > Longer-term direction
> >
> > The first goal is narrow: a faster alternative to JDBC for databases that
> > ship ADBC drivers. Longer term, I think ADBC could also become a general
> > way for non-JVM sources to integrate with Spark. ADBC already covers
> > catalog metadata (GetObjects, GetTableSchema), driver-planned partitioned
> > reads, and bulk ingest, with partitioned ingest proposed in
> > apache/arrow-adbc#4317. Substrait pushdown would let a driver accept more
> > than SQL. Together, these could let catalogs and table formats with an
> ADBC
> > driver plug into Spark, for example through a TableCatalog backed by
> > GetObjects, without a Spark-specific connector. To try the idea out, I've
> > put together a small proof of concept: an ADBC driver for Iceberg built
> on
> > iceberg-rust (https://github.com/tokoko/adbc-iceberg). It's early and
> > experimental so far though.
> >
> > Thanks to everyone who gave early feedback on the issue. Holden has
> offered
> > to shepherd the SPIP. Comments are welcome here or on the doc.
> >
> > Thanks,
> > Tornike
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [email protected]
>
>

Reply via email to