Tornike Gurgenidze created SPARK-60255:
------------------------------------------
Summary: [SPIP] ADBC Data Source
Key: SPARK-60255
URL: https://issues.apache.org/jira/browse/SPARK-60255
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 5.0.0
Reporter: Tornike Gurgenidze
Summary
Add a built-in ADBC (Arrow Database Connectivity) data source to Spark,
alongside the existing JDBC data source. It reads and writes through any ADBC
driver and passes Arrow column batches straight into Spark as ColumnarBatches,
so data stays in columns from the database to Spark.
*SPIP document:* [SPIP: ADBC Data
Source|https://docs.google.com/document/d/1hewLmAc8yFCY4PYepE02Yz8iACCAWkRUZuhzHyyMqBY/edit?tab=t.0]
*Prototype:* [tokoko/spark-adbc|https://github.com/tokoko/spark-adbc]
Motivation
JDBC returns data one row at a time, and Spark reads each value separately. For
analytical databases that store or send data in columns, results are converted
from columns to rows at the driver, read value by value in Spark, and often
converted back to columns by vectorized engines (Comet, Gluten). On large reads
this conversion dominates. The spark-adbc prototype shows 3-10x read throughput
over JDBC on column-oriented databases.
Proposal
* New optional data source: \{{spark.read.format("adbc")}}. No public API
changes; JDBC is unchanged.
* Reads: Arrow batches wrapped as ColumnarBatch via ArrowColumnVector, without
copying.
* Pushdown of column pruning, filters, limit, Top-N and aggregates.
* Partitioning: JDBC-style range partitioning, plus driver partitioning through
ADBC executePartitioned.
* Writes: per-task appends through ADBC bulk ingest (same at-least-once
guarantee as the JDBC writer).
* Pushdown semantics without per-database dialect code in Spark: SQL generated
from driver-reported metadata, with an optional Substrait-based mode.
Non-goals
* Replacing or changing the JDBC data source.
* Atomic distributed writes in the first version (depends on
apache/arrow-adbc#4317).
* Structured Streaming.
Related upstream work
* [apache/arrow-adbc#4317|https://github.com/apache/arrow-adbc/pull/4317] -
partitioned bulk ingest
* [apache/arrow#49796|https://github.com/apache/arrow/pull/49796] -
dialect-related Flight SQL SqlInfo codes
*
[substrait-io/substrait#1283|https://github.com/substrait-io/substrait/issues/1283]
- SqlQuery read type
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
