GitHub user CurtHagenlocher added a comment to the discussion: Proposal: ADBC file drivers with pluggable storage
> Would you be up for turning what you have in your Example section above into > a code example so we can see what it looks like? This is the 2026 version of "works on my machine" which is that Claude *claims* it works on my machine... Changes to ADBC, including new "drivers": https://github.com/CurtHagenlocher/arrow-adbc/tree/file-api Changes to datafusion-table-providers: https://github.com/CurtHagenlocher/datafusion-table-providers/tree/file-drivers I'm somewhat kidding; everything is indeed vibe-coded but I did validate that it works under both WSL and straight Windows. I could try to validate the changes under MacOS if that helps anyone, but right now this is mostly about the utility / desirability of this idea. The demo is in `crates/adbc/examples/file_drivers_demo.rs` in the datafusion-table-providers branch. It writes budget.xlsx, uploads archive.zip (12 monthly CSVs) to an S3 emulator, writes the four manifests, and produces the following output showing what it's done: ``` providers: s3, zip (from C:\Users\curt\AppData\Local\Temp\.tmpGjCBFh\adbc/providers), plus the built-in file drivers: excel, csv (from C:\Users\curt\AppData\Local\Temp\.tmpGjCBFh\adbc/drivers) prices = C:/Users/curt/AppData/Local/Temp/.tmpGjCBFh/data/budget.xlsx :: Prices orders = zip:s3://adbc-demo/archive.zip!/ :: orders SELECT o.region, p.description, SUM(o.qty) AS units, ROUND(SUM(o.qty * p.unit_price), 2) AS revenue FROM orders o JOIN prices p ON o.sku = p.sku WHERE o.region <> 'APAC' AND o.qty >= 3 AND o.order_date >= '2024-07-01' AND p.unit_price < 100 GROUP BY o.region, p.description ORDER BY o.region, revenue DESC Physical plan: SortPreservingMergeExec: [region@0 ASC NULLS LAST, revenue@3 DESC] SortExec: expr=[region@0 ASC NULLS LAST, revenue@3 DESC], preserve_partitioning=[true] ProjectionExec: expr=[region@0 as region, description@1 as description, sum(o.qty)@2 as units, round(sum(o.qty * p.unit_price)@3, 2) as revenue] AggregateExec: mode=FinalPartitioned, gby=[region@0 as region, description@1 as description], aggr=[sum(o.qty), sum(o.qty * p.unit_price)] RepartitionExec: partitioning=Hash([region@0, description@1], 24), input_partitions=24 AggregateExec: mode=Partial, gby=[region@1 as region, description@2 as description], aggr=[sum(o.qty), sum(o.qty * p.unit_price)] HashJoinExec: mode=Partitioned, join_type=Inner, on=[(sku@0, sku@0)], projection=[qty@1, region@2, description@4, unit_price@5] RepartitionExec: partitioning=Hash([sku@0], 24), input_partitions=12 AdbcScanExec: target=orders, columns=["sku", "qty", "region"], filters=[region != Utf8("APAC"), qty >= Int64(3), order_date >= Utf8("2024-07-01")], partitions=12 RepartitionExec: partitioning=Hash([sku@0], 24), input_partitions=24 FilterExec: unit_price@2 < 100 RepartitionExec: partitioning=RoundRobinBatch(24), input_partitions=1 AdbcScanExec: target=Prices, columns=["sku", "description", "unit_price"], partitions=1 +--------+----------------+-------+---------+ | region | description | units | revenue | +--------+----------------+-------+---------+ | AMER | Cable, 10 m | 689 | 21359.0 | | AMER | Bracket, steel | 694 | 8675.0 | | AMER | Anchor bolt | 689 | 585.65 | | EMEA | Cable, 10 m | 587 | 18197.0 | | EMEA | Bracket, steel | 583 | 7287.5 | | EMEA | Anchor bolt | 590 | 501.5 | +--------+----------------+-------+---------+ ``` GitHub link: https://github.com/apache/arrow-adbc/discussions/4877#discussioncomment-18910658 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
