I was working on open sourcing a file+sql+api(rest/grpc/graphql) "pluggable" package, but it has stalled for legal reasons..
For file sources it was using pyarrow.dataset or datafusion. For sql sources it was using ADBC, DBAPI, Arrow-ODBC. Additional custom sources could be injected imported python classes. You basically configure a "gateway" definition that includes schema (aka contract) and a source definition. There are two end user functions: get_data() and write_data() that works on any combination of the above. If you want to swap in a different source, change your gateway definition and not your code.. Lately I was testing the new adbc_bridge driver package and was able to plug it in with zero code changes needed. Here's the common ground approach I took between SQL and FILE sources.. https://github.com/apache/arrow/issues/39128 - adds plain text DNF "expressions" to generate pyarrow.compute expressions or datafusion expressions. https://github.com/tobymao/sqlglot/issues/3217 - adds plain text DNF "expressions" to generate sqlglot expressions. Couple of file sourcing items: For file sources you designate a primary "date" or "timestamp" column and then decide whether to date partition your data using YYYYMMDD, YYYYMM or YYYY subdirectories when writing. You could also specify additional partitioning columns to create more subdirectries using any of the partitioning flavors supported by pyarrow (Hive, Directory, Filename). Defining primary key columns on a file dataset would create indexes on the client side output depending on your desired output format: aka pandas index. Reading and Writing supported pandas, pyarrow tables, pyarrow datasets, pyarrow scanner, pyarrow recordbatcheader, polars dataframe, pandas, etc.. as inputs or outputs with automatic detection when passing any of the objects to write_data(). For ADBC it would create a recordbatchreader first out of the source before calling adbc_ingest(). pyarrow.filesystem was pluggable as part of the source definition so hdfs, azure blob, s3, gcs, etc.. were all supported. This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See https://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to https://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy. For a list of BlackRock's office addresses worldwide, see https://www.blackrock.com/corporate/about-us/contacts-locations. © 2026 BlackRock, Inc. All rights reserved.
