I wanted to draw attention to a PR[1] in the DataFusion repository that proposes a API designed to abstract away access to remote datasources, to allow plugging in sources such as S3.
It has had some great discussion and community engagement so far but given it is a fairly significant change to how I/O is performed in DataFusion and potentially sets the stage for some even greater changes in follow on PRs, I wanted to raise it in this forum too Andrew [1] https://github.com/apache/arrow-datafusion/pull/811
