Hello Bruce,
On Fri, 18 Jun 2021 12:08:39 +0000 "Badger, Bruce" <[email protected]> wrote: > Dear pyarrow.fs team, > > We are implementing a custom pyarrow.fs filesystem to map the contents of > internal file stores as filesystems for use in Arrow. > > We have a suite of unit tests which exercise the internal parts of our > implementation, and we can run pyarrow.tests.test_fs to ensure that the > supplied pyarrow.fs filesystems work as we have them installed. > > I would like to include our custom filesystem as a sibling of the included > pyarrow.fs filesystems in the pyarrow.tests.test_fs tests in order to ensure > that our filesystem conforms to the expectations of the pyarrow.fs > implementation, and continues to conform as pyarrow.fs and our internal > systems evolve. > > Are the pyarrow.tests.test_fs tests extensible to allow the testing of custom > filesystems in addition to, and as peer of, the supplied filesystems? If so, > how is this intended to work? ... and if not, may I suggest that this be > added as a feature for the (hopefully not too distant) future? I don't think we intend to make the PyArrow test suite extensible. It is a test suite for PyArrow, not for third-party libraries. That said, it's probably easy to take those tests and copy/adapt them inside your own project. Be aware, however, that most tests for the PyArrow filesystems are written in C++. The Python tests do not intend to cover all functionality in detail. Regards Antoine.
