Hi Ben,
> > Our main concern is that this new arrow::dataset::RadosFormat class will > be > > deriving from the arrow::dataset::FileFormat class, which seems to raise > a > > conceptual mismatch as there isn’t really a RADOS format > > IIUC RADOS doesn't interact with a filesystem directly, so RadosFileFormat > would > indeed be a conceptually problematic point of extension. If a RADOS file > system > is not viable then I think the ideal approach would be to directly > implement the > Fragment [1] and Dataset [2] interfaces, forgoing a FileFormat > implementation altogether. > Unfortunately the only example we have of this approach is > InMemoryFragment, > which simply wraps a vector of record batches. > This is what we will go with, as this seems to be the quickest way for us to have a PoC and start experimenting with this. Thanks a lot for the invaluable feedback! 🙏
