Chang She created ARROW-18275:
---------------------------------
Summary: Allow custom reader/writer implementation for arrow
dataset read/write path
Key: ARROW-18275
URL: https://issues.apache.org/jira/browse/ARROW-18275
Project: Apache Arrow
Issue Type: New Feature
Components: Python
Affects Versions: 10.0.0
Reporter: Chang She
We're implementing a "versionable" data format where the read/write path has
some metadata handling which we currently can't plug into the native pyarrow
write_dataset and pa.dataset.dataset mechanism.
What we've done currently is have our own `lance.write_dataset` and
`lance.dataset` interfaces which knows about the versioning. And if you use the
native arrow ones, it reads/writes an unversioned dataset.
It would be great if:
1. the arrow interfaces provided a way for custom data formats to provide their
own Arrow compliant reader/writer implementations, so we can delete our custom
interface and stick with native pyarrow interface.
2. the pyarrow interface can support custom kwargs like "version=5" or
"as_of=<timestamp>" or "version='latest'"
for reference, this is what our custom C++ dataset implementation looks like:
https://github.com/eto-ai/lance/blob/main/cpp/include/lance/arrow/dataset.h
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
