[
https://issues.apache.org/jira/browse/ARROW-18275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633917#comment-17633917
]
Chang She commented on ARROW-18275:
-----------------------------------
Yeah, it would largely be sufficient. I created this issue after some
previous discussions with Weston. For example, when I created ARROW-18275,
I was not aware that Dataset wasn’t meant to be an extension point.
But basically, the goal is the same:
1. Custom formats can tell pa.dataset.dataset to use the correct FileFormat
based on the filename / extension (instead of requiring the user to pass a
custom FileFormat subclass instance every time).
2. Custom formats can provide custom read/write behavior to handle:
- more complex metadata (eg versioning)
- more complex scan options (eg offset, limit)
- (planned) indexes
Currently we’ve extended Arrow FilesystemDataset/Scanner in C++ then
wrapped it in a custom cython class inheriting from pyarrow Dataset so the
resulting user API is still fully Arrow compatible — but the user has to do
“lance.dataset” and “lance.write_dataset” instead of just being able to go
through native pyarrow APIs.
Thanks!
On Mon, Nov 14, 2022 at 3:11 AM Miles Granger (Jira) <[email protected]>
> [Python] Allow custom reader/writer implementation for arrow dataset
> read/write path
> ------------------------------------------------------------------------------------
>
> Key: ARROW-18275
> URL: https://issues.apache.org/jira/browse/ARROW-18275
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Python
> Affects Versions: 10.0.0
> Reporter: Chang She
> Priority: Major
> Labels: Python
>
> We're implementing a "versionable" data format where the read/write path has
> some metadata handling which we currently can't plug into the native pyarrow
> write_dataset and pa.dataset.dataset mechanism.
> What we've done currently is have our own `lance.write_dataset` and
> `lance.dataset` interfaces which knows about the versioning. And if you use
> the native arrow ones, it reads/writes an unversioned dataset.
> It would be great if:
> 1. the arrow interfaces provided a way for custom data formats to provide
> their own Arrow compliant reader/writer implementations, so we can delete our
> custom interface and stick with native pyarrow interface.
> 2. the pyarrow interface can support custom kwargs like "version=5" or
> "as_of=<timestamp>" or "version='latest'"
> for reference, this is what our custom C++ dataset implementation looks like:
> https://github.com/eto-ai/lance/blob/main/cpp/include/lance/arrow/dataset.h
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
