[
https://issues.apache.org/jira/browse/ARROW-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miles Granger updated ARROW-18163:
----------------------------------
Summary: [Python] registering new data formats (was: [python] registering
new data formats)
> [Python] registering new data formats
> -------------------------------------
>
> Key: ARROW-18163
> URL: https://issues.apache.org/jira/browse/ARROW-18163
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Python
> Affects Versions: 9.0.0
> Reporter: Chang She
> Priority: Major
>
> Context: we're creating a new data format for computer vision
> (https://github.com/eto-ai/lance) with a C++ core.
> We've implemented the integration so you can read Lance datasets into pyarrow
> like:
> ```python
> import lance
> import pyarrow.dataset as ds
> ds.dataset(uri, format=lance.LanceFileFormat())
> ```
> Would it possible to create a file format registry? like:
> ```python
> ds.register_file_format(
> ext='lance',
> format=lance.LanceFileFormat(),
> dataset=lance.FileSystemDataset
> )
> ```
> which would enable: `ds.dataset('/my/dataset.lance')` to execute successfully.
> The optional third argument would be to help expose format specific
> optimizations. e.g, Lance has much better random access performance so
> pushing limit/offset parameters to Lance allows for much faster paging,
> especially over deeply nested data and/or image blobs.
> Thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
