Wes McKinney created ARROW-228:
----------------------------------
Summary: [Python] Create an Arrow-cpp-compatible interface for
reading bytes from Python file-like objects
Key: ARROW-228
URL: https://issues.apache.org/jira/browse/ARROW-228
Project: Apache Arrow
Issue Type: New Feature
Components: Python
Reporter: Wes McKinney
Assignee: Wes McKinney
In practice, IO interfaces in PyArrow will need to be bidirectional
- Exposing internal IO interfaces written purely in C++ to Python users as
file-like objects
- Exposing Python file-like objects to the C++ IO subsystem
To do this efficiently, we may want to introduce an arrow::Buffer subclass that
manages the lifetime of a PyBytes object in a GIL-safe way (i.e., on
destruction, the GIL is acquired and the object's refcount is decremented). We
can still implement a Read method that copies bytes into some other buffer,
after which the PyBytes is immediately destroyed.
Outside of these byte buffer management issues, wrapping a file-like object
(having read() -> bytes, seek(), tell(), and other basic file methods) is
fairly straightforward, and will allow any of the current or upcoming IO
adapters to read either from native classes (file system, HDFS, etc.) or
arbitrary Python streams.
To give a concrete example: consider the output of a GET http request -- this
can be put in a {{io.BytesIO}} object and then treated as a first class citizen
alongside the native (C++) IO classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
