Maarten Breddels created ARROW-10709:
----------------------------------------
Summary: [Python] Difficult to make an efficient zero-copy file
reader in Python
Key: ARROW-10709
URL: https://issues.apache.org/jira/browse/ARROW-10709
Project: Apache Arrow
Issue Type: Improvement
Components: Python
Reporter: Maarten Breddels
There is an option to do efficient data transport using file.read_buffer()
using zero memory copies (benchmarking have confirmed that, very nice!).
However, file.read_buffer() when backed by a Python object (via PythonFile),
will call PythonFile.read() via PyReadableFile::Read. A 'normal' file.read()
that does memory copying, also calls the PythonFile.read() method, but only
allows for a bytes object (PyBytes_Check is used in PyReadableFile::Read).
This makes it hard to create 1 file object in Python land that supports normal
.read() (and thus needs to returns a bytes object) and to also support a
zero-copy route where .read() can return a memory view.
Possibly the strict check on PyBytes_Check can me lifted by also allowing
trying PyObject_GetBuffer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
