pitrou commented on code in PR #13264:
URL: https://github.com/apache/arrow/pull/13264#discussion_r885582892
##########
python/pyarrow/io.pxi:
##########
@@ -436,14 +440,22 @@ cdef class NativeFile(_Weakrefable):
def read1(self, nbytes=None):
"""Read and return up to n bytes.
- Alias for read, needed to match the BufferedIOBase interface.
+ A short result does not imply that EOF is imminent.
Parameters
----------
nbytes : int
The maximum number of bytes to read.
"""
- return self.read(nbytes=None)
+ if nbytes is None:
+ # The expectation when passing `nbytes=None` is not to read the
+ # entire file but to issue a single underlying read call up to
+ # a reasonable size (the use case being to read a bufferable
+ # amount of bytes, such as with io.TextIOWrapper).
+ nbytes = self._default_chunk_size
+ else:
+ nbytes = min(nbytes, self._default_chunk_size)
Review Comment:
Hmm, you're right that it may not be necessary, Python's IO stack is happily
letting you read1() large sizes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
