chadrik commented on a change in pull request #11022: [BEAM-7746] Resolve
typing issues in filesystem
URL: https://github.com/apache/beam/pull/11022#discussion_r387300879
##########
File path: sdks/python/apache_beam/io/filesystemio.py
##########
@@ -175,10 +190,31 @@ def readall(self):
res.append(data)
return b''.join(res)
+ @classmethod
+ def create_buffered(cls, # type: Type[DownloaderStreamT]
+ downloader, # type: Downloader
+ read_buffer_size=io.DEFAULT_BUFFER_SIZE, mode='rb'):
+ # type: (...) -> BinaryIO
+ # In the python3 typeshed, io.BufferedReader and io.BufferedWriter do not
+ # inherit from typing.BinaryIO, and BinaryIO is not a Protocol, so a class
+ # must inherit from it. We could roll our own BinaryIO Protocol, but the
+ # stubs for io.Buffered* do not have the required 'mode' or 'name' attrs to
+ # meet the protocol (the classes seem to expose 'mode' and 'name'
+ # conditionally, if their underlying io.RawIOBase possess the attributes).
+ # Thus, it is necessary to cast these types to BinaryIO either way.
+ return cast(
+ BinaryIO,
+ io.BufferedReader(
+ cls(downloader, read_buffer_size=read_buffer_size, mode=mode),
+ buffer_size=read_buffer_size))
Review comment:
should I create separate args for the buffer size of the `DownloaderStream`
and the `io.BufferedReader`? The treatment of these varies between
filesystems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
