hoshimura commented on pull request #16735: URL: https://github.com/apache/beam/pull/16735#issuecomment-1031340508
@charlesccychen So, the current implementation will raise an error for files of size > MIN_WRITE_SIZE = 5 * 1024 * 1024 but < MAX_WRITE_SIZE = 5 * 1024 * 1024 * 1024. because the while case in put() of s3io.py will then be true for one iteration causing the resulting self.buffer to be of zero length. This has the implication that finish() will fail with a client error since no check was made for zero length data. I don't know if there has to be some record of that put has written data already so that an error in finish() will only be interpreted as an error for true zerolength files but not for files with a zerolength buffer when reaching finish() -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
