ulekarsiddhant0-boop opened a new pull request, #3410: URL: https://github.com/apache/fory/pull/3410
Motivation Currently, the Python Buffer implementation assumes that the input data always supports the len() function. When a user tries to initialize a Buffer with a stream-like object (such as io.BytesIO), it raises a TypeError. This PR enables Buffer to handle stream-like objects and provides a method to read data from them incrementally. Modifications Buffer Initialization: Updated Buffer.__init__ in buffer.pyx to check for the existence of a .read() method. If detected, it bypasses the len() check to prevent crashes with stream-like objects. New Method read_from_stream: Added a Cython method to allow reading a specific number of bytes from a Python stream into the underlying C++ CBuffer. Memory Safety: Implemented a mechanism to store a reference to the data chunks read from the stream (using self.data_chunk), ensuring the memory remains valid for the lifetime of the Buffer object and preventing dangling pointers in the C++ layer. Verification Build: Successfully verified the changes using bazel build //:serialization on Windows. Compilation: Confirmed that Cython correctly generates the C++ source and that the MSVC compiler handles the CBuffer interface without errors. Manual Test: Verified that io.BytesIO can now be passed to the Buffer without triggering a TypeError. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
