kerwin-zk commented on PR #9005:
URL:
https://github.com/apache/incubator-gluten/pull/9005#issuecomment-2735189246
> @kerwin-zk
>
> > The purpose of this buffer is to ensure that the size of each read
operation does not exceed this value.
>
> The current logic is to read `1 << 20 - kPaddedSize` each time. What if
the stream contains incomplete data ?Seems like the current logic cannot
guarantee that the read operation doesn't overpass the underlying buffer stream.
@marin-ma When reading large data in a stream, such as the serialized data
of a RowVector, it is read based on the actual length. If the actual length
exceeds the buffer, it will be read multiple times, and this method ensures
that the complete data can be read. [ void readBytes(uint8_t* bytes, int32_t
size)
{](https://github.com/apache/incubator-gluten/blob/a0eaf8245989058e1ad0f15e0edf2829e4b69136/cpp/velox/shuffle/GlutenByteStream.h#L152)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
