daniel-adam-tfs commented on issue #540:
URL: https://github.com/apache/arrow-go/issues/540#issuecomment-3450599018
> > However, the call to io.CopyN might trigger a rellocation
>
> So i think we can change to `io.ReadFull`? Then there is no reallocation:
>
> b := bytes.NewBuffer(p.decompressBuffer.Bytes()[:lenCompressed])
> if n, err := io.ReadFull(rd, b); err != nil || n != lenCompressed {
> return nil, err
> }
Yeah, that version would work, without the weird extra allocation size.
Though the behavior between io.ReadFull and io.CopyN is different - one single
read vs. multiple reads in a loop. @zeroshade was there a reason why io.CopyN
was used instead of io.ReadFull?
Also, another idea I just had - with no compression I could read directly to
the output buffer, right? For the nocodec decoder the Decode method is just a
copy:
https://github.com/apache/arrow-go/blob/6e2e50b133a959916acc66bead16f5c7285ba0b1/parquet/compress/compress.go#L131-L136
So I instead of reading to p.decompressBuffer and then copying to buf, I
could just read directly to buf in this case and avoid the intermediate buffer
totally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
