joechenrh commented on code in PR #485:
URL: https://github.com/apache/arrow-go/pull/485#discussion_r2309336089
##########
parquet/file/column_reader_types.gen.go.tmpl:
##########
@@ -45,8 +46,13 @@ func (cr *{{.Name}}ColumnChunkReader) Skip(nvalues int64)
(int64, error) {
// total is the number of rows that were read, valuesRead is the actual number
of physical values
// that were read excluding nulls
func (cr *{{.Name}}ColumnChunkReader) ReadBatch(batchSize int64, values
[]{{.name}}, defLvls, repLvls []int16) (total int64, valuesRead int, err error)
{
- return cr.readBatch(batchSize, defLvls, repLvls, func(start, len int64)
(int, error) {
- return
cr.curDecoder.(encoding.{{.Name}}Decoder).Decode(values[start:start+len])
- })
+ return cr.readBatch(batchSize, defLvls, repLvls,func(start, len int64) (int,
error) {
+ n, err :=
cr.curDecoder.(encoding.{{.Name}}Decoder).Decode(values[start:start+len])
+ if err == nil && cr.postProcess != nil {
+ cr.postProcess(values[start:start+len])
+ }
+ return n, err
+ },
+ )
Review Comment:
I found another problem with the current `serializedPageReader`, either
using memory allocator or buffer pool.
For example, suppose each data page contains 4 `ByteArray`, and we want to
read 16 data. We first read the data in 1st and 2nd page, store them in
`values[0:8]`. When reading the 3rd page, if the pool/allocator give us a
previously used buffer, the value in `values[0:8]` might be overwritten
(depends on the decoder we use).
So I add the logic here to make a copy of the buffer. But I don't think it's
the best way to solve this. Could you have some suggestions? @zeroshade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
