DuanWeiFan opened a new issue, #41159: URL: https://github.com/apache/arrow/issues/41159
### Describe the enhancement requested ### Problem - While we are running some resource heavy process using Parquet FileWriter WriteBuffered(), we notice that WriteVlqInt() kept showing up on the top of the list for number of allocation when we pprof the program. This resulted in CPU profile having to be spent more on garbage collection. WriteBuffered() was writing at a speed of **320k rows/sec**. <img width="1399" alt="image" src="https://github.com/apache/arrow/assets/52736754/fcd044d8-b6a9-4996-a989-6863dfaafea7";> When looking into the function - Parquet BitWriter `WriteVlqInt`, we think it might be able to reuse the `buf [binary.MaxVarintLen64]byte` it is writing to instead of allocating a new one for every call. https://github.com/apache/arrow/blob/daa2efad5bae144072a9c46f6a8978dc6d7363f9/go/parquet/internal/utils/bit_writer.go#L164-L174 Similar to the `b.raw` being reused by `WriteAligned()`, we think `buf` could potentially be reused as well. https://github.com/apache/arrow/blob/daa2efad5bae144072a9c46f6a8978dc6d7363f9/go/parquet/internal/utils/bit_writer.go#L151-L160 ### Proposed Solution By making the minor code change as the below snippet, we saw a huge reduction on the number of allocation made by `WriteVlqInt()`. WriteBuffered() write speed was then improved to writing **650k rows/sec**, doubling the write rate we got before the change. If this make sense, we would like to submit a PR. ``` // BitWriter is a utility for writing values of specific bit widths to a stream // using a uint64 as a buffer to build up between flushing for efficiency. type BitWriter struct { wr WriterAtWithLen buffer uint64 byteoffset int bitoffset uint raw [8]byte buf [binary.MaxVarintLen64]byte } ... // WriteVlqInt writes v as a vlq encoded integer byte-aligned to the underlying writer // without buffering. func (b *BitWriter) WriteVlqInt(v uint64) bool { b.Flush(true) nbytes := binary.PutUvarint(b.buf[:], v) if _, err := b.wr.WriteAt(b.buf[:nbytes], int64(b.byteoffset)); err != nil { log.Println(err) return false } b.byteoffset += nbytes return true } ``` Huge thanks to @hhoughgg for finding and figuring out the solution!! ### Component(s) Go -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
