ShiKaiWi opened a new issue, #6276: URL: https://github.com/apache/arrow-rs/issues/6276
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** <!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] (This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*) --> In my use case where reading data from a big parquet file on disk, the snappy decoding costs too much cpu resources in my flamegraph. And digging into the codebase, I find that `resize` is used to initialize the buffer for the decoding destination buffer, but `resize` shows a bad performance when the expected size is large enough. Here is the code location about the `resize`: https://github.com/apache/arrow-rs/blob/25d39c13fc8937e86373821093d063202484ac00/parquet/src/compression.rs#L211-L226 And here is an example showing the bad performance of `resize` a vector: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=5321aaeda859ea9f664ae7952ade2fb6 And the example's output says: ``` init with macro, num_elems:10000, cost:7.74µs init with resize, num_elems:20000, cost:196.694µs init with set_len, num_elems:30000, cost:5.171µs ``` It proves `resize` a vector to 10000 costs too much more time than `vec!` macro or `set_len`. **Describe the solution you'd like** <!-- A clear and concise description of what you want to happen. --> In the example above, `set_len` shows a good performance, so I guess `resize` can be replaced with `set_len` if the capacity is enough: ```rust let new_len = offset + len; if output_buf.capacity() >= new_len { unsafe { output_buf.set_len(new_len); } } else { output_buf.resize(new_len, 0); } ``` Although `unsafe` block is introduced here, however it is safe considering: 1. the element of the vector is byte (no worry about the deconstruction); 2. the uninitialized content will be overwritten immediately. **Describe alternatives you've considered** <!-- A clear and concise description of any alternative solutions or features you've considered. --> **Additional context** <!-- Add any other context or screenshots about the feature request here. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
