etseidl commented on code in PR #6244:
URL: https://github.com/apache/arrow-rs/pull/6244#discussion_r1716107728
##########
parquet/src/arrow/array_reader/fixed_len_byte_array.rs:
##########
@@ -165,57 +165,65 @@ impl ArrayReader for FixedLenByteArrayReader {
// TODO: An improvement might be to do this conversion on read
let array: ArrayRef = match &self.data_type {
ArrowType::Decimal128(p, s) => {
- let decimal = binary
- .iter()
- .map(|opt| Some(i128::from_be_bytes(sign_extend_be(opt?))))
- .collect::<Decimal128Array>()
+ let nulls = binary.nulls().cloned();
+ let decimal = binary.iter().map(|o| match o {
Review Comment:
Wow. I did a quick test with `Float16`, and was able to double the
performance again using `into_parts()` to deconstruct the
`FixedSizeBinaryArray`. It's pretty ugly though. I think it would take some
time to rework the array code to be able to do this more cleanly and robustly.
@alamb how do you feel about merging this as is and then working on the even
faster version as a follow-on? Then we could get some win for 53.0 and more in
54.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
