Fokko commented on PR #3876: URL: https://github.com/apache/parquet-java/pull/3876#issuecomment-6926059061
I reran the benchmark from my previous review on 6493ae2274276dd3f8e9de96dbab1265bb151a8b, using the same setup: 4M floats, uncompressed, no dictionary, single fork, 3 warmup and 5 measurement iterations (M4 Max, JDK 17). | ms/op | dim | V1 vector / list | V2 vector / list | |---|---|---|---| | write | 8 | 96.1 / 95.9 | 99.2 / 99.2 | | write | 128 | 71.6 / 77.2 | 81.4 / 78.4 | | write | 1024 | 72.9 / 77.4 | 78.2 / 78.7 | | readRecords | 8 | 71.2 / 79.0 | 58.0 / 59.7 | | readRecords | 128 | 79.0 / 67.0 | 60.3 / 51.5 | | readRecords | 1024 | 51.2 / 49.7 | 39.5 / 39.8 | | readPages | 8 | 3.48 / 3.40 | 3.36 / 3.38 | | readPages | 128 | 3.42 / 3.34 | 3.29 / 3.33 | | readPages | 1024 | 3.38 / 3.39 | 3.51 / 3.34 | With this commit the length check runs during record assembly rather than in `ParquetFileReader`, so `readRecords` is now the comparison that matters. In the single-fork run, dim=128 looked about 17% slower for vectors, so I reran `readRecords` with 3 forks: | readRecords ms/op (3 forks) | V1 vector / list | V2 vector / list | |---|---|---| | dim 8 | 74.0 ± 4.3 / 72.5 ± 4.8 | 67.4 ± 2.6 / 65.7 ± 2.1 | | dim 128 | 68.6 ± 1.1 / 65.8 ± 0.8 | 57.5 ± 4.2 / 58.2 ± 3.2 | That gap was a single-fork outlier. The largest difference left is about 4% at V1 dim=128, which doesn't show up for V2. So validating the lengths in the record reader doesn't add any meaningful overhead 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
