daniel-adam-tfs commented on PR #547:
URL: https://github.com/apache/arrow-go/pull/547#issuecomment-3800030963
By stealing the buffers and conditionally reusing the buffers based on the
encryption/compression I got a decent improvement mostly in the allocated
memory:
ReadInt32 benchmark:
```
goos: darwin
goarch: amd64
pkg: github.com/apache/arrow-go/v18/parquet/file
cpu: Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
│ bench/BenchmarkReadInt32.main.txt │
bench/BenchmarkReadInt32.opt4.txt │
│ sec/op │
sec/op vs base │
ReadInt32/V1Page-8 29.48m ± 2%
27.43m ± 2% -6.97% (p=0.001 n=7)
ReadInt32/V2Page-8 28.59m ± 1%
27.59m ± 6% ~ (p=0.053 n=7)
ReadInt32/V1PageSnappy-8 29.27m ± 1%
28.76m ± 2% -1.74% (p=0.001 n=7)
ReadInt32/V2PageSnappy-8 29.25m ± 2%
28.55m ± 1% -2.38% (p=0.001 n=7)
ReadInt32/V1PageEncrypted-8 30.39m ± 2%
28.49m ± 1% -6.25% (p=0.001 n=7)
ReadInt32/V2PageEncrypted-8 30.48m ± 1%
28.67m ± 2% -5.91% (p=0.001 n=7)
ReadInt32/V1PageSnappyEncrypted-8 30.42m ± 1%
29.60m ± 1% -2.71% (p=0.001 n=7)
ReadInt32/V2PageSnappyEncrypted-8 30.34m ± 1%
29.52m ± 2% -2.73% (p=0.001 n=7)
geomean 29.77m
28.57m -4.04%
│ bench/BenchmarkReadInt32.main.txt │
bench/BenchmarkReadInt32.opt4.txt │
│ B/op │
B/op vs base │
ReadInt32/V1Page-8 18.670Mi ± 0%
5.386Mi ± 0% -71.15% (p=0.001 n=7)
ReadInt32/V2Page-8 13.357Mi ± 0%
5.386Mi ± 0% -59.67% (p=0.001 n=7)
ReadInt32/V1PageSnappy-8 13.045Mi ± 0%
8.433Mi ± 0% -35.35% (p=0.001 n=7)
ReadInt32/V2PageSnappy-8 13.045Mi ± 0%
8.433Mi ± 0% -35.35% (p=0.001 n=7)
ReadInt32/V1PageEncrypted-8 22.241Mi ± 0%
8.957Mi ± 0% -59.73% (p=0.001 n=7)
ReadInt32/V2PageEncrypted-8 22.241Mi ± 0%
8.957Mi ± 0% -59.73% (p=0.001 n=7)
ReadInt32/V1PageSnappyEncrypted-8 16.30Mi ± 0%
11.69Mi ± 0% -28.28% (p=0.001 n=7)
ReadInt32/V2PageSnappyEncrypted-8 16.30Mi ± 0%
11.69Mi ± 0% -28.28% (p=0.001 n=7)
geomean 16.53Mi
8.305Mi -49.76%
│ bench/BenchmarkReadInt32.main.txt │
bench/BenchmarkReadInt32.opt4.txt │
│ allocs/op │
allocs/op vs base │
ReadInt32/V1Page-8 8.739k ± 0%
8.635k ± 0% -1.19% (p=0.001 n=7)
ReadInt32/V2Page-8 8.698k ± 0%
8.634k ± 0% -0.74% (p=0.001 n=7)
ReadInt32/V1PageSnappy-8 8.717k ± 0%
8.655k ± 0% -0.71% (p=0.001 n=7)
ReadInt32/V2PageSnappy-8 8.717k ± 0%
8.655k ± 0% -0.71% (p=0.001 n=7)
ReadInt32/V1PageEncrypted-8 10.100k ± 0%
9.996k ± 0% -1.03% (p=0.001 n=7)
ReadInt32/V2PageEncrypted-8 10.100k ± 0%
9.996k ± 0% -1.03% (p=0.001 n=7)
ReadInt32/V1PageSnappyEncrypted-8 10.08k ± 0%
10.02k ± 0% -0.61% (p=0.001 n=7)
ReadInt32/V2PageSnappyEncrypted-8 10.08k ± 0%
10.02k ± 0% -0.61% (p=0.001 n=7)
geomean 9.378k
9.301k -0.83%
```
ReadInt32Buffered benchmark:
```
goos: darwin
goarch: amd64
pkg: github.com/apache/arrow-go/v18/parquet/file
cpu: Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
│
bench/BenchmarkReadInt32Buffered.main.txt │
bench/BenchmarkReadInt32Buffered.opt4.txt │
│ sec/op
│ sec/op vs base │
ReadInt32Buffered/V1Page-8
29.11m ± 2% 27.53m ± 3% -5.43% (p=0.001 n=7)
ReadInt32Buffered/V2Page-8
28.58m ± 6% 27.59m ± 3% -3.45% (p=0.001 n=7)
ReadInt32Buffered/V1PageSnappy-8
29.06m ± 6% 28.66m ± 1% -1.37% (p=0.007 n=7)
ReadInt32Buffered/V2PageSnappy-8
29.01m ± 1% 28.45m ± 1% -1.92% (p=0.001 n=7)
ReadInt32Buffered/V1PageEncrypted-8
30.22m ± 1% 28.67m ± 1% -5.12% (p=0.001 n=7)
ReadInt32Buffered/V2PageEncrypted-8
30.34m ± 1% 28.63m ± 1% -5.64% (p=0.001 n=7)
ReadInt32Buffered/V1PageSnappyEncrypted-8
30.03m ± 3% 29.40m ± 1% -2.08% (p=0.038 n=7)
ReadInt32Buffered/V2PageSnappyEncrypted-8
29.83m ± 2% 29.88m ± 1% ~ (p=0.902 n=7)
geomean
29.52m 28.59m -3.13%
│
bench/BenchmarkReadInt32Buffered.main.txt │
bench/BenchmarkReadInt32Buffered.opt4.txt │
│ B/op
│ B/op vs base │
ReadInt32Buffered/V1Page-8
15.545Mi ± 0% 5.640Mi ± 0% -63.72% (p=0.001 n=7)
ReadInt32Buffered/V2Page-8
10.231Mi ± 0% 5.640Mi ± 0% -44.87% (p=0.001 n=7)
ReadInt32Buffered/V1PageSnappy-8
10.232Mi ± 0% 7.203Mi ± 0% -29.60% (p=0.001 n=7)
ReadInt32Buffered/V2PageSnappy-8
10.232Mi ± 0% 7.203Mi ± 0% -29.60% (p=0.001 n=7)
ReadInt32Buffered/V1PageEncrypted-8
19.115Mi ± 0% 9.211Mi ± 0% -51.81% (p=0.001 n=7)
ReadInt32Buffered/V2PageEncrypted-8
19.115Mi ± 0% 9.211Mi ± 0% -51.81% (p=0.001 n=7)
ReadInt32Buffered/V1PageSnappyEncrypted-8
13.49Mi ± 0% 10.46Mi ± 0% -22.45% (p=0.001 n=7)
ReadInt32Buffered/V2PageSnappyEncrypted-8
13.49Mi ± 0% 11.95Mi ± 0% -11.45% (p=0.001 n=7)
geomean
13.51Mi 8.042Mi -40.45%
│
bench/BenchmarkReadInt32Buffered.main.txt │
bench/BenchmarkReadInt32Buffered.opt4.txt │
│ allocs/op
│ allocs/op vs base │
ReadInt32Buffered/V1Page-8
8.738k ± 0% 8.655k ± 0% -0.95% (p=0.001 n=7)
ReadInt32Buffered/V2Page-8
8.696k ± 0% 8.655k ± 0% -0.47% (p=0.001 n=7)
ReadInt32Buffered/V1PageSnappy-8
8.716k ± 0% 8.665k ± 0% -0.59% (p=0.001 n=7)
ReadInt32Buffered/V2PageSnappy-8
8.716k ± 0% 8.665k ± 0% -0.59% (p=0.001 n=7)
ReadInt32Buffered/V1PageEncrypted-8
10.10k ± 0% 10.02k ± 0% -0.81% (p=0.001 n=7)
ReadInt32Buffered/V2PageEncrypted-8
10.10k ± 0% 10.02k ± 0% -0.82% (p=0.001 n=7)
ReadInt32Buffered/V1PageSnappyEncrypted-8
10.08k ± 0% 10.03k ± 0% -0.50% (p=0.001 n=7)
ReadInt32Buffered/V2PageSnappyEncrypted-8
10.08k ± 0% 10.04k ± 0% -0.40% (p=0.001 n=7)
geomean
9.377k 9.317k -0.64%
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
