richardstartin opened a new pull request #7931:
URL: https://github.com/apache/pinot/pull/7931
Following on from #7930, it's possible to speed up building of V4 raw
indexes slightly (~10%) by using a compression buffer and only using memory
mapping for huge chunks. The size of the compression is fixed because the size
of the chunk buffer itself is fixed, so the potential for very large
compression buffers which exists with V3 and V4 does not apply here.
master
```
Benchmark (_chunkCompressionType)
(_distribution) (_maxChunkSize) (_records) Mode Cnt Score
Error Units
BenchmarkRawForwardIndexWriter.writeV4 LZ4
UNIFORM(1000,10000) 1048576 100000 ss 5 9857.203 ±
167.084 ms/op
BenchmarkRawForwardIndexWriter.writeV4:b LZ4
UNIFORM(1000,10000) 1048576 100000 ss 5 7781048545.000
#
BenchmarkRawForwardIndexWriter.writeV4:kb LZ4
UNIFORM(1000,10000) 1048576 100000 ss 5 7598680.000
#
BenchmarkRawForwardIndexWriter.writeV4:mb LZ4
UNIFORM(1000,10000) 1048576 100000 ss 5 7420.000
#
BenchmarkRawForwardIndexWriter.writeV4 LZ4
EXP(0.001) 1048576 100000 ss 5 1802.487 ± 112.871
ms/op
BenchmarkRawForwardIndexWriter.writeV4:b LZ4
EXP(0.001) 1048576 100000 ss 5 1416542875.000
#
BenchmarkRawForwardIndexWriter.writeV4:kb LZ4
EXP(0.001) 1048576 100000 ss 5 1383340.000
#
BenchmarkRawForwardIndexWriter.writeV4:mb LZ4
EXP(0.001) 1048576 100000 ss 5 1350.000
#
```
branch
```
Benchmark (_chunkCompressionType)
(_distribution) (_maxChunkSize) (_records) Mode Cnt Score
Error Units
BenchmarkRawForwardIndexWriter.writeV4 LZ4
UNIFORM(1000,10000) 1048576 100000 ss 5 8788.638 ±
256.238 ms/op
BenchmarkRawForwardIndexWriter.writeV4:b LZ4
UNIFORM(1000,10000) 1048576 100000 ss 5 7781048545.000
#
BenchmarkRawForwardIndexWriter.writeV4:kb LZ4
UNIFORM(1000,10000) 1048576 100000 ss 5 7598680.000
#
BenchmarkRawForwardIndexWriter.writeV4:mb LZ4
UNIFORM(1000,10000) 1048576 100000 ss 5 7420.000
#
BenchmarkRawForwardIndexWriter.writeV4 LZ4
EXP(0.001) 1048576 100000 ss 5 1625.317 ± 48.117
ms/op
BenchmarkRawForwardIndexWriter.writeV4:b LZ4
EXP(0.001) 1048576 100000 ss 5 1416542875.000
#
BenchmarkRawForwardIndexWriter.writeV4:kb LZ4
EXP(0.001) 1048576 100000 ss 5 1383340.000
#
BenchmarkRawForwardIndexWriter.writeV4:mb LZ4
EXP(0.001) 1048576 100000 ss 5 1350.000
#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
