clintropolis commented on a change in pull request #11004:
URL: https://github.com/apache/druid/pull/11004#discussion_r596554833
##########
File path:
processing/src/main/java/org/apache/druid/segment/data/VSizeLongSerde.java
##########
@@ -413,9 +412,81 @@ public void close() throws IOException
}
}
+ /**
+ * Unpack bitpacked long values from an underlying contiguous memory block
+ */
public interface LongDeserializer
{
+ /**
+ * Unpack long value at the specified row index
+ */
long get(int index);
+
+ /**
+ * Unpack a contiguous vector of long values at the specified start index
of length and adjust them by the supplied
+ * delta base value.
+ */
+ default void getDelta(long[] out, int outPosition, int startIndex, int
length, long base)
+ {
+ for (int i = 0; i < length; i++) {
+ out[outPosition + i] = base + get(startIndex + i);
+ }
+ }
+
+ /**
+ * Unpack a non-contiguous vector of long values at the specified indexes
and adjust them by the supplied delta base
+ * value.
+ */
+ default int getDelta(long[] out, int outPosition, int[] indexes, int
length, int indexOffset, int limit, long base)
Review comment:
I did the first 8 bits so far without pushdown, the results are 1-2
millis slower.
without pushdown:
```
Benchmark
(distribution) (encoding) (filteredRowCountPercentage) (rows)
ColumnarLongsSelectRowsFromGeneratorBenchmark.selectRowsVectorized
uniform-1 lz4-auto 1.0 5000000
0.0 avgt 5 20409.851 ± 998.810 us/op
ColumnarLongsSelectRowsFromGeneratorBenchmark.selectRowsVectorized
uniform-2 lz4-auto 1.0 5000000
0.0 avgt 5 18935.144 ± 199.459 us/op
ColumnarLongsSelectRowsFromGeneratorBenchmark.selectRowsVectorized
uniform-4 lz4-auto 1.0 5000000
0.0 avgt 5 18668.448 ± 538.985 us/op
ColumnarLongsSelectRowsFromGeneratorBenchmark.selectRowsVectorized
uniform-8 lz4-auto 1.0 5000000
0.0 avgt 5 19010.981 ± 695.243 us/op
```
compared to with pushdown:
```
Benchmark
(distribution) (encoding) (filteredRowCountPercentage) (rows)
ColumnarLongsSelectRowsFromGeneratorBenchmark.selectRowsVectorized
uniform-1 lz4-auto 1.0 5000000
0.0 avgt 5 17892.362 ± 102.776 us/op
ColumnarLongsSelectRowsFromGeneratorBenchmark.selectRowsVectorized
uniform-2 lz4-auto 1.0 5000000
0.0 avgt 5 17796.103 ± 417.847 us/op
ColumnarLongsSelectRowsFromGeneratorBenchmark.selectRowsVectorized
uniform-4 lz4-auto 1.0 5000000
0.0 avgt 5 17879.496 ± 237.066 us/op
ColumnarLongsSelectRowsFromGeneratorBenchmark.selectRowsVectorized
uniform-8 lz4-auto 1.0 5000000
0.0 avgt 5 17508.260 ± 560.856 us/op
```
I'll run the rest, but based on the results so far, since only the first 4
deserializers implement `getTable`, I would be in favor of leaving the pushdown
in place since it isn't _that_ much extra complexity.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
