kinolaev commented on issue #11648:
URL: https://github.com/apache/iceberg/issues/11648#issuecomment-4857849728
I wrote a simple benchmark to compare the performance of
`Deletes.toPositionIndexes` between the current version and the version in my
PR:
```java
@Fork(1)
@State(Scope.Benchmark)
@Warmup(iterations = 3)
@Measurement(iterations = 5)
@BenchmarkMode(Mode.SingleShotTime)
@Timeout(time = 10, timeUnit = TimeUnit.MINUTES)
public class ToPositionIndexesBenchmark {
@Param({"1000000"})
private int numDeletes;
@Param({"1", "10", "100", "1000", "10000", "100000", "1000000"})
private int numDataFiles;
private List<StructLike> deletes;
private String pathTemplate =
"s3://pretty-long-bucket-name/pretty-long-namespace-name/pretty-long-table-name/data/%07d-data.parquet";
@Setup
public void setupBenchmark() {
deletes = Lists.newArrayListWithExpectedSize(numDeletes);
int numDeletesPerDataFile = numDeletes / numDataFiles;
for (int index = 0; index < numDeletes; index++) {
String path = String.format(Locale.ROOT, pathTemplate, index /
numDeletesPerDataFile);
deletes.add(new PositionDeleteRow(path, index %
numDeletesPerDataFile));
}
}
@Benchmark
@Threads(1)
public void current(Blackhole blackhole) {
blackhole.consume(Deletes.toPositionIndexesMain(CloseableIterable.withNoopClose(deletes)));
}
@Benchmark
@Threads(1)
public void pr15714(Blackhole blackhole) {
blackhole.consume(Deletes.toPositionIndexes(CloseableIterable.withNoopClose(deletes)));
}
// ...
}
```
Here are the results:
```
Benchmark (numDataFiles) (numDeletes) Mode Cnt
Score Error Units
ToPositionIndexesBenchmark.current 1 1000000 ss 5
0.315 ± 0.017 s/op
ToPositionIndexesBenchmark.current 10 1000000 ss 5
0.322 ± 0.047 s/op
ToPositionIndexesBenchmark.current 100 1000000 ss 5
0.323 ± 0.035 s/op
ToPositionIndexesBenchmark.current 1000 1000000 ss 5
0.135 ± 0.003 s/op
ToPositionIndexesBenchmark.current 10000 1000000 ss 5
0.326 ± 0.016 s/op
ToPositionIndexesBenchmark.current 100000 1000000 ss 5
0.186 ± 0.060 s/op
ToPositionIndexesBenchmark.current 1000000 1000000 ss 5
0.803 ± 0.127 s/op
ToPositionIndexesBenchmark.pr15714 1 1000000 ss 5
0.033 ± 0.043 s/op
ToPositionIndexesBenchmark.pr15714 10 1000000 ss 5
0.034 ± 0.042 s/op
ToPositionIndexesBenchmark.pr15714 100 1000000 ss 5
0.025 ± 0.047 s/op
ToPositionIndexesBenchmark.pr15714 1000 1000000 ss 5
0.026 ± 0.047 s/op
ToPositionIndexesBenchmark.pr15714 10000 1000000 ss 5
0.050 ± 0.001 s/op
ToPositionIndexesBenchmark.pr15714 100000 1000000 ss 5
0.058 ± 0.007 s/op
ToPositionIndexesBenchmark.pr15714 1000000 1000000 ss 5
0.605 ± 0.159 s/op
```
When a delete file contains only one position per data file, the updated
version performs at the same level. Otherwise, the updated version is 3 to 12
times faster, depending on the ratio of positions per data file.
If this improvement is still not sufficient, we should consider skipping the
cache for position deletes entirely, just as is already done for DVs.
Additionally, it's worth to noting that the cache affects not only rewrite
operations but all read operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
