laskoviymishka commented on code in PR #1771:
URL: https://github.com/apache/iceberg-go/pull/1771#discussion_r3844843474
##########
table/scanner.go:
##########
@@ -206,6 +206,9 @@ func openManifest(io io.IO, manifest iceberg.ManifestFile,
if err != nil {
return nil, err
}
+ if !p {
Review Comment:
Now that we `continue` on `!p` here, the `p &&` in the `if p && m {` a few
lines down is dead, since `p` is always true by the time we reach it.
I'd collapse it to `if m {` so the control-flow intent reads cleanly;
otherwise the next reader has to stop and convince themselves `p` can't be
false at that gate.
##########
table/scanner_bench_test.go:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package table
+
+import (
+ "bytes"
+ "encoding/binary"
+ "fmt"
+ "testing"
+
+ "github.com/apache/iceberg-go"
+ iceio "github.com/apache/iceberg-go/io"
+)
+
+func BenchmarkOpenManifestPartitionRejectsMostEntries(b *testing.B) {
+ const (
+ entryCount = 1_000
+ rejectThrough = 950
+ )
+
+ spec := partitionedSpec()
+ schema := simpleSchema()
+ snapshotID := int64(1)
+ entries := make([]iceberg.ManifestEntry, entryCount)
+ for i := range entries {
+ value := int32(i)
+ builder, err := iceberg.NewDataFileBuilder(
+ spec,
+ iceberg.EntryContentData,
+ fmt.Sprintf("mem://default/table/data/file-%d.parquet",
i),
+ iceberg.ParquetFile,
+ map[int]any{1000: value},
+ nil,
+ nil,
+ 1,
+ 100,
+ )
+ if err != nil {
+ b.Fatal(err)
+ }
+
+ bound := make([]byte, 4)
+ binary.LittleEndian.PutUint32(bound, uint32(value))
+ entries[i] = iceberg.NewManifestEntry(
+ iceberg.EntryStatusADDED,
+ &snapshotID,
+ nil,
+ nil,
+ builder.
+ LowerBoundValues(map[int][]byte{1: bound}).
+ UpperBoundValues(map[int][]byte{1: bound}).
+ Build(),
+ )
+ }
+
+ manifestPath := "mem://default/table/metadata/manifest.avro"
+ var manifestBytes bytes.Buffer
+ manifest, err := iceberg.WriteManifest(
+ manifestPath,
+ &manifestBytes,
+ 2,
+ spec,
+ schema,
+ snapshotID,
+ entries,
+ )
+ if err != nil {
+ b.Fatal(err)
+ }
+
+ fs := iceio.NewMemFS()
+ if err := fs.WriteFile(manifestPath, manifestBytes.Bytes()); err != nil
{
+ b.Fatal(err)
+ }
+
+ partitionFilter := func(df iceberg.DataFile) (bool, error) {
+ return df.Partition()[1000].(int32) >= rejectThrough, nil
Review Comment:
One caveat on the reported percentages: this partition filter is a raw
map-lookup-plus-compare, but in production the partition evaluator runs through
`buildPartitionEvaluator` → `ExpressionEvaluator` → `GetPartitionRecord`, which
is a good bit heavier. So the absolute count of skipped metricsEval calls is
real, but the savings *fractions* will read larger here than in a real scan
where the denominator is bigger.
I'd either build the filter via `buildPartitionEvaluator` for an end-to-end
number, or drop a one-line comment noting the filter is deliberately simplified
and the percentages reflect metricsEval savings only. Either's fine, just worth
making explicit.
##########
table/scanner_internal_test.go:
##########
@@ -285,6 +285,99 @@ func
TestEqualityDeletePartitionKeyDistinguishesSignedZero(t *testing.T) {
assert.NotEqual(t, negative, positive)
}
+func TestOpenManifestShortCircuitsMetricsEvaluation(t *testing.T) {
+ spec := partitionedSpec()
+ schema := simpleSchema()
+ snapshotID := int64(1)
+
+ builder, err := iceberg.NewDataFileBuilder(
+ spec,
+ iceberg.EntryContentData,
+ "mem://default/table/data/file.parquet",
+ iceberg.ParquetFile,
+ map[int]any{1000: int32(7)},
+ nil,
+ nil,
+ 1,
+ 100,
+ )
+ require.NoError(t, err)
+
+ entry := iceberg.NewManifestEntry(
+ iceberg.EntryStatusADDED,
+ &snapshotID,
+ nil,
+ nil,
+ builder.Build(),
+ )
+ manifestPath := "mem://default/table/metadata/manifest.avro"
+ var manifestBytes bytes.Buffer
+ manifest, err := iceberg.WriteManifest(
+ manifestPath,
+ &manifestBytes,
+ 2,
+ spec,
+ schema,
+ snapshotID,
+ []iceberg.ManifestEntry{entry},
+ )
+ require.NoError(t, err)
+
+ fs := iceio.NewMemFS()
+ require.NoError(t, fs.WriteFile(manifestPath, manifestBytes.Bytes()))
+
+ tests := []struct {
+ name string
+ partitionMatches bool
+ metricsMatches bool
+ wantEvaluations []string
+ wantEntries int
+ }{
+ {
+ name: "partition rejection skips metrics",
+ partitionMatches: false,
+ metricsMatches: true,
+ wantEvaluations: []string{"partition"},
Review Comment:
This case proves metricsEval isn't *called* when the partition filter
rejects, which is exactly right. The more interesting behavior change, though,
is that a metricsEval *error* is now swallowed on that path, and nothing covers
it.
Before this PR, if the partition filter returned `(false, nil)` and
metricsEval would have errored, that error propagated; now it's skipped
entirely. That's almost certainly what we want since we're not going to read
those files anyway, but it's an untested contract shift. I'd add a case where
`partitionMatches` is false and the metrics closure returns an error, asserting
`openManifest` still returns a nil error. wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
