raminqaf opened a new pull request, #29636:
URL: https://github.com/apache/flink/pull/29636
## What is the purpose of the change
`v['user']` and `CAST(v AS ROW<...>)` look up each object field by name on
every row. Objects of one stream mostly share their keys, so the field is
usually at the same index as in the previous row. This PR adds a
`VariantFieldGetter` that encodes the key once and checks the remembered index
before it searches.
```sql
SELECT v['user'], v['ts'], v['kind'], v['z'] FROM events
```
JMH, JDK 17, 4 lookups per row, 3 forks:
| Objects | `getField` today | With the
getter |
|--------------------------------------------|-------------------|-----------------|
| 13 keys, same shape in every row | 214 ns, 192-288 B | 44 ns, 96
B |
| 40 keys, binary search | 158 ns, 288 B | 44 ns, 96
B |
| 14 to 20 keys, field index moves every row | 296 ns, 192-288 B | 242 ns,
96 B |
The 96 B are the 4 returned views. Encoding the key once without the hint
gives no speedup, so the gain comes from the remembered index. The index is a
hint that is always verified, so one getter can be shared between threads.
Two known differences to `Variant#getField`:
- An object written before FLINK-41180 can hold the same key twice. For such
an object the getter can return the other of the two fields.
- The getter assumes a `BinaryVariant`, like the row writer and
`VariantSerializer`. A UDF that returns another `Variant` implementation now
fails with a `ClassCastException` on `v['k']` in the same projection.
## Brief change log
- New `@Internal` `VariantFieldGetter` in flink-core, one per key.
- `BinaryVariant#getField(String)` and the getter share one lookup. It
decodes the object header through new `BinaryVariantUtil` helpers instead of
`handleObject`, because the JIT does not reliably remove `handleObject`'s
capturing lambda on this path. `handleObject` is built on the same helpers.
- The generated code for `v['literal']` keeps one getter per call site.
The VARIANT to ROW and STRUCTURED cast keeps one per field.
## Verifying this change
This change added tests and can be verified as follows:
- `VariantFieldGetterTest` drives one getter through objects whose field
index stays, moves, runs out of range and misses.
- The shared lookup is covered by `BinaryVariantTest`, including binary
search, the UTF-16 fallback and nested objects. The consumers are covered by
`FromVariantConverterTest`, `CastFunctionITCase` and `VariantSemanticTest`,
which includes the key `special ' " \ \u0000` that now reaches the generated
code inside the serialized getter.
<details>
<summary>JMH harness</summary>
Rows are 64 objects, cycled. Master's `getField` was measured from a build
of the base commit.
```java
@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class FinalLookupBench {
@Param({"event", "wide", "drift"})
public String shape;
private Variant[] rows;
private VariantFieldGetter[] getters;
private int row;
@Setup
public void setup() {
rows = Shapes.rows(shape); // event: 13 keys; wide: 36 fillers + 4
looked-up keys; drift: event + random subset of 8 keys sorting ahead
getters = new VariantFieldGetter[Shapes.LOOKED_UP.length];
for (int i = 0; i < getters.length; i++) {
getters[i] = new VariantFieldGetter(Shapes.LOOKED_UP[i]); //
"user", "ts", "kind", "z"
}
}
@Benchmark
public int getField() {
final Variant v = rows[row = (row + 1) & 63];
int sum = 0;
for (String name : Shapes.LOOKED_UP) {
sum += v.getField(name).getInt();
}
return sum;
}
@Benchmark
public int getter() {
final Variant v = rows[row = (row + 1) & 63];
int sum = 0;
for (VariantFieldGetter getter : getters) {
sum += getter.getField(v).getInt();
}
return sum;
}
}
```
Run with `-f 3 -wi 5 -w 1s -i 5 -r 1s -prof gc`.
</details>
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes, VARIANT
field access by a literal key and the cast to ROW. A miss runs the same search
as `getField`.
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.311 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
