SEPURI-SAI-KRISHNA commented on PR #29268:
URL: https://github.com/apache/flink/pull/29268#issuecomment-5798304052
Not a maintainer, and I'm late, this is already merged. Noticed one thing
while reading it.
Dropping `stream.keyBy(...)` also drops the key type check that
`KeyedStream` does. `OneInputStateTransformation#keyBy` only extracts the key
type, so that was the only place it happened.
Before, a `byte[]` key failed at build time:
```
InvalidProgramException: Type byte[] cannot be used as key. Contained
UNSUPPORTED key types: byte[]
```
Now that job builds, runs and writes a savepoint. Same for an enum key and a
POJO without `hashCode`. A `String` key behaves the same as before. I checked
this through `SavepointWriter` with only `StateBootstrapTransformation` swapped
between the two versions.
The partitioner falls back to identity `hashCode`, so equal keys don't share
a key group: two `byte[]{1, 2, 3}` give key groups 48 and 39 at maxParallelism
128. A job restoring that savepoint would reject the key type in its own
`keyBy` anyway, so nothing breaks silently downstream, you just find out much
later instead of up front.
I'm happy to open a JIRA and a small PR putting the check back next to the
new `PartitionTransformation`, with a case in
`StateBootstrapTransformationTest`, if you think it's worth it.
Unrelated question: `StreamOperatorContextBuilder` passes a managed memory
fraction of 1.0, and the bootstrap config sets `STATE_BACKEND` to 1.0. That
meant 1.0 of the private 64MB before, now it's 1.0 of the slot's managed
memory. Is that intended when other operators share the slot?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
