[jira] [Commented] (BEAM-8157) Key encoding for state requests is not consistent across SDKs
[ https://issues.apache.org/jira/browse/BEAM-8157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936899#comment-16936899 ] Maximilian Michels commented on BEAM-8157: -- It does have a potential issues for portable Java pipelines. The potential number of effected users should be small though. > Key encoding for state requests is not consistent across SDKs > - > > Key: BEAM-8157 > URL: https://issues.apache.org/jira/browse/BEAM-8157 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.13.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.17.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > The Flink runner requires the internal key to be encoded without a length > prefix (OUTER context). The user state request handler exposes a serialized > version of the key to the Runner. This key is encoded with the NESTED context > which may add a length prefix. We need to convert it to OUTER context to > match the Flink runner's key encoding. > So far this has not caused the Flink Runner to behave incorrectly. However, > with the upcoming support for Flink 1.9, the state backend will not accept > requests for keys not part of any key group/partition of the operator. This > is very likely to happen with the encoding not being consistent. > **NOTE** This is only applicable to the Java SDK, as the Python SDK uses > OUTER encoding for the key in state requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8157) Key encoding for state requests is not consistent across SDKs
[ https://issues.apache.org/jira/browse/BEAM-8157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936835#comment-16936835 ] Thomas Weise commented on BEAM-8157: I would probably not backport since 2.16 doesn't have this issue (it will only become relevant with Flink 1.9). > Key encoding for state requests is not consistent across SDKs > - > > Key: BEAM-8157 > URL: https://issues.apache.org/jira/browse/BEAM-8157 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.13.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.17.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > The Flink runner requires the internal key to be encoded without a length > prefix (OUTER context). The user state request handler exposes a serialized > version of the key to the Runner. This key is encoded with the NESTED context > which may add a length prefix. We need to convert it to OUTER context to > match the Flink runner's key encoding. > So far this has not caused the Flink Runner to behave incorrectly. However, > with the upcoming support for Flink 1.9, the state backend will not accept > requests for keys not part of any key group/partition of the operator. This > is very likely to happen with the encoding not being consistent. > **NOTE** This is only applicable to the Java SDK, as the Python SDK uses > OUTER encoding for the key in state requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8157) Key encoding for state requests is not consistent across SDKs
[
https://issues.apache.org/jira/browse/BEAM-8157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933836#comment-16933836
]
Maximilian Michels commented on BEAM-8157:
--
To conclude the investigation here, this only was a problem with the portable
Java SDK. The Python SDK uses {{OUTER}} context by default.
We should make the key encoding consistent. There is a good argument for using
the {{NESTED}} context, as this also gets rid of workaround for removing the
length prefix from key coders. So this requires
1. Updating the Python SDK to use {{NESTED}} context for state requests
2. Updating the Flink Runner to use {{NESTED}} context for the key encoding
> Key encoding for state requests is not consistent across SDKs
> -
>
> Key: BEAM-8157
> URL: https://issues.apache.org/jira/browse/BEAM-8157
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.13.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.17.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> The Flink runner requires the internal key to be encoded without a length
> prefix (OUTER context). The user state request handler exposes a serialized
> version of the key to the Runner. This key is encoded with the NESTED context
> which may add a length prefix. We need to convert it to OUTER context to
> match the Flink runner's key encoding.
> So far this has not caused the Flink Runner to behave incorrectly. However,
> with the upcoming support for Flink 1.9, the state backend will not accept
> requests for keys not part of any key group/partition of the operator. This
> is very likely to happen with the encoding not being consistent.
> **NOTE** This is only applicable to the Java SDK, as the Python SDK uses
> OUTER encoding for the key in state requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
