[
https://issues.apache.org/jira/browse/BEAM-8271?focusedWorklogId=372031&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372031
]
ASF GitHub Bot logged work on BEAM-8271:
----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Jan/20 00:13
Start Date: 15/Jan/20 00:13
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10595: [BEAM-8271]
Properly encode/decode StateGetRequest/Response continuation token
URL: https://github.com/apache/beam/pull/10595
This behavior is the same as Java, and corrects an inconsistency with
bytes/str that might have caused problems in python3.
This is a followup to #9056.
Note that this issue has its own Jira.
R: @robertwb
R: @udim
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/)
| --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/)
| --- | ---
See
[.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md)
for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 372031)
Remaining Estimate: 0h
Time Spent: 10m
> StateGetRequest/Response continuation_token should be string
> ------------------------------------------------------------
>
> Key: BEAM-8271
> URL: https://issues.apache.org/jira/browse/BEAM-8271
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
> Reporter: Chad Dombrova
> Assignee: Chad Dombrova
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I've been working on adding typing to the python code and I came across a
> discrepancy between regarding the type of the continuation token. The .proto
> defines it as bytes, but the code treats it as a string (i.e. unicode):
>
> {code:java}
> // A request to get state.
> message StateGetRequest {
> // (Optional) If specified, signals to the runner that the response
> // should resume from the following continuation token.
> //
> // If unspecified, signals to the runner that the response should start
> // from the beginning of the logical continuable stream.
> bytes continuation_token = 1;
> }
> // A response to get state representing a logical byte stream which can be
> // continued using the state API.
> message StateGetResponse {
> // (Optional) If specified, represents a token which can be used with the
> // state API to get the next chunk of this logical byte stream. The end of
> // the logical byte stream is signalled by this field being unset.
> bytes continuation_token = 1;
> // Represents a part of a logical byte stream. Elements within
> // the logical byte stream are encoded in the nested context and
> // concatenated together.
> bytes data = 2;
> }
> {code}
> From FnApiRunner.StateServicer:
> {code:python}
> def blocking_get(self, state_key, continuation_token=None):
> with self._lock:
> full_state = self._state[self._to_key(state_key)]
> if self._use_continuation_tokens:
> # The token is "nonce:index".
> if not continuation_token:
> token_base = 'token_%x' % len(self._continuations)
> self._continuations[token_base] = tuple(full_state)
> return b'', '%s:0' % token_base
> else:
> token_base, index = continuation_token.split(':')
> ix = int(index)
> full_state = self._continuations[token_base]
> if ix == len(full_state):
> return b'', None
> else:
> return full_state[ix], '%s:%d' % (token_base, ix + 1)
> else:
> assert not continuation_token
> return b''.join(full_state), None
> {code}
> This could be a problem in python3.
> All other id values are string, whereas bytes is reserved for data, so I
> think that the proto should be changed to string.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
