[
https://issues.apache.org/jira/browse/BEAM-8298?focusedWorklogId=378688&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378688
]
ASF GitHub Bot logged work on BEAM-8298:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Jan/20 09:43
Start Date: 29/Jan/20 09:43
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10705: [BEAM-8298]
Implement side input caching.
URL: https://github.com/apache/beam/pull/10705#discussion_r372275421
##########
File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
##########
@@ -759,50 +759,57 @@ def __init__(self,
@contextlib.contextmanager
def process_instruction_id(self, bundle_id, cache_tokens):
- if getattr(self._context, 'cache_token', None) is not None:
+ if getattr(self._context, 'user_state_cache_token', None) is not None:
raise RuntimeError(
- 'Cache tokens already set to %s' % self._context.cache_token)
- # TODO Also handle cache tokens for side input, if present:
- # https://issues.apache.org/jira/browse/BEAM-8298
+ 'Cache tokens already set to %s'
+ % self._context.user_state_cache_token)
+ self._context.side_input_cache_tokens = {}
user_state_cache_token = None
for cache_token_struct in cache_tokens:
if cache_token_struct.HasField("user_state"):
# There should only be one user state token present
assert not user_state_cache_token
user_state_cache_token = cache_token_struct.token
+ elif cache_token_struct.HasField("side_input"):
+ self._context.side_input_cache_tokens[
+ cache_token_struct.side_input.transform_id,
+ cache_token_struct.side_input.side_input_id
+ ] = cache_token_struct.token
+ self._context.bundle_cache_token = bundle_id
Review comment:
Bundle caching for get requests is a good improvement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 378688)
Time Spent: 2h (was: 1h 50m)
> Implement state caching for side inputs
> ---------------------------------------
>
> Key: BEAM-8298
> URL: https://issues.apache.org/jira/browse/BEAM-8298
> Project: Beam
> Issue Type: Improvement
> Components: runner-core, sdk-py-harness
> Reporter: Maximilian Michels
> Assignee: Jing Chen
> Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Caching is currently only implemented for user state.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
