[
https://issues.apache.org/jira/browse/BEAM-7678?focusedWorklogId=289269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289269
]
ASF GitHub Bot logged work on BEAM-7678:
----------------------------------------
Author: ASF GitHub Bot
Created on: 05/Aug/19 23:56
Start Date: 05/Aug/19 23:56
Worklog Time Spent: 10m
Work Description: ecanzonieri commented on pull request #9238:
[BEAM-7678] Fixes bug in output element_type generation in Kv PipelineVisitor
URL: https://github.com/apache/beam/pull/9238#discussion_r310835165
##########
File path: sdks/python/apache_beam/pipeline.py
##########
@@ -641,8 +641,12 @@ def visit_transform(self, transform_node):
if len(transform_node.outputs) == 1:
# The runner often has expectations about the output types as well.
output, = transform_node.outputs.values()
- output.element_type = transform_node.transform.infer_output_type(
- pcoll.element_type)
+ output.element_type = (
Review comment:
that makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 289269)
Time Spent: 40m (was: 0.5h)
> typehints with_output_types annotation doesn't work for stateful DoFn
> ----------------------------------------------------------------------
>
> Key: BEAM-7678
> URL: https://issues.apache.org/jira/browse/BEAM-7678
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.13.0
> Reporter: Enrico Canzonieri
> Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The output types typehints seem to be ignored when using a stateful DoFn, but
> the same typehint works perfectly when used without state. This issue
> prevents a custom Coder from being used because Beam will default to one of
> the {{FastCoders}} (I believe Pickle).
> Example code:
> {code}
> @typehints.with_output_types(Message)
> class StatefulDoFn(DoFn):
> COUNTER_STATE = BagStateSpec('counter', VarIntCoder())
> def process(self, element, counter=DoFn.StateParam(COUNTER_STATE)):
> (key, messages) = element
> newMessage = Message()
> return [newMessage]
> {code}
> The example code is just defining a stateful DoFn for python. The used runner
> is the Flink 1.6.4 portable runner.
> Finally, overriding {{infer_output_type}} to return a
> {{typehints.List[Message]}} solves the issue.
> Looking at the code, it seems to me that in
> [https://github.com/apache/beam/blob/v2.13.0/sdks/python/apache_beam/pipeline.py#L643]
> we do not take the typehints into account.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
