[
https://issues.apache.org/jira/browse/BEAM-7678?focusedWorklogId=289292&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289292
]
ASF GitHub Bot logged work on BEAM-7678:
----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Aug/19 00:52
Start Date: 06/Aug/19 00:52
Worklog Time Spent: 10m
Work Description: ecanzonieri commented on issue #9238: [BEAM-7678] Fixes
bug in output element_type generation in Kv PipelineVisitor
URL: https://github.com/apache/beam/pull/9238#issuecomment-518453454
I agree that we should have a unit test for this. In the case of the manual
test the Message class lives in a separate library and it's a very complex
class (several hundreds of lines of code). My guess is that for some reason the
inference is not able to work for that class.
I had spent some time trying to write a test doing exactly what you
suggested before submitting the pr. The problem is that I can't find an easy
way to create a type that would fail `infer_output_type`.
In my first attempt I had created a new class inside the `pipeline_test.py`
as follow:
```python
class TestOutputType(object):
def __init__(self, value):
self.value = value
```
I had created a stateful DoFn that would hit the Kv PipelineVisitor
(verified via debugger) but the `infer_output_type` was able to find the proper
type. In other words I wasn't able to get the unit test to fail with the older
code. Replacing the `TestOutputType` with the `Message` would correctly fail
the test before the fix. Is there any obvious way to create a type that will
fail inference?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 289292)
Time Spent: 50m (was: 40m)
> typehints with_output_types annotation doesn't work for stateful DoFn
> ----------------------------------------------------------------------
>
> Key: BEAM-7678
> URL: https://issues.apache.org/jira/browse/BEAM-7678
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.13.0
> Reporter: Enrico Canzonieri
> Priority: Minor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The output types typehints seem to be ignored when using a stateful DoFn, but
> the same typehint works perfectly when used without state. This issue
> prevents a custom Coder from being used because Beam will default to one of
> the {{FastCoders}} (I believe Pickle).
> Example code:
> {code}
> @typehints.with_output_types(Message)
> class StatefulDoFn(DoFn):
> COUNTER_STATE = BagStateSpec('counter', VarIntCoder())
> def process(self, element, counter=DoFn.StateParam(COUNTER_STATE)):
> (key, messages) = element
> newMessage = Message()
> return [newMessage]
> {code}
> The example code is just defining a stateful DoFn for python. The used runner
> is the Flink 1.6.4 portable runner.
> Finally, overriding {{infer_output_type}} to return a
> {{typehints.List[Message]}} solves the issue.
> Looking at the code, it seems to me that in
> [https://github.com/apache/beam/blob/v2.13.0/sdks/python/apache_beam/pipeline.py#L643]
> we do not take the typehints into account.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
