Hi Lien, > First time writing the email list, so please tell me if I'm doing this all wrong. Not at all! This is exactly the kind of question this list is for
I have a couple of questions that may help us debug: - Can you share the full stacktrace? - What version of Beam are you using? There were some changes to the way we use typehints in the most recent Beam release (2.21) that might be causing this [1]. If you're using 2.21 could you try reverting to the old behavior (call `apache_beam.typehints.disable_type_annotations()` before constructing the pipeline) to see if that helps? Thanks, Brian [1] https://beam.apache.org/blog/python-typing/ On Mon, Jun 8, 2020 at 4:15 AM Lien Michiels <[email protected]> wrote: > > Hi everyone, > > First time writing the email list, so please tell me if I'm doing this all > wrong. > > I'm building a streaming pipeline to be run on the DataflowRunner that > reads from PubSub and writes to BQ using the Python 3 SDK. > > I can get the pipeline started fine with the DirectRunner, but as soon as > I try to deploy to DataFlow it throws the following error: > > File > "/usr/local/lib/python3.7/site-packages/apache_beam/coders/coders.py", line > 221, in to_type_hint > raise NotImplementedError('BEAM-2717') > > I've tried narrowing down what exactly could be causing the issue and it > appears to be caused by the second step in my pipeline, which transforms > the bytes read from PubSub to my own internal Proto format: > > def parse_message_blobs(x: bytes) -> ActionWrapper: > action_wrapper = ActionWrapper() > action_wrapper.ParseFromString(x) > > return action_wrapper > > which is applied as a Map step. > > I've added typehints to all downstream steps as follows: > def partition_by_environment( > x: ActionWrapper, num_partitions: int, environments: List[str] > ) -> int: > > I'd really appreciate it if anyone could let me know what I'm doing wrong, > or what exactly is the issue this error is referring to. I read the JIRA > ticket, but did not understand how it is related to my issue here. > > Thanks! > Kind regards, > Lien >
