[GitHub] [beam] udim commented on a change in pull request #11070: [BEAM-8280] Blog post: Python typing changes
udim commented on a change in pull request #11070: URL: https://github.com/apache/beam/pull/11070#discussion_r430617100 ## File path: website/src/_posts/2020-03-06-python-typing.md ## @@ -0,0 +1,117 @@ +--- +layout: post +title: "Python SDK Typing Changes" +date: 2020-03-06 00:00:01 -0800 +excerpt_separator: +categories: blog python typing +authors: + - chadrik + - udim + +--- + + +TODO excerpt + + + +Python supports type annotations on functions (PEP 484). Static type checkers, +such as mypy, are used to verify adherence to these types. +For example: +```py +def f(v: int) -> int: + return v[0] +``` +Running mypy on the above code will give the error: +`Value of type "int" is not indexable`. + +We've recently made changes to Beam in 2 areas: + +Adding type hints throughout Beam. TODO expand + +Second, we've added support for Python 3 type annotations. This allows SDK +users to specify a DoFn's type hints in one place. +We've also expanded Beam's support of `typing` module types. + +For more background see: +[Ensuring Python Type Safety](https://beam.apache.org/documentation/sdks/python-type-safety/). + +# Beam Is Typed + +TODO + +# New Ways to Annotate + +## Python 3 Syntax Annotations + +Coming in Beam 2.21 (BEAM-8280), you will be able to use Python annotation +syntax to specify input and output types. + +For example, this new form: +```py +class MyDoFn(beam.DoFn): + def process(self, element: int) -> typing.Text: +yield str(element) +``` +is equivalent to this: +```py +@beam.typehints.with_input_types(int) +@beam.typehints.with_output_types(typing.Text) +class MyDoFn(beam.DoFn): + def process(self, element): +yield str(element) +``` + +One of the advantages of the new form is that you may already be using it +in tandem with a static type checker such as mypy, thus getting additional +type checking for free. + +This feature will be enabled by default, and there will be 2 mechanisms in +place to disable it: +1. Calling `apache_beam.typehints.disable_type_annotations()` before pipeline +construction will disable the new feature completely. +1. Decorating a function with `@apache_beam.typehints.no_annotations` will +tell Beam to ignore annotations for it. + +Uses of Beam's `with_input_type`, `with_output_type` methods and decorators will +still work and take precedence over annotations. + +Sidebar: + +> You might ask: couldn't we use mypy to type check Beam pipelines? The main issue +is that such a tool would have to understand type relations between +pipeline graph nodes, e.g., that the type of element passed to a transform +should be consistent with its annotated input type. Review comment: Meanwhile, we will start working on adding type checking to PCollection types in the coming weeks (I don't have the details yet). The idea is to type check things like this within Beam as well: ``` def expand(pcoll: PCollection[str]) -> PCollection[Tuple[str, int]] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] udim commented on a change in pull request #11070: [BEAM-8280] Blog post: Python typing changes
udim commented on a change in pull request #11070: URL: https://github.com/apache/beam/pull/11070#discussion_r428996230 ## File path: website/src/_posts/2020-03-06-python-typing.md ## @@ -0,0 +1,117 @@ +--- +layout: post +title: "Python SDK Typing Changes" +date: 2020-03-06 00:00:01 -0800 +excerpt_separator: +categories: blog python typing +authors: + - chadrik + - udim + +--- + + +TODO excerpt + + + +Python supports type annotations on functions (PEP 484). Static type checkers, +such as mypy, are used to verify adherence to these types. +For example: +```py +def f(v: int) -> int: + return v[0] +``` +Running mypy on the above code will give the error: +`Value of type "int" is not indexable`. + +We've recently made changes to Beam in 2 areas: + +Adding type hints throughout Beam. TODO expand + +Second, we've added support for Python 3 type annotations. This allows SDK +users to specify a DoFn's type hints in one place. +We've also expanded Beam's support of `typing` module types. + +For more background see: +[Ensuring Python Type Safety](https://beam.apache.org/documentation/sdks/python-type-safety/). + +# Beam Is Typed + +TODO + +# New Ways to Annotate + +## Python 3 Syntax Annotations + +Coming in Beam 2.21 (BEAM-8280), you will be able to use Python annotation +syntax to specify input and output types. + +For example, this new form: +```py +class MyDoFn(beam.DoFn): + def process(self, element: int) -> typing.Text: +yield str(element) +``` +is equivalent to this: +```py +@beam.typehints.with_input_types(int) +@beam.typehints.with_output_types(typing.Text) +class MyDoFn(beam.DoFn): + def process(self, element): +yield str(element) +``` + +One of the advantages of the new form is that you may already be using it +in tandem with a static type checker such as mypy, thus getting additional +type checking for free. + +This feature will be enabled by default, and there will be 2 mechanisms in +place to disable it: +1. Calling `apache_beam.typehints.disable_type_annotations()` before pipeline +construction will disable the new feature completely. +1. Decorating a function with `@apache_beam.typehints.no_annotations` will +tell Beam to ignore annotations for it. + +Uses of Beam's `with_input_type`, `with_output_type` methods and decorators will +still work and take precedence over annotations. + +Sidebar: + +> You might ask: couldn't we use mypy to type check Beam pipelines? The main issue +is that such a tool would have to understand type relations between +pipeline graph nodes, e.g., that the type of element passed to a transform +should be consistent with its annotated input type. Review comment: I'm thinking of rephrasing this to just mention dynamically generate pipelines. As an aside, do you think that adding types to PCollections (such as PCollection[Tuple[K, Iterable[V]]]) would obviate the need for a plugin? ## File path: website/src/_posts/2020-03-06-python-typing.md ## @@ -0,0 +1,117 @@ +--- +layout: post +title: "Python SDK Typing Changes" +date: 2020-03-06 00:00:01 -0800 +excerpt_separator: +categories: blog python typing +authors: + - chadrik + - udim + +--- + + +TODO excerpt + + + +Python supports type annotations on functions (PEP 484). Static type checkers, +such as mypy, are used to verify adherence to these types. +For example: +```py +def f(v: int) -> int: + return v[0] +``` +Running mypy on the above code will give the error: +`Value of type "int" is not indexable`. + +We've recently made changes to Beam in 2 areas: + +Adding type hints throughout Beam. TODO expand + +Second, we've added support for Python 3 type annotations. This allows SDK +users to specify a DoFn's type hints in one place. +We've also expanded Beam's support of `typing` module types. + +For more background see: +[Ensuring Python Type Safety](https://beam.apache.org/documentation/sdks/python-type-safety/). + +# Beam Is Typed + +TODO + +# New Ways to Annotate + +## Python 3 Syntax Annotations + +Coming in Beam 2.21 (BEAM-8280), you will be able to use Python annotation +syntax to specify input and output types. + +For example, this new form: +```py +class MyDoFn(beam.DoFn): + def process(self, element: int) -> typing.Text: +yield str(element) +``` +is equivalent to this: +```py +@beam.typehints.with_input_types(int) +@beam.typehints.with_output_types(typing.Text) +class MyDoFn(beam.DoFn): + def process(self, element): +yield str(element) +``` + +One of the advantages of the new form is that you may already be using it +in tandem with a static type checker such as mypy, thus getting additional +type checking for free. + +This feature will be enabled by default, and there will be 2 mechanisms in +place to disable it: +1. Calling `apache_beam.typehints.disable_type_annotations()` before pipeline +construction will disable the new feature completely. +1. Decorating a function with `@apache_beam.typehints.no_annotations` will +tell
[GitHub] [beam] udim commented on a change in pull request #11070: [BEAM-8280] Blog post: Python typing changes
udim commented on a change in pull request #11070: URL: https://github.com/apache/beam/pull/11070#discussion_r428993726 ## File path: website/src/_posts/2020-03-06-python-typing.md ## @@ -0,0 +1,117 @@ +--- +layout: post +title: "Python SDK Typing Changes" +date: 2020-03-06 00:00:01 -0800 +excerpt_separator: +categories: blog python typing +authors: + - chadrik + - udim + +--- + + +TODO excerpt + + + +Python supports type annotations on functions (PEP 484). Static type checkers, +such as mypy, are used to verify adherence to these types. +For example: +```py +def f(v: int) -> int: + return v[0] +``` +Running mypy on the above code will give the error: +`Value of type "int" is not indexable`. + +We've recently made changes to Beam in 2 areas: + +Adding type hints throughout Beam. TODO expand + +Second, we've added support for Python 3 type annotations. This allows SDK +users to specify a DoFn's type hints in one place. +We've also expanded Beam's support of `typing` module types. + +For more background see: +[Ensuring Python Type Safety](https://beam.apache.org/documentation/sdks/python-type-safety/). + +# Beam Is Typed + +TODO + +# New Ways to Annotate + +## Python 3 Syntax Annotations + +Coming in Beam 2.21 (BEAM-8280), you will be able to use Python annotation +syntax to specify input and output types. + +For example, this new form: +```py +class MyDoFn(beam.DoFn): + def process(self, element: int) -> typing.Text: +yield str(element) +``` +is equivalent to this: +```py +@beam.typehints.with_input_types(int) +@beam.typehints.with_output_types(typing.Text) +class MyDoFn(beam.DoFn): + def process(self, element): +yield str(element) +``` + +One of the advantages of the new form is that you may already be using it +in tandem with a static type checker such as mypy, thus getting additional +type checking for free. Review comment: Changed. Runtime is a relative term. :) In Beam I like to say that we have static type checking during pipeline construction, and runtime type checking when the pipeline is running (the latter is off by default). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] udim commented on a change in pull request #11070: [BEAM-8280] Blog post: Python typing changes
udim commented on a change in pull request #11070: URL: https://github.com/apache/beam/pull/11070#discussion_r421718341 ## File path: website/src/_posts/2020-03-06-python-typing.md ## @@ -0,0 +1,117 @@ +--- +layout: post +title: "Python SDK Typing Changes" +date: 2020-03-06 00:00:01 -0800 +excerpt_separator: +categories: blog python typing +authors: + - chadrik + - udim + +--- + + +TODO excerpt + + + +Python supports type annotations on functions (PEP 484). Static type checkers, +such as mypy, are used to verify adherence to these types. +For example: +```py +def f(v: int) -> int: + return v[0] +``` +Running mypy on the above code will give the error: +`Value of type "int" is not indexable`. + +We've recently made changes to Beam in 2 areas: + +Adding type hints throughout Beam. TODO expand + +Second, we've added support for Python 3 type annotations. This allows SDK +users to specify a DoFn's type hints in one place. +We've also expanded Beam's support of `typing` module types. + +For more background see: +[Ensuring Python Type Safety](https://beam.apache.org/documentation/sdks/python-type-safety/). + +# Beam Is Typed + +TODO Review comment: Yes, that works This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
