dianfu commented on a change in pull request #13230: URL: https://github.com/apache/flink/pull/13230#discussion_r490225135
########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. Review comment: ```suggestion Most transformations require a user-defined function as input to define the functionality of the transformation. The following describes different ways of defining user-defined functions. ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical Review comment: It seems that the description doesn't apply here. ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by +pickle seriallizer. For more details about the `Pickle Serialization`, please refer to [DataTypes]({% link dev/python/datastream-api-users-guide/data_types.md + %}#pickle-serialization). + +## Lambda Functions +As shown in the following example, all operations can also accept a lambda function to describe the operation: Review comment: ```suggestion As shown in the following example, all the transformations can also accept a lambda function to define the functionality of the transformation: ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not Review comment: ```suggestion <span class="label label-info">Note</span> In Python DataStream API, users can specify the output type information of the transformation explicitly. If not ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) Review comment: ```suggestion data_stream = env.from_collection([1, 2, 3, 4, 5], type_info=Types.INT()) ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by +pickle seriallizer. For more details about the `Pickle Serialization`, please refer to [DataTypes]({% link dev/python/datastream-api-users-guide/data_types.md + %}#pickle-serialization). + +## Lambda Functions Review comment: ```suggestion ## Lambda Function ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. Review comment: ```suggestion # Implementing MapFunction ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function Review comment: ```suggestion Different Function interfaces are provided for different transformations in the Python DataStream API. For example, `MapFunction` is provided for the `map` transformation, `FilterFunction` is provided for the `filter` transformation, etc. Users can implement the corresponding Function interface according to the type of the transformation. Take MapFunction for instance: ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by +pickle seriallizer. For more details about the `Pickle Serialization`, please refer to [DataTypes]({% link dev/python/datastream-api-users-guide/data_types.md + %}#pickle-serialization). + +## Lambda Functions +As shown in the following example, all operations can also accept a lambda function to describe the operation: +<p> +{% highlight python %} +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(lambda x: x + 1, output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> Operations ConnectedStream.map() and ConnectedStream.flat_map() are +only support implementing CoMapFunction and CoFlatMapFunction Interface. + +## Python Function +Users can directly pass a python function to the operation as well: +<p> +{% highlight python %} +def my_map_func(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) Review comment: ```suggestion data_stream = env.from_collection([1, 2, 3, 4, 5], type_info=Types.INT()) ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by +pickle seriallizer. For more details about the `Pickle Serialization`, please refer to [DataTypes]({% link dev/python/datastream-api-users-guide/data_types.md Review comment: ```suggestion the pickle seriallizer. For more details about the `Pickle Serialization`, please refer to [DataTypes]({% link dev/python/datastream-api-users-guide/data_types.md ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by Review comment: ```suggestion specified, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by +pickle seriallizer. For more details about the `Pickle Serialization`, please refer to [DataTypes]({% link dev/python/datastream-api-users-guide/data_types.md + %}#pickle-serialization). + +## Lambda Functions +As shown in the following example, all operations can also accept a lambda function to describe the operation: +<p> +{% highlight python %} +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(lambda x: x + 1, output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> Operations ConnectedStream.map() and ConnectedStream.flat_map() are Review comment: ```suggestion <span class="label label-info">Note</span> ConnectedStream.map() and ConnectedStream.flat_map() don't support lambda function and must accept `CoMapFunction` and `CoFlatMapFunction` separately. ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by +pickle seriallizer. For more details about the `Pickle Serialization`, please refer to [DataTypes]({% link dev/python/datastream-api-users-guide/data_types.md + %}#pickle-serialization). + +## Lambda Functions +As shown in the following example, all operations can also accept a lambda function to describe the operation: +<p> +{% highlight python %} +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(lambda x: x + 1, output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> Operations ConnectedStream.map() and ConnectedStream.flat_map() are +only support implementing CoMapFunction and CoFlatMapFunction Interface. + +## Python Function +Users can directly pass a python function to the operation as well: +<p> +{% highlight python %} +def my_map_func(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(my_map_func, output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> The arguements of the python function must be matched with the corresponding Review comment: this line seems unnecessary for me, isn't this obvious? ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by +pickle seriallizer. For more details about the `Pickle Serialization`, please refer to [DataTypes]({% link dev/python/datastream-api-users-guide/data_types.md + %}#pickle-serialization). + +## Lambda Functions +As shown in the following example, all operations can also accept a lambda function to describe the operation: +<p> +{% highlight python %} +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) Review comment: ```suggestion data_stream = env.from_collection([1, 2, 3, 4, 5], type_info=Types.INT()) ``` ########## File path: docs/dev/python/datastream-api-users-guide/operators.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section gives a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function +interface and pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implementing a MapFunction that returns plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by +pickle seriallizer. For more details about the `Pickle Serialization`, please refer to [DataTypes]({% link dev/python/datastream-api-users-guide/data_types.md + %}#pickle-serialization). + +## Lambda Functions +As shown in the following example, all operations can also accept a lambda function to describe the operation: +<p> +{% highlight python %} +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(lambda x: x + 1, output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> Operations ConnectedStream.map() and ConnectedStream.flat_map() are +only support implementing CoMapFunction and CoFlatMapFunction Interface. + +## Python Function +Users can directly pass a python function to the operation as well: Review comment: ```suggestion Users can also use Python function: ``` ########## File path: docs/dev/stream/operators/index.md ########## @@ -1152,6 +1418,62 @@ someStream.filter(...).slotSharingGroup("name") </table> </div> + +<div data-lang="python" markdown="1"> + +<br /> + +<table class="table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 20%">Transformation</th> + <th class="text-center">Description</th> + </tr> + </thead> + <tbody> + <tr> + <td>Start new chain</td> + <td> + <p>Begin a new chain, starting with this operator. The two Review comment: The section above the tab should also be updated for Python tab? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
