Repository: incubator-beam Updated Branches: refs/heads/python-sdk 513835487 -> 2b8d9f704
Update README to reflect Dataflow to Apache Beam migration Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/97038f32 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/97038f32 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/97038f32 Branch: refs/heads/python-sdk Commit: 97038f320fc0ec2bd774c5b33bd68484bb6f78c3 Parents: 5138354 Author: Frances Perry <[email protected]> Authored: Thu Oct 20 15:05:45 2016 -0700 Committer: Frances Perry <[email protected]> Committed: Thu Oct 20 15:09:03 2016 -0700 ---------------------------------------------------------------------- sdks/python/README.md | 287 +++++++++++++-------------------------------- 1 file changed, 79 insertions(+), 208 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/97038f32/sdks/python/README.md ---------------------------------------------------------------------- diff --git a/sdks/python/README.md b/sdks/python/README.md index e003aab..d8f3d89 100644 --- a/sdks/python/README.md +++ b/sdks/python/README.md @@ -16,31 +16,22 @@ specific language governing permissions and limitations under the License. --> -== This page is currently being updated. == +# Apache Beam (incubating) - Python SDK -# Cloud Dataflow SDK for Python +[Apache Beam](http://beam.incubator.apache.org) is a unified model for defining both batch and streaming data-parallel processing pipelines. Beam provides a set of language-specific SDKs for constructing pipelines. These pipelines can be executed on distributed processing backends like [Apache Spark](http://spark.apache.org/), [Apache Flink](http://flink.apache.org), and [Google Cloud Dataflow](http://cloud.google.com/dataflow). -[Google Cloud Dataflow](https://cloud.google.com/dataflow/) -provides a simple, powerful programming model for building both batch -and streaming parallel data processing pipelines. - -The Dataflow SDK for Python provides access to Dataflow capabilities -from the Python programming language. +Apache Beam for Python provides access to Beam capabilities from the Python programming language. ## Table of Contents - * [Status of this Release](#status-of-this-release) - * [Signing up for Alpha Batch Cloud Execution](#signing-up-for-alpha-batch-cloud-execution) - * [Overview of Dataflow Programming](#overview-of-dataflow-programming) + * [Overview of the Beam Programming Model](#overview-of-the-programming-model) * [Getting Started](#getting-started) - * [Setting up an environment](#setting-up-an-environment) - * [Install ``pip``](#install-pip) - * [Install ``virtualenv``](#install-virtualenv) - * [Install ``setuptools``](#install-setuptools) - * [Getting the Dataflow software](#getting-the-dataflow-software) - * [Create and activate virtual environment](#create-and-activate-virtual-environment) + * [Set up your environment](#set-up-your-environment) + * [Install pip](#install-pip) + * [Install virtualenv](#install-virtualenv) + * [Get Apache Beam](#get-apache-beam) + * [Create and activate a virtual environment](#create-and-activate-a-virtual-environment) * [Download and install](#download-and-install) - * [Notes on installing with ``setup.py install``](#notes-on-installing-with-setuppy-install) - * [Local execution of a pipeline](#local-execution-of-a-pipeline) + * [Execute a pipeline locally](#execute-a-pipeline-locally) * [A Quick Tour of the Source Code](#a-quick-tour-of-the-source-code) * [Simple Examples](#simple-examples) * [Basic pipeline](#basic-pipeline) @@ -51,199 +42,105 @@ from the Python programming language. * [Counting words with GroupByKey](#counting-words-with-groupbykey) * [Type hints](#type-hints) * [BigQuery](#bigquery) - * [Combiner Examples](#combiner-examples) - * [More Examples](#more-examples) + * [Combiner examples](#combiner-examples) * [Organizing Your Code](#organizing-your-code) * [Contact Us](#contact-us) -## Status of this Release - -This is a version of Google Cloud Dataflow SDK for Python that is -still early in its development, and significant changes -should be expected before the first stable version. - -Google recently -[announced its intention](http://googlecloudplatform.blogspot.com/2016/01/Dataflow-and-open-source-proposal-to-join-the-Apache-Incubator.html) -to donate the Google Cloud Dataflow SDKs and Programming Model to -the Apache Software Foundation (ASF), after which they will be called the -Apache Beam SDKs. - -The SDK for Java is actively transitioning to -[Apache Beam](http://beam.incubator.apache.org/), -an ASF incubator project. The SDK for Python will be added -to Apache Beam soon after. Expect many renames. - -## Signing up for Alpha Batch Cloud Execution +## Overview of the Programming Model -Google Cloud Dataflow now provides Alpha support for Batch pipelines written -with the SDK for Python. This Alpha program is designed to give customers access -to the service for early testing. Customers are advised -not to use this feature in production systems. If you are interested in -being considered to participate in the Alpha program, -please submit this [form](http://goo.gl/forms/o4w14whz9x). -Note that filling the form does not guarantee entry to the Alpha program. +The key concepts of the programming model are: -## Overview of Dataflow Programming - -For an introduction to the programming model, please read -[Dataflow Programming Model](https://cloud.google.com/dataflow/model/programming-model) -but note that some examples on that site use only Java. -The key concepts of the programming model are - -* [`PCollection`](https://cloud.google.com/dataflow/model/pcollection): -represents a collection of data, which could be bounded or unbounded in size. -* [`PTransform`](https://cloud.google.com/dataflow/model/transforms): -represents a computation that transforms input PCollections into output +* PCollection - represents a collection of data, which could be bounded or unbounded in size. +* PTransform - represents a computation that transforms input PCollections into output PCollections. -* [`Pipeline`](https://cloud.google.com/dataflow/model/pipelines): -manages a directed acyclic graph of PTransforms and PCollections that is ready +* Pipeline - manages a directed acyclic graph of PTransforms and PCollections that is ready for execution. -* `Runner`: -specifies where and how the Pipeline should execute. - -This release has some significant limitations: +* Runner - specifies where and how the Pipeline should execute. -* We provide only one PipelineRunner, the `DirectPipelineRunner`. -* The Google Cloud Dataflow service does not yet accept jobs from this SDK. -* Triggers are not supported. -* The SDK works only on Python 2.7. +For a further, detailed introduction, please read the +[Beam Programming Model](http://beam.incubator.apache.org/learn/programming-guide.md). ## Getting Started -### Setting up an environment - -If this is the first time you are installing the Dataflow SDK, you may need to -set up your machine's Python development environment. - -#### Install ``pip`` +### Set up your environment `pip` is Python's package manager. If you already have `pip` installed (type `pip -V` to check), please make sure to have at least version 7.0.0. -There are several ways to install `pip`; use whichever works for you. +#### Install `pip` -Preferred option: install using your system's package manager, which may be -*one* of the following commands, depending on your Linux distribution: - -```sh - sudo yum install python-pip - sudo apt-get install python-pip - sudo zypper install python-pip -``` - -Otherwise, if you have `easy_install` (likely if you are on MacOS): - - sudo easy_install pip - -Or you may have to install the bootstrapper. Download the following script -to your system: https://bootstrap.pypa.io/get-pip.py -You can fetch it with your browser or use a command-line program, such as *one* -of the following: - -```sh - curl -O https://bootstrap.pypa.io/get-pip.py - wget https://bootstrap.pypa.io/get-pip.py -``` - -After downloading `get-pip.py`, run it to install `pip`: - -```sh -python ./get-pip.py -``` +Check if you already have `pip`, Python's package manager, installed by running <code>pip -V</code>. If not, [install pip](https://pip.pypa.io/en/stable/installing/). -#### Install ``virtualenv`` +#### Install `virtualenv` -We recommend installing in a -[Python virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) -for initial experiments. If you do not have `virtualenv` version 13.1.0 -or later (type `virtualenv --version` to check), it will install a too-old -version of `setuptools` in the virtual environment. To install (or upgrade) -your `virtualenv`: +It's recommended that you install a [Python virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) +for initial experiments. Check if you have it installed by running `virtualenv --version`. If you do not have `virtualenv` version 13.1.0 or later, install (or upgrade) your `virtualenv`: - pip install --upgrade virtualenv +`pip install --upgrade virtualenv` -#### Install ``setuptools`` +If you are not going to use a Python virtual environment (not recommended!), ensure `setuptools` version 17.1 or newer is installed on your machine. (run `easy_install --version` to check). If not, install `setuptools`: -If you are not going to use a Python virtual environment (but we recommend you -do; see the previous section), ensure `setuptools` version 17.1 or newer is -installed (type `easy_install --version` to check). If you do not have that -installed: +`pip install --upgrade setuptools` - pip install --upgrade setuptools +### Get Apache Beam -### Getting the Dataflow software - -#### Create and activate virtual environment +#### Create and activate a virtual environment A virtual environment is a directory tree containing its own Python -distribution. To create a virtual environment: +distribution. To create a virtual environment, create a directory and run: - virtualenv /path/to/directory +``` +virtualenv /path/to/directory +``` -A virtual environment needs to be activated for each shell that is to use it; -activating sets some environment variables that point to the virtual -environment's directories. To activate a virtual environment in Bash: +A virtual environment needs to be activated for each shell that is to use it. +Activating it sets some environment variables that point to the virtual +environment's directories. To activate a virtual environment in Bash, run: - . /path/to/directory/bin/activate +``` +. /path/to/directory/bin/activate +``` That is, source the script `bin/activate` under the virtual environment directory you created. #### Download and install -Install the latest tarball from GitHub by browsing to -<https://github.com/GoogleCloudPlatform/DataflowPythonSDK/releases/latest> -and copying one of the "Source code" links. The `.tar.gz` file is smaller; -we'll assume you use that one. With a virtual environment active, paste the -URL into a ``pip install`` shell command, executing something like this: - -```sh -pip install https://github.com/GoogleCloudPlatform/DataflowPythonSDK/archive/vX.Y.Z.tar.gz -``` - -#### Notes on installing with ``setup.py install`` - -We recommend installing using ``pip install``, as described above. -However, you also may install from an unpacked source code tree. -You can get such a tree by un-tarring the ``.tar.gz`` file or -by using ``git clone``. From a source tree, you can install by running - - cd DataflowPythonSDK* - python setup.py install --root / - python setup.py test +1. Clone the Apache Beam repo from GitHub: + `git clone https://github.com/apache/incubator-beam.git --branch python-sdk` -The ``--root /`` prevents Dataflow from being installed as an ``egg`` package. -This workaround prevents failures if Dataflow is installed in the same virtual -environment as another package under the ``google`` top-level package. +2. Navigate to the `python` directory: + `cd incubator-beam/sdks/python/` -If you get import errors during or after installing with ``setup.py``, -uninstall the package: +3. Create the Apache Beam Python SDK installation package: + `python setup.py sdist` - pip uninstall python-dataflow +4. Navigate to the `dist` directory: + `cd dist/` -and use the ``pip install`` method described above to re-install it. +5. Install the Apache Beam SDK + `pip install apache-beam-sdk-*.tar.gz` -## Local execution of a pipeline +### Execute a pipeline locally -The `$VIRTUAL_ENV/lib/python2.7/site-packages/google/cloud/dataflow/examples` -subdirectory (the `google/cloud/dataflow/examples` subdirectory in the -source distribution) has many examples large and small. +The Apache Beam [examples](https://github.com/apache/incubator-beam/tree/python-sdk/sdks/python/apache_beam/examples) directory has many examples. All examples can be run locally by passing the arguments required by the example script. -All examples can be run locally by passing the arguments required by the -example script. For instance, to run `wordcount.py`, try: +For example, to run `wordcount.py`, run: - python -m google.cloud.dataflow.examples.wordcount --output OUTPUT_FILE +``` +python -m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt --output output.txt +``` ## A Quick Tour of the Source Code -You can follow along this tour by, with your virtual environment -active, running a `pydoc` server on a local port of your choosing -(this example uses port 8888). +With your virtual environment active, you can follow along this tour by running a `pydoc` server on a local port of your choosing (this example uses port 8888): - pydoc -p 8888 +``` +pydoc -p 8888 +``` -Now open your browser and go to -http://localhost:8888/google.cloud.dataflow.html +Open your browser and go to +http://localhost:8888/apache_beam.html Some interesting classes to navigate to: @@ -258,6 +155,8 @@ Some interesting classes to navigate to: ## Simple Examples +The following examples demonstrate some basic, fundamental concepts for using Apache Beam's Python SDK. For more detailed examples, Beam provides a [directory of examples](https://github.com/apache/incubator-beam/tree/python-sdk/sdks/python/apache_beam/examples) for Python. + ### Basic pipeline A basic pipeline will take as input an iterable, apply the @@ -280,9 +179,7 @@ p.run() ### Basic pipeline (with Map) -The `Map` `PTransform` takes a callable, which will be applied to each -element of the input `PCollection` and must return an element to go -into the output `PCollection`. +The `Map` `PTransform` returns one output per input. It takes a callable that is applied to each element of the input `PCollection` and returns an element to the output `PCollection`. ```python import apache_beam as beam @@ -300,6 +197,8 @@ p.run() A `FlatMap` is like a `Map` except its callable returns a (possibly empty) iterable of elements for the output `PCollection`. +The `FlatMap` transform returns zero to many output per input. It accepts a callable that is applied to each element of the input `PCollection` and returns an iterable with zero or more elements to the output `PCollection`. + ```python import apache_beam as beam p = beam.Pipeline('DirectPipelineRunner') @@ -336,9 +235,7 @@ p.run() ### Counting words -This example shows how to read a text file from -[Google Cloud Storage](https://cloud.google.com/storage/) -and count its words. +This example shows how to read a text file from [Google Cloud Storage](https://cloud.google.com/storage/) and count its words. ```python import re @@ -355,10 +252,8 @@ p.run() ### Counting words with GroupByKey -This is a somewhat forced example of `GroupByKey` to achieve the same -functionality of the previous example without using -`beam.combiners.Count.PerElement`. It demonstrates also the use of a -wildcard to specify the text file source. +This is a somewhat forced example of `GroupByKey` to count words as the previous example did, but without using `beam.combiners.Count.PerElement`. As shown in the example, you can use a wildcard to specify the text file source. + ```python import re import apache_beam as beam @@ -399,9 +294,7 @@ p.run() ### BigQuery -This example calculates the number of tornadoes per month (from weather data). -The input is read from a BigQuery table and the output is written to a -different table specified by the user, along with a target project. +This example reads weather data from a BigQuery table, calculates the number of tornadoes per month, and writes the results to a table you specify. ```python import apache_beam as beam @@ -425,8 +318,7 @@ p = beam.Pipeline(argv=['--project', project]) p.run() ``` -This pipeline calculates the number of tornadoes per month, but it uses -a query to filter out the input instead of using the whole table. +This pipeline, like the one above, calculates the number of tornadoes per month, but it uses a query to filter out the input instead of using the whole table. ```python import apache_beam as beam @@ -448,11 +340,7 @@ p.run() ### Combiner Examples -Combiners are used to create a `PCollection` that contains the sums -(or max or min) of each of the keys in the initial `PCollecion`. -Such standard Python functions can be used directly as combiner -functions. In fact, any function "reducing" an iterable to a -single value can be used. +Combiner transforms use "reducing" functions, such as sum, min, or max, to combine multiple values of a `PCollection` into a single value. ```python import apache_beam as beam @@ -467,33 +355,16 @@ SAMPLE_DATA = [('a', 1), ('b', 10), ('a', 2), ('a', 3), ('b', 20)] p.run() ``` -The `google/cloud/dataflow/examples/cookbook/combiners_test.py` file in the -source distribution contains more combiner examples. - -### More Examples - -The `google/cloud/dataflow/examples` subdirectory in the -source distribution has some larger examples. +The [combiners_test.py](https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/combiners_test.py) file contains more combiner examples. ## Organizing Your Code -Many projects will grow to multiple source code files. It is beneficial to -organize the project so that all the code involved in running a workflow can be -built as a Python package so that it can be installed in the VM workers -executing a job. +Many projects will grow to multiple source code files. It is recommended that you organize your project so that all code involved in running your pipeline can be built as a Python package. This way, the package can easily be installed in the VM workers executing the job. -Please follow the example in `google/cloud/dataflow/examples/complete/juliaset`. -If the code is organized in this fashion then you can use the `--setup_file` -command line option to create a source distribution out of the project files, -stage the resulting tarball and later install it in the workers executing the -job. +Follow the [Juliaset example](https://github.com/apache/incubator-beam/tree/python-sdk/sdks/python/apache_beam/examples/complete/juliaset). If the code is organized in this fashion, you can use the `--setup_file` command line option to create a source distribution out of the project files, stage the resulting tarball, and later install it in the workers executing the job. -## Contact Us +## More Information -We welcome all usage-related questions on -[Stack Overflow](https://stackoverflow.com/questions/tagged/google-cloud-dataflow) -tagged with `google-cloud-dataflow`. +Please report any issues on [JIRA](https://issues.apache.org/jira/browse/BEAM/component/12328910). -Please use the -[issue tracker](https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues) -on GitHub to report any bugs, comments or questions regarding SDK development. +If youâre interested in contributing to the Beam SDK, start by reading the [Contribute](http://beam.incubator.apache.org/contribute/) guide.
