liferoad commented on code in PR #25719:
URL: https://github.com/apache/beam/pull/25719#discussion_r1136264862
##########
examples/notebooks/get-started/learn_beam_basics_by_doing.ipynb:
##########
@@ -0,0 +1,1068 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a
href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/learn_beam_basics_by_doing.ipynb\";
target=\"_parent\"><img
src=\"https://colab.research.google.com/assets/colab-badge.svg\"; alt=\"Open In
Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "V913rQcmLS72"
+ },
+ "source": [
+ "# Welcome to Apache Beam!\n",
+ "\n",
+ "This notebook will be your introductory guide to Beam's main concepts
and its uses. This tutorial **does not** assume any prior Apache Beam
knowledge.\n",
+ "\n",
+ "We'll cover what Beam is, what it does, and a few basic
transforms!\n",
+ "\n",
+ "We aim to give you familiarity with:\n",
+ "- Creating a `Pipeline`\n",
+ "- Creating a `PCollection`\n",
+ "- Performing basic `PTransforms`\n",
+ " - Map\n",
+ " - Filter\n",
+ " - FlatMap\n",
+ " - Combine\n",
+ "- Applications\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Lduh-9oXt3P_"
+ },
+ "source": [
+ "## How To Approach This Tutorial\n",
+ "\n",
+ "This tutorial was designed for someone who likes to **learn by
doing**. As such, there will be opportunities for you to practice writing your
own code in these notebooks with the answer hidden in a cell below.\n",
+ "\n",
+ "Codes that require editing will be with an `...` and each cell title
will say `Edit This Code`. However, you are free to play around with the other
cells if you would like to add something beyond our tutorial.\n",
+ "\n",
+ "It may be tempting to just copy and paste solutions, but even if you
do look at the Answer cells, try typing out the solutions manually. The muscle
memory will be very helpful.\n",
+ "\n",
+ "> Tip: For those who would like to learn concepts more from the
ground up, check out these
[notebooks](https://beam.apache.org/get-started/tour-of-beam/)!"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "42B-64Lvef3K"
+ },
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "We'll assume you have familiarity with Python or Pandas, but you
should be able to follow along even if you’re coming from a different
programming language. We'll also assume you understand programming concepts
like functions, objects, arrays, and dictionaries.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SeDD0nardXyL"
+ },
+ "source": [
+ "## Running CoLab\n",
+ "\n",
+ "To navigate through different sections, use the table of contents.
From View drop-down list, select Table of contents.\n",
+ "\n",
+ "To run a code cell, you can click the Run cell button at the top left
of the cell, or by select it and press `Shift+Enter`. Try modifying a code cell
and re-running it to see what happens."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UdMPnMDDkGc8"
+ },
+ "source": [
+ "To begin, we have to set up our environment. Let's install and import
Apache Beam by running the cell below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "5cPHukaOgDDM"
+ },
+ "outputs": [],
+ "source": [
+ "# Remember: You can press shift+enter to run this cell\n",
+ "!pip install --quiet apache-beam\n",
+ "import apache_beam as beam"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "30l8_MD-undP"
+ },
+ "outputs": [],
+ "source": [
+ "# Set the logging level to reduce verbose information\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gyoo1gLKtZmU"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "---\n",
+ "\n",
+ "\n",
+ "\n",
+ "---\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "N29KJTfdMCtB"
+ },
+ "source": [
+ "# What is Apache Beam?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "B2evhzEuMu8a"
+ },
+ "source": [
+ "Apache Beam is a library for data processing. It is often used for
[Extract-Transform-Load
(ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load) jobs, where
we:\n",
+ "1. *Extract* from a data source\n",
+ "2. *Transform* that data\n",
+ "3. *Load* that data into a data sink (like a database)\n",
+ "\n",
+ "Apache Beam makes these jobs easy with the ability to process
everything at the same time and its unified model and open-source SDKs. There
are many more parts of Beam, but throughout these tutorials, we will break down
each part to show you how they will all fit together.\n",
+ "\n",
+ "For this tutorial, you will use these Beam SDKs to build your own
`Pipeline` to process your data.\n",
+ "\n",
+ "Below, we will run through creating the heart of the `Pipeline`.
There are three main abstractions in Beam:\n",
+ "1. `Pipeline`\n",
+ "2. `PCollection`\n",
+ "3. `PTransform`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WecDJbfqpWb1"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "---\n",
+ "\n",
+ "\n",
+ "---\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FditMJAIAp9Q"
+ },
+ "source": [
+ "# 1. Design Your Pipeline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_xYj6e3Hvt1W"
+ },
+ "source": [
+ "## 1.1 What is a Pipeline"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "bVJCh9FLBtU9"
+ },
+ "source": [
+ "A `Pipeline` describes the whole cycle of your data processing task,
starting from the data sources to the processing transforms you will apply to
them until your desired output.\n",
+ "\n",
+ "`Pipeline` is responsible for reading, processing, and saving the
data. \n",
+ "Each `PTransform` is done on or outputs a `PCollection`, and this
process is done in your `Pipeline`.\n",
+ "More glossary details can be found at
[here](https://beam.apache.org/documentation/glossary/).\n",
+ "\n",
+ "A diagram of this process is shown below:\n",
+ "\n",
+ "<img
src='https://beam.apache.org/images/design-your-pipeline-linear.svg'>"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "cyn8_mB6zN4m"
+ },
+ "source": [
+ "In code, this process will look like this:\n",
+ "\n",
+ "\n",
+ "```\n",
+ "# Each `step` represents a specific transform. After `step3`, it will
save the data reference to `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ ">The pipe operator `|` applies the `PTransform` on the right side of
the pipe to the input `PCollection`.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gUyk2UWypI7g"
+ },
+ "source": [
+ "Pipelines can quickly grow long, so it's sometimes easier to read if
we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "7hmlbjPlvZVY"
+ },
+ "source": [
+ "Sometimes, the transform names aren't very descriptive. Beam allows
each transform, or step, to have a unique label, or description. This makes it
a lot easier to debug, and it's in general a good practice to start.\n",
+ "\n",
+ "> You can use the right shift operator `>>` to add a label to your
transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "GXQ__Kwxvr3J"
+ },
+ "source": [
+ "## 1.2 Loading Our Data\n",
+ "\n",
+ "Now, you can try to write your own pipeline!\n",
+ "\n",
+ "First, let's load the example data we will be using throughout this
tutorial into our file directory. This
[dataset](https://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection) consists
of a **collection of SMS messages in English tagged as either \"spam\" or
\"ham\" (a legitimate SMS\n",
+ ")**.\n",
+ "\n",
+ "For this tutorial, we will create a pipeline to **explore the dataset
using Beam to count words in SMS messages that contain spam or ham**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "BkZ0wxsBPuvO"
+ },
+ "outputs": [],
+ "source": [
+ "# Creates a data directory with our dataset SMSSpamCollection\n",
+ "!mkdir -p data\n",
+ "!gsutil cp gs://apachebeamdt/SMSSpamCollection data/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "TiHFDIMnLTbm"
+ },
+ "source": [
+ "**What does the data look like?**\n",
+ "\n",
+ "This dataset is a `txt` file with 5,574 rows and 4 columns recording
the following attributes:\n",
+ "1. `Column 1`: The label (either `ham` or `spam`)\n",
+ "2. `Column 2`: The SMS as raw text (type `string`)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! head data/SMSSpamCollection"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "r-0WM38KNaTI"
+ },
+ "source": [
+ "## 1.3 Writing Our Own Pipeline\n",
+ "\n",
+ "Now that we understand our dataset, let's go into creating our
pipeline.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1Ajz7-WQCVfj"
+ },
+ "source": [
+ "To initialize a `Pipeline`, you first assign your pipeline
`beam.Pipeline()` to a name. Assign your pipeline to the name, `pipeline`, in
the code cell below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "cXm0n80E9ZwT"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Edit This Code Cell\n",
+ "..."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "q0gzEcos9hjp"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Answer\n",
+ "pipeline = beam.Pipeline()"
Review Comment:
I noticed this as well and tried to hide this from the users. Since you
noticed this as well, I added `pipeline = beam.Pipeline()` explicitly to each
cell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
![https://colab.research.google.com/assets/colab-badge.svg\"](https://colab.research.google.com/assets/colab-badge.svg\"){kind=link}
