rshamunov commented on code in PR #25507: URL: https://github.com/apache/beam/pull/25507#discussion_r1169870976
########## playground/load_your_code.md: ########## @@ -0,0 +1,594 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# How to Add an Example/Snippet/Learning Content into Apache Beam Playground + +Adding a new example, code snippet or Tour of Beam learning unit into the Playground is a three-step process: + +1. Prepare a code snippet. +2. Add the code snippet to Apache Beam and/or Playground. +3. Create a link to view the code snippet in Playground or to embed in a website page. + + +Playground sources and output presentation formats: + + + +The guide will walk through all steps. + + +# Table of Contents + +- [Step 1. Prepare a code snippet](#step-1-prepare-a-code-snippet) + * [Data access limitations](#data-access-limitations) + * [Emphasizing parts of code](#emphasizing-parts-of-code) + * [Named Sections](#named-sections) +- [Step 2. Add the code snippet to the Apache Beam repo and/or Playground](#step-2-add-the-code-snippet-to-the-apache-beam-repo-andor-playground) + * [Source 1. How to add an example to Playground Examples Catalog](#source-1-how-to-add-an-example-to-playground-examples-catalog) + + [1. Add the file with example to a directory](#1-add-the-file-with-example-to-a-directory) + + [2. Add metadata to describe example](#2-add-metadata-to-describe-the-example) + - [Kafka emulator](#kafka-emulator) + + [3. Make a PR](#3-make-a-pr) + + [4. Save the snippet ID](#4-save-the-snippet-id) + * [Source 2. How to add an unlisted example to Playground](#source-2-how-to-add-an-unlisted-example-to-playground) + * [Source 3. How to add a Tour of Beam unit](#source-3-how-to-add-a-tour-of-beam-unit) + * [Source 4. How to add a snippet with the app alone](#source-4-how-to-add-a-snippet-with-the-app-alone) + * [Source 5. How to load code from GitHub or other HTTPS URL](#source-5-how-to-load-code-from-github-or-other-https-url) +- [Step 3. Create a link or embed snippet](#step-3-create-a-link-or-embed-snippet) + * [Link to a snippet](#link-to-a-snippet) + + [Link to an example from the Playground Examples Catalog](#link-to-an-example-from-the-playground-examples-catalog) + + [Link to snippet (`unlisted` Playground Examples Catalog item)](#link-to-snippet---unlisted--playground-examples-catalog-item-) + + [Link to a Tour of Beam unit](#link-to-a-tour-of-beam-unit) + + [Link to a user-shared snippet](#link-to-a-user-shared-snippet) + + [Link to a GitHub or HTTPS URL snippet](#link-to-a-github-or-https-url-snippet) + + [Link to an empty editor](#link-to-an-empty-editor) + * [Embedding a snippet into HTML](#embedding-a-snippet-into-html) + + [Embedding a snippet from Playground Examples Catalog](#embedding-a-snippet-from-playground-examples-catalog) + + [Embedding a user-shared snippet](#embedding-a-user-shared-snippet) + + [Embedding a snippet from other sources](#embedding-a-snippet-from-other-sources) + * [Embedding a snippet into the Apache Beam website](#embedding-a-snippet-into-the-apache-beam-website) +- [Snippet view options](#snippet-view-options) + * [Read-only sections](#read-only-sections) + * [Folding everything except named sections](#folding-everything-except-named-sections) + * [Hiding everything except a named section](#hiding-everything-except-a-named-section) + * [Linking to multiple examples](#linking-to-multiple-examples) + + +## Step 1. Prepare a code snippet + +Playground runs example code snippets using Apache Beam Direct Runner +and requires that a code snippet is a complete runnable code. + +### Data access limitations + +For security reasons, the snippets in Playground cannot access arbitrary internet resources. +Depending on your data source you should do the following to access data: + +| Source | Notes | +|----------|-----------------------------------------------------------| +| File | Put to a GCS bucket in `apache-beam-testing` project. | +| BigQuery | The table must be added to `apache-beam-testing` project. | + +Machine Learning: PyTorch, SKLearn -- change the playground itself. (request for enhancement to add a dependency to the Python image). +To request adding, please send an email to dev@... + +CLONING 3RD PARTY REPOS. +IF ML MODELS -- FROM PLAYGROUND BEAM ML EXAMPLES SUPPORT + +### Emphasizing parts of code + +Playground provides multiple features to help focus users on certain parts of the code. + +Playground automatically applies the following to all snippets: +- Folds a comment if a snippet starts with one. +- Folds imports. + +### Named Sections + +Playground supports *Named Sections* to tag code blocks and provide the following view options: +- Fold all blocks except tagged code blocks. + This can be useful to help user focus on specific code blocks and features presented in a snippet. +- Hide all code except tagged code blocks. + This can be useful to create runnable snippets illustrating specific concepts or transforms, + and hide all non-essential code blocks. + Such snippet can be embedded on a website to make examples in documentation and tutorials runnable. +- Make certain code parts read-only. + This feature can be useful to create learning units where user modifications are desired + only in certain parts of the code. +Please see [Snippet View Options](#snippet-view-options) section for details how different view options can be used. + +If you do not need any of those view options, skip to the [next step](#step-2-add-the-code-snippet-to-the-apache-beam-repo-and-playground). + +*Named Sections* are defined with the following syntax: + +``` +// [START section_name] +void method() { +... +} +// [END section_name] +``` + +Create a named section for each part of your code that you want the above features for. +To learn more details about the syntax please see +the [README of the editor](https://pub.dev/packages/flutter_code_editor) that Playground uses. + +## Step 2. Add the code snippet to the Apache Beam repo and/or Playground + +There are several types of code snippets in the Playground: + +1. Public Example — a code snippet displayed in the Playground Examples Catalog. + See [how to add a new public example here](#source-1-how-to-add-an-example-to-playground-examples-catalog). +2. Unlisted Example — the same as a public example, but it does not show in the example dropdown + and can only be accessed by direct linking. These are typically embedded on a website. + See [how to add a new unlisted example here](#source-2-how-to-add-an-unlisted-example-to-playground). +3. [Tour of Beam](https://github.com/apache/beam/tree/master/learning/tour-of-beam) learning unit. + See [how to add a new Tour of Beam unit here](#source-3-how-to-add-a-tour-of-beam-unit). +4. User-shared code snippets do not require a PR and should be used for code + not displayed on Beam resources. + See [how to add a snippet with the app alone here](#source-4-how-to-add-a-snippet-with-the-app-alone). +5. GitHub or other HTTPS URL sources. + See [how to load a snippet from external GitHub or other HTTPS URL here](#source-5-how-to-load-code-from-github-or-other-https-url). + +See the [workflow above](#how-to-add-an-examplesnippetlearning-content-into-apache-beam-playground) how artifacts map to these sources. + +### Source 1. How to add an example to Playground Examples Catalog + +Playground Examples Catalog helps users discover example snippets +and is a recommended way to add examples. Playground automatically scans, +verifies and deploys example snippets from the directories listed below. + +#### 1. Add the file with example to a directory + +Playground Java, Python, and Go examples are automatically picked from these +predefined directories by the `playground_examples_ci.yml` GitHub workflow +after a PR is merged to Beam repo: +- `/examples` +- `/learning/katas` +- `/sdks`. + +Adding Scala example snippets automatically is not supported, +and Scala example snippets can be added to the catalog manually. + +#### 2. Add metadata to describe the example + +Playground relies on metadata comments block to identify and place an example into the database, +and present in the Examples Catalog. +See [this](https://github.com/apache/beam/blob/3e080ff212d8ed7208c8486b515bb73c5d294475/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java#L20-L36) for an example. +Playground automatically removes metadata comments block before storing the example in database +so the metadata is not visible to end users. The block is in the format of a YAML map: + +```yaml +beam-playground: + # Name of the Beam example that will be displayed in the Playground Examples Catalog. Required. + name: "" + # Description of the Beam example that will be displayed in the Playground Examples Catalog. Required. + description: "" + # Contains information about pipeline options of the Beam example/test/kata. Optional. + pipeline_options: "--name1 value1 --name2 value2" + # The line number to scroll to when the snippet is loaded. + # Note that lines of the metadata block are cut so line numbers after it are shifted. Required. + context_line: 1 + # Categories this example is included into. Non-existent categories will be created. + # Optional, defaults to no categories making the example unlisted. + categories: + - "Combiners" + - "Core Transforms" + # Tags by which this snippet can be found in the Example Catalog. Optional. + tags: + - "numbers" + - "count" + # Helps user to identify example's complexity. Values: BASIC|MEDIUM|ADVANCED. Required. + complexity: BASIC + # Specifies the example to be loaded as default when its SDK selected in the Playground. + # See section "Default examples" below. Optional, defaults to false. + default_example: true + # If the snippet has a Colab notebook, can link the URL of the Colab notebook that is based on this snippet. + url_notebook: "https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb"; + # Specifies if the given example consists of multiple files or not. Optional, defaults to false. + multifile: true + # Specifies example caching by Playground. Optional, defaults to false (the output is cached). + always_run: true + # Datasets which will be used by emulators. Optional. + # Please see section "Kafka emulator" for more information. + datasets: + # Dataset name + CountWords: + # Dataset location. Only "local" is supported. Required. + location: local + # Dataset format. Supported values are "avro" and "json". Required. + format: avro + # List of emulators to start during pipeline execution. Currently only `kafka` type is supported. Optional. + emulators: + - type: kafka + topic: + # Dataset id. Will be used as a topic name. + id: dataset + # Name of dataset specified in "datasets" section. + source_dataset: "CountWords" +``` + +For metadata reference see fields in "Tag" class [here](infrastructure/models.py). + +##### Default examples + +Each SDK must have a single default example. +If there is none, the user will see the error in the app and face a blank editor. +If there are more than one, it is not defined which one will be selected. + +##### Kafka emulator + +Examples which require Kafka server emulator, need to include the `emulators` tag +and provide `dataset` in the example's tag. You can refer to an example +[here](/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountJson.java). + +1. Add your dataset in either JSON or Avro format into the `playground/backend/datasets` path. + +2. Add the following elements to the example's metadata tag: + ```YAML + emulators: + - type: kafka + topic: + id: dataset + source_dataset: <dataset_name> + datasets: + <dataset_name>: + location: local + format: json # or 'avro' + ``` + replace `<dataset_name>` with the name of your dataset file without the file name extension. + +3. Use the exact string `"kafka_server:9092"` as a server name in your code snippet. + This string will be replaced by the actual host name and port automatically + before the compilation step by Playground. + +>**Kafka emulator limitations:** +> - Playground Kafka emulator currently supports only Beam Java SDK +> - exact string `"kafka_server:9092"` should be present in the code snippet; + any other variation like `"kafa_server" + ":9092"` will not work + +#### 3. Make a PR + +Make a PR with code snippet into [the Apache Beam repository](https://github.com/apache/beam) +following the [Contribution guide](https://beam.apache.org/contribute/). +Verify that all pre-commit tests are passing. + +Playground CI will verify and deploy the example to Playground Example Catalog when the PR is merged. + +#### 4. Save the snippet ID + +The snippet will be assigned an ID. +You can find it in the address bar of the browser when you select it in the dropdown. + +For example, in this URL: + +``` +https://play.beam.apache.org/?path=SDK_JAVA_MinimalWordCount&sdk=java +``` + +the ID is: `SDK_JAVA_MinimalWordCount`. + +You will need the snippet ID to embed the Playground with snippet into a website page. + +### Source 2. How to add an unlisted example to Playground + +Not all examples must be visible in the example dropdown. +Some examples are best in the context of Apache Beam documentation. +To embed them into the documentation use unlisted examples. +They work and are checked and cached the same way as public examples. + +Proceed the same way as with [Source 1. Playground Examples Catalog](#source-1-how-to-add-an-example-to-playground-examples-catalog) except: +1. Use the directory `/learning/beamdoc` Review Comment: Replace "use" to something more meaningful, e.g. add an example file to the directory ./beam/learning/beamdoc/<sdk>. It'd be better to have a dedicated subfolder for each SDK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
