liferoad commented on code in PR #35931:
URL: https://github.com/apache/beam/pull/35931#discussion_r2304033050
##########
website/www/site/content/en/documentation/sdks/yaml.md:
##########
@@ -742,6 +742,74 @@ or writing dated sources and sinks, e.g.
would write to files like `gs://path/to/2016/08/04/dated-output*.json`.
+A user can also use the `% include` directive to pull in other common
templates:
+
+<PATH_TO_YOUR_REPO>/pipeline.yaml
+```sh
+pipeline:
+ transforms:
+ - name: Read from GCS
+ type: ReadFromText
+ config:
+{% include '<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml' %}
+ - name: Write to GCS
+ type: WriteToText
+ input: Read from GCS
+ config:
+ path: "gs://MY-BUCKET/wordCounts/"
+```
+
+<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml
+```sh
+ path: {{readFromText.path}}
+```
+
+This pipeline can be ran like this:
+
+```sh
+python -m apache_beam.yaml.main \
+ --yaml_pipeline_file=pipeline.yaml \
+ --jinja_variables='{"readFromText": {"path":
"gs://dataflow-samples/shakespeare/kinglear.txt"}}'
+```
+
+The `% import` jinja directive can also be used to pull in macros:
+
+<PATH_TO_YOUR_REPO>/pipeline.yaml
+```sh
+{% import '<PATH_TO_YOUR_REPO>/macros.yaml' as macros %}
+
+pipeline:
+ type: chain
+ transforms:
+
+# Read in text file
+{{ macros.readFromText(readFromText) | indent(4, true) }}
+
+# Write to text file on GCS, locally, etc
+ - name: Write to GCS
+ type: WriteToText
+ input: Read from GCS
+ config:
+ path: "gs://MY-BUCKET/wordCounts/"
+```
+
+<PATH_TO_YOUR_REPO>/macros.yaml
+```sh
+{%- macro readFromText(params) -%}
+- name: Read from GCS
+ type: ReadFromText
+ config:
+ path: "{{ params.path }}"
+{%- endmacro -%}
+```
+
+This pipeline can be ran with the same command as in the `% include` example
+above.
+
+There are many more ways to import and even use template inheritance using
+Jinja as seen
[here](https://jinja.palletsprojects.com/en/stable/templates/#import)
Review Comment:
can we link our examples here for users to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
