aviemzur commented on a change in pull request #5:
URL: https://github.com/apache/incubator-liminal/pull/5#discussion_r524040479
##########
File path: README.md
##########
@@ -36,52 +36,53 @@ perform), application servers, and more.
## Example YAML config file
```yaml
+---
name: MyPipeline
-owner: Bosco Albert Baracus
+volumes:
+ - volume: myvol1
+ local:
+ path: /tmp/liminal_tests
pipelines:
- pipeline: my_pipeline
+ owner: Bosco Albert Baracus
start_date: 1970-01-01
timeout_minutes: 45
schedule: 0 * 1 * *
+ default_arg_loaded: check
+ default_array_loaded: [2, 3, 4]
+ default_object_loaded:
+ key1: val1
+ key2: val2
metrics:
- namespace: TestNamespace
- backends: [ 'cloudwatch' ]
+ namespace: TestNamespace
+ backends: [ ]
tasks:
- - task: my_static_input_task
+ - task: my_python_task
type: python
description: static input task
- image: my_static_input_task_image
- source: helloworld
+ image: my_python_task_img
+ source: write_inputs
env_vars:
- env1: "a"
- env2: "b"
- input_type: static
- input_path: '[ { "foo": "bar" }, { "foo": "baz" } ]'
- output_path: /output.json
- cmd: python -u hello_world.py
- - task: my_parallelized_static_input_task
+ NUM_FILES: 10
+ NUM_SPLITS: 3
+ mounts:
+ - mount: mymount
+ volume: myvol1
+ path: /mnt/vol1
+ cmd: python -u write_inputs.py
+ - task: my_parallelized_python_task
type: python
- description: parallelized static input task
- image: my_static_input_task_image
+ description: parallelized python task
+ image: my_parallelized_python_task_img
+ source: write_outputs
env_vars:
- env1: "a"
- env2: "b"
- input_type: static
- input_path: '[ { "foo": "bar" }, { "foo": "baz" } ]'
- split_input: True
- executors: 2
- cmd: python -u helloworld.py
- - task: my_task_output_input_task
- type: python
- description: task with input from other task's output
- image: my_task_output_input_task_image
- source: helloworld
- env_vars:
- env1: "a"
- env2: "b"
- input_type: task
- input_path: my_static_input_task
- cmd: python -u hello_world.py
+ FOO: BAR
+ parallelism: 3
+ mounts:
Review comment:
Mounting is the same syntax regardless of the type of volume
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
