pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777532368
##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -96,7 +100,107 @@ def set_current_dataflow_job_id(job_id):
)
-class BeamRunPythonPipelineOperator(BaseOperator, BeamDataflowMixin):
+class BeamBasePipelineOperator(BaseOperator, BeamDataflowMixin, ABC):
+ """
+ Abstract base class for Beam Pipeline Operators.
+
+ :param runner: Runner on which pipeline will be run. By default
"DirectRunner" is being used.
+ Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+ See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+ See: https://beam.apache.org/documentation/runners/capability-matrix/
+
+ :type runner: str
+ :param default_pipeline_options: Map of default pipeline options.
+ :type default_pipeline_options: dict
+ :param pipeline_options: Map of pipeline options.The key must be a
dictionary.
+ The value can contain different types:
+
+ * If the value is None, the single option - ``--key`` (without value)
will be added.
+ * If the value is False, this option will be skipped
+ * If the value is True, the single option - ``--key`` (without value)
will be added.
+ * If the value is list, the many options will be added for each key.
+ If the value is ``['A', 'B']`` and the key is ``key`` then the
``--key=A --key-B`` options
+ will be left
+ * Other value types will be replaced with the Python textual
representation.
+
+ When defining labels (``labels`` option), you can also provide a
dictionary.
Review comment:
Not sure this is relevant indeed, It was like this in the child classes,
I have removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
