eladkal commented on code in PR #36495:
URL: https://github.com/apache/airflow/pull/36495#discussion_r1438625527
##########
tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py:
##########
@@ -36,6 +36,14 @@
catchup=False,
tags=["synapse", "example"],
) as dag:
+ # [START howto_operator_azure_synapse_run_pipeline]
+ run_spark_job = AzureSynapseRunPipelineOperator(
+ task_id="run_pipeline_job",
+ spark_pool="provpipelinepool",
+ payload=PIPELINE_JOB_PAYLOAD, # type: ignore
+ )
+ # [END howto_operator_azure_synapse_run_pipeline]
Review Comment:
Let me explain
the code block of run_pipeline1 already exist
https://github.com/apache/airflow/blob/5983506df370325f7b23a182798341d17d091a32/tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py#L41-L46
You just need to add the `howto_operator_azure_synapse_run_pipeline` around
it.
If you will check your code you will see that you duplicated the block:
```
begin = EmptyOperator(task_id="begin")
# [START howto_operator_azure_synapse_run_pipeline]
run_pipeline1 = AzureSynapseRunPipelineOperator(
task_id="run_pipeline1",
azure_synapse_conn_id="azure_synapse_connection",
pipeline_name="Pipeline 1",
azure_synapse_workspace_dev_endpoint="azure_synapse_workspace_dev_endpoint",
)
# [END howto_operator_azure_synapse_run_pipeline]
begin >> run_pipeline1
begin = EmptyOperator(task_id="begin")
run_pipeline1 = AzureSynapseRunPipelineOperator(
task_id="run_pipeline1",
azure_synapse_conn_id="azure_synapse_connection",
pipeline_name="Pipeline 1",
azure_synapse_workspace_dev_endpoint="azure_synapse_workspace_dev_endpoint",
)
```
##########
tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py:
##########
@@ -36,6 +36,14 @@
catchup=False,
tags=["synapse", "example"],
) as dag:
+ # [START howto_operator_azure_synapse_run_pipeline]
+ run_spark_job = AzureSynapseRunPipelineOperator(
+ task_id="run_pipeline_job",
+ spark_pool="provpipelinepool",
+ payload=PIPELINE_JOB_PAYLOAD, # type: ignore
+ )
+ # [END howto_operator_azure_synapse_run_pipeline]
Review Comment:
Let me explain
the code block of `run_pipeline1` already exist
https://github.com/apache/airflow/blob/5983506df370325f7b23a182798341d17d091a32/tests/system/providers/microsoft/azure/example_synapse_run_pipeline.py#L41-L46
You just need to add the `howto_operator_azure_synapse_run_pipeline` around
it.
If you will check your code you will see that you duplicated the block:
```
begin = EmptyOperator(task_id="begin")
# [START howto_operator_azure_synapse_run_pipeline]
run_pipeline1 = AzureSynapseRunPipelineOperator(
task_id="run_pipeline1",
azure_synapse_conn_id="azure_synapse_connection",
pipeline_name="Pipeline 1",
azure_synapse_workspace_dev_endpoint="azure_synapse_workspace_dev_endpoint",
)
# [END howto_operator_azure_synapse_run_pipeline]
begin >> run_pipeline1
begin = EmptyOperator(task_id="begin")
run_pipeline1 = AzureSynapseRunPipelineOperator(
task_id="run_pipeline1",
azure_synapse_conn_id="azure_synapse_connection",
pipeline_name="Pipeline 1",
azure_synapse_workspace_dev_endpoint="azure_synapse_workspace_dev_endpoint",
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
