Taragolis commented on issue #35637:
URL: https://github.com/apache/airflow/issues/35637#issuecomment-1811983282
Unable to reproduce on latest main.
This DAG created fine without any issues
```python
import pendulum
from airflow.decorators import task_group
from airflow.models.dag import DAG
from airflow.providers.amazon.aws.sensors.glue import GlueJobSensor
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
S3_PATH = "s3://foo/bar"
EXTRA_KWARGS_GLUE = {}
with DAG(
"issue_35637",
start_date=pendulum.datetime(2023, 6, 1, tz="UTC"),
schedule=None,
catchup=False,
tags=["issue", "35637"]
):
@task_group(group_id="foo-bar", default_args=None)
def run_glue_job(job_name: str, script_args: dict):
submit_glue_job = GlueJobOperator(
# BaseOperator kwargs
task_id="submit_glue_job",
retries=0,
wait_for_completion=False,
# GlueJobOperator kwargs
job_name=job_name,
script_location=f"{S3_PATH}/glue_script.py",
script_args=script_args,
update_config=True,
**EXTRA_KWARGS_GLUE,
)
wait_on_glue_job = GlueJobSensor(
task_id="wait_on_glue_job",
job_name=job_name,
run_id=submit_glue_job.output,
)
submit_glue_job >> wait_on_glue_job
run_glue_job(job_name="foo", script_args={})
```
You need to check on [more recent version of
Airflow](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#release-notes),
there is such of changes and fixes in Airflow Core since then, include
`task_group` decorator.
In additional you need to check that all of your Airflow components use same
amazon-provider. The changes which add this field is 7.4.0
(https://github.com/apache/airflow/pull/30162) and maybe your
scheduler/dag_processor/worker use different version (6.0.0 ???) which are
below of this version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
