vchiapaikeo commented on issue #29366: URL: https://github.com/apache/airflow/issues/29366#issuecomment-1465278077
@uranusjr , what do you think about something like this to remove invalid keys from OperatorPartial's self.kwargs? https://github.com/apache/airflow/pull/30056 I'm using the below DAG to test and it's working as expected - valid args like `env` are applied to BashOperator and invalid ones like `abcdef` are deleted from self.kwargs. ```py from airflow import DAG from airflow.decorators import task from airflow.operators.bash import BashOperator DEFAULT_TASK_ARGS = { "owner": "gcp-data-platform", "start_date": "2023-03-11", "retries": 0, "abcdef": 12312, "env": { "HELLO": "WORLD", }, } @task def make_list(): # This can also be from an API call, checking a database, -- almost anything you like, as long as the # resulting list/dictionary can be stored in the current XCom backend. return [ "echo 1 ${HELLO}", "echo 2 ${HELLO}", "echo 3 ${HELLO}", ] with DAG( schedule_interval="@daily", max_active_runs=1, max_active_tasks=10, catchup=False, dag_id="test_expand_with_default_args", default_args=DEFAULT_TASK_ARGS, ) as dag: xcom_output = make_list.override(do_xcom_push=True)() t = BashOperator.partial(task_id="bash_op").expand(bash_command=xcom_output) ``` Granted, this does make the call to `validate_mapping_kwargs` in `__attrs_post_init__` obsolete. Do we really need this validation step if we are guaranteed to only be passing valid kwargs through now? Successful DAG: <img width="1440" alt="image" src="https://user-images.githubusercontent.com/9200263/224566371-2452ce6c-42ed-4271-9b10-50c03ef47aaa.png";> Mapped Task (1): <img width="1440" alt="image" src="https://user-images.githubusercontent.com/9200263/224566386-f067f09e-d98d-4054-83dd-ed3b7bc37058.png";> If this looks good to you, I can work on unit tests here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/9200263/224566371-2452ce6c-42ed-4271-9b10-50c03ef47aaa.png"](https://user-images.githubusercontent.com/9200263/224566371-2452ce6c-42ed-4271-9b10-50c03ef47aaa.png"){kind=link}
![https://user-images.githubusercontent.com/9200263/224566386-f067f09e-d98d-4054-83dd-ed3b7bc37058.png"](https://user-images.githubusercontent.com/9200263/224566386-f067f09e-d98d-4054-83dd-ed3b7bc37058.png"){kind=link}