Dom created AIRFLOW-5819:
----------------------------
Summary: AWSBatchOperator has invalid default value for
array_properties
Key: AIRFLOW-5819
URL: https://issues.apache.org/jira/browse/AIRFLOW-5819
Project: Apache Airflow
Issue Type: Bug
Components: operators
Affects Versions: 1.10.6
Reporter: Dom
We upgraded our Airflow version from 1.10.3 to 1.10.6 and saw our
AWSBatchOperator throwing the following error (easily fixed by changing the
default param from None to {}):
{code:java}
[2019-10-30 13:10:15.392] INFO:airflow.task.operators.execute:93 Running AWS
Batch Job - Job definition: batch-jobdef-1 - on queue AwsBatch-batch-queue
[2019-10-30 13:10:15.393] INFO:airflow.task.operators.execute:95
AWSBatchOperator overrides: {'command': ['--start_datetime', '2019-10-29']}
[2019-10-30 13:10:15.433] INFO:airflow.task.operators.execute:121 AWS Batch Job
has failed executed
[2019-10-30 13:10:15.445] ERROR:airflow.task.handle_failure:1058 Parameter
validation failed:
Invalid type for parameter arrayProperties, value: None, type: <class
'NoneType'>, valid types: <class 'dict'>
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/awsbatch_operator.py",
line 108, in execute
containerOverrides=self.overrides)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357,
in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 634,
in _make_api_call
api_params, operation_model, context=request_context)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 682,
in _convert_to_request_dict
api_params, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/validate.py", line 297,
in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter arrayProperties, value: None, type: <class
'NoneType'>, valid types: <class 'dict'>During handling of the above exception,
another exception occurred:Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 930, in _run_raw_task
result = task_copy.execute(context=context)
File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/awsbatch_operator.py",
line 122, in execute
raise AirflowException(e)
airflow.exceptions.AirflowException: Parameter validation failed:
Invalid type for parameter arrayProperties, value: None, type: <class
'NoneType'>, valid types: <class 'dict'>
[2019-10-30 13:10:15.447] INFO:airflow.task.handle_failure:1087 All retries
failed; marking task as FAILED{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
