Mariusz Strzelecki created AIRFLOW-4809:
-------------------------------------------
Summary: s3_delete_objects_operator fails on empty list of keys
Key: AIRFLOW-4809
URL: https://issues.apache.org/jira/browse/AIRFLOW-4809
Project: Apache Airflow
Issue Type: Bug
Components: contrib, operators
Affects Versions: 1.10.3
Reporter: Mariusz Strzelecki
Assignee: Mariusz Strzelecki
When s3_delete_objects_operator is used in a dynamic way (for example list of
keys comes from s3_list_operator via XCom) there might be a case when the list
of keys is empty. In my case it happens when chained operators are removing old
files from S3 and there are no old files yet (because this is very first run of
DAG).
In case of empty `keys` hook raises an exception (via boto3):
{noformat}
[2019-06-18 13:23:53,790] {{base_task_runner.py:101}} INFO - Job 115: Subtask
delete_old_files [2019-06-18 13:23:53,790] {{cli.py:517}} INFO - Running
<TaskInstance: xxxxxx.delete_old_files 2019-06-17T00:00:00+00:00 [running]> on
host 82f571f444f5
[2019-06-18 13:23:56,199] {{__init__.py:1580}} ERROR - An error occurred
(MalformedXML) when calling the DeleteObjects operation: The XML you provided
was not well-formed or did not validate against our published schema
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py",
line 1441, in _run_raw_task
result = task_copy.execute(context=context)
File
"/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/s3_delete_objects_operator.py",
line 80, in execute
response = s3_hook.delete_objects(bucket=self.bucket, keys=self.keys)
File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line
542, in delete_objects
Delete=delete_dict)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 357,
in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 661,
in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (MalformedXML) when calling
the DeleteObjects operation: The XML you provided was not well-formed or did
not validate against our published schema
{noformat}
I already have a patch that checks if there is anything to delete and if not,
just returns from the operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
