bradford stephens created AIRFLOW-1477:
------------------------------------------
Summary: S3KeySensor throws 400 Error
Key: AIRFLOW-1477
URL: https://issues.apache.org/jira/browse/AIRFLOW-1477
Project: Apache Airflow
Issue Type: Bug
Components: core
Affects Versions: 1.8.1
Environment: Ubuntu Linux 16.10
Reporter: bradford stephens
Despite configuring an S3 connection, S3KeySensor throws an error:
{code:java}
[2017-07-31 20:06:34,090] {base_task_runner.py:95} INFO - Subtask: [2017-07-31
20:06:34,088] {models.py:1430} ERROR - S3ResponseError: 400 Bad Request
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1387, in run
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: result =
task_copy.execute(context=context)
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py", line 76,
in execute
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: while
not self.poke(context):
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/airflow/operators/sensors.py", line
533, in poke
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: return
hook.check_for_key(self.bucket_key, self.bucket_name)
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py", line 262, in
check_for_key
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: bucket =
self.get_bucket(bucket_name)
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/airflow/hooks/S3_hook.py", line 222, in
get_bucket
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: return
self.connection.get_bucket(bucket_name)
[2017-07-31 20:06:34,091] {base_task_runner.py:95} INFO - Subtask: File
"/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py", line
506, in get_bucket
[2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask: return
self.head_bucket(bucket_name, headers=headers)
[2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask: File
"/home/bradford/.local/lib/python2.7/site-packages/boto/s3/connection.py", line
553, in head_bucket
[2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:
response.status, response.reason, body)
[2017-07-31 20:06:34,092] {base_task_runner.py:95} INFO - Subtask:
S3ResponseError: S3ResponseError: 400 Bad Request
{code}
I have tried both with and without the host parameter in the connection.
The DAG code includes:
{code:python}
inputsensor = S3KeySensor(
task_id='check_s3_for_file_in_s3',
bucket_key='test-input-temp.json',
wildcard_match=False,
bucket_name='ventera-staging-data',
s3_conn_id='S3Connection',
timeout=18*60*60,
poke_interval=120,
dag=dag)
{code}
The "extra" field in S3Connection has been set with the aws keys.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
