[
https://issues.apache.org/jira/browse/AIRFLOW-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235011#comment-17235011
]
ASF GitHub Bot commented on AIRFLOW-5115:
-----------------------------------------
dsynkov commented on a change in pull request #12389:
URL: https://github.com/apache/airflow/pull/12389#discussion_r526430786
##########
File path: tests/providers/amazon/aws/sensors/test_s3_key.py
##########
@@ -32,32 +32,40 @@ def
test_bucket_name_none_and_bucket_key_as_relative_path(self):
and bucket_key is provided as relative path rather than s3:// url.
:return:
"""
+ op = S3KeySensor(task_id='s3_key_sensor', bucket_key="file_in_bucket")
with self.assertRaises(AirflowException):
- S3KeySensor(task_id='s3_key_sensor', bucket_key="file_in_bucket")
+ op.poke(None)
def test_bucket_name_provided_and_bucket_key_is_s3_url(self):
"""
Test if exception is raised when bucket_name is provided
while bucket_key is provided as a full s3:// url.
:return:
"""
+ op = S3KeySensor(
+ task_id='s3_key_sensor', bucket_key="s3://test_bucket/file",
bucket_name='test_bucket'
+ )
with self.assertRaises(AirflowException):
- S3KeySensor(
- task_id='s3_key_sensor', bucket_key="s3://test_bucket/file",
bucket_name='test_bucket'
- )
+ op.poke(None)
@parameterized.expand(
[
['s3://bucket/key', None, 'key', 'bucket'],
['key', 'bucket', 'key', 'bucket'],
]
)
- def test_parse_bucket_key(self, key, bucket, parsed_key, parsed_bucket):
+ @mock.patch('airflow.providers.amazon.aws.sensors.s3_key.S3Hook')
Review comment:
Thanks; I added it as a separate test since per my understanding it
looks like rendering requires creating a `TaskInstance`. Let me know if that's
what you had in mind.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> S3KeySensor template_fields for bucket_name & bucket_key do not support Jinja
> variables
> ---------------------------------------------------------------------------------------
>
> Key: AIRFLOW-5115
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5115
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
> Affects Versions: 1.9.0
> Reporter: Dmitriy Synkov
> Assignee: Dmitriy Synkov
> Priority: Minor
> Labels: easyfix, patch
> Fix For: 2.0.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> In all Airflow operators (which inherit form {{BaseOperator}}) there is a
> {{template_fields}} attribute defined as ["which fields will get
> jinjafied"|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L218-L219]).
> For the {{S3KeySensor}} op in specific, these are {{template_fields =
> ('bucket_key', 'bucket_name')}}.
> The {{bucket_key}} kwarg, however, has some input validation in that the
> {{bucket_key}} needs to begin with the S3 protocol {{s3://}}; this exception
> is thrown by the
> [constructor|https://github.com/apache/airflow/blob/master/airflow/sensors/s3_key_sensor.py#L71-L74],
> which makes it impossible to use Jinja strings as an arg to {{bucket_key}},
> since these don't get rendered in the scope of the DAG {{*.py}} file itself.
> Below is an example; I'm using Airflow 1.9.0 with Python 3.5.3:
> Given the below DAG code, where "my_s3_key" is
> {{s3://bucket/prefix/object.txt:}}
> {code:java}
> dag = DAG('sample_dag', start_date=datetime(2019, 8, 1, 12, 15))
> s3_variable_sensor = S3KeySensor(
> task_id='s3_variable_sensor',
> bucket_key=Variable.get('my_s3_key'),
> dag=dag
> )
> s3_jinja_sensor = S3KeySensor(
> task_id='s3_jinja_sensor',
> bucket_key="{{ var.value.my_s3_key }}",
> dag=dag
> )
> {code}
> Executing the first task will run just fine while the next task will throw
> the following exception:
> {code:java}
> airflow.exceptions.AirflowException: Please provide a bucket_name.
> {code}
> This ticket is to propose a code change that will move input validation out
> of the constructor to allow for Jinja-templated strings to be passed into
> both {{bucket_name}} and {{bucket_key}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
