o-nikolas commented on code in PR #34890:
URL: https://github.com/apache/airflow/pull/34890#discussion_r1357444134
##########
airflow/providers/amazon/aws/operators/lambda_function.py:
##########
@@ -62,13 +61,15 @@ class LambdaCreateFunctionOperator(BaseOperator):
:param aws_conn_id: The AWS connection ID to use
"""
+ aws_hook_class = LambdaHook
template_fields: Sequence[str] = (
"function_name",
"runtime",
"role",
"handler",
"code",
"config",
+ *AwsBaseOperator.template_fields,
Review Comment:
Rather than adding this to every operator class, can we do it the other way
around where the base class has this field already with these defaults and we
update it with anything new that the concrete class wants to add (if anything)?
It would need to be a mutable type of course, I'm not sure if it being
immutable is a requirement or not.
##########
docs/apache-airflow-providers-amazon/_partials/generic_parameters.rst:
##########
@@ -0,0 +1,69 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+aws_conn_id
+ Reference to :ref:`Amazon Web Services Connection <howto/connection:aws>`
ID.
+ If this parameter is set to ``None`` then the default boto3 behaviour is
used without lookup connection.
+ Otherwise use credentials stored into the Connection. Default:
``aws_default``
Review Comment:
```suggestion
If this parameter is set to ``None`` then the default boto3 behaviour is
used without a connection lookup.
Otherwise use the credentials stored in the Connection. Default:
``aws_default``
```
##########
docs/apache-airflow-providers-amazon/_partials/generic_parameters.rst:
##########
@@ -0,0 +1,69 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+aws_conn_id
+ Reference to :ref:`Amazon Web Services Connection <howto/connection:aws>`
ID.
+ If this parameter is set to ``None`` then the default boto3 behaviour is
used without lookup connection.
+ Otherwise use credentials stored into the Connection. Default:
``aws_default``
+
+region_name
+ AWS Region Name. If this parameter is set to ``None`` or omitted then
**region_name** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of connection value. Default:
``None``
+
+verify
+ Whether or not to verify SSL certificates.
+
+ * ``False`` - do not validate SSL certificates.
+ * **path/to/cert/bundle.pem** - A filename of the CA cert bundle to uses.
You can specify this argument
+ if you want to use a different CA cert bundle than the one used by
botocore.
+
+ If this parameter is set to ``None`` or omitted then **verify** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of from connection value. Default:
``None``
+
+botocore_config
+ Use provided dictionary to construct a `botocore.config.Config`_.
+ This configuration will able to use for
:ref:`howto/connection:aws:avoid-throttling-exceptions`, configure timeouts and
etc.
Review Comment:
```suggestion
botocore_config
The provided dictionary is used to construct a `botocore.config.Config`_.
This configuration can be used to configure
:ref:`howto/connection:aws:avoid-throttling-exceptions`, timeouts, etc.
```
##########
docs/apache-airflow-providers-amazon/_partials/generic_parameters.rst:
##########
@@ -0,0 +1,69 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+aws_conn_id
+ Reference to :ref:`Amazon Web Services Connection <howto/connection:aws>`
ID.
+ If this parameter is set to ``None`` then the default boto3 behaviour is
used without lookup connection.
+ Otherwise use credentials stored into the Connection. Default:
``aws_default``
+
+region_name
+ AWS Region Name. If this parameter is set to ``None`` or omitted then
**region_name** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of connection value. Default:
``None``
+
+verify
+ Whether or not to verify SSL certificates.
+
+ * ``False`` - do not validate SSL certificates.
+ * **path/to/cert/bundle.pem** - A filename of the CA cert bundle to uses.
You can specify this argument
+ if you want to use a different CA cert bundle than the one used by
botocore.
+
+ If this parameter is set to ``None`` or omitted then **verify** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of from connection value. Default:
``None``
Review Comment:
```suggestion
* ``False`` - Do not validate SSL certificates.
* **path/to/cert/bundle.pem** - A filename of the CA cert bundle to use.
You can specify this argument
if you want to use a different CA cert bundle than the one used by
botocore.
If this parameter is set to ``None`` or is omitted then **verify** from
:ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will be used.
Otherwise use the specified value instead of the connection value.
Default: ``None``
```
##########
docs/apache-airflow-providers-amazon/operators/lambda.rst:
##########
@@ -68,10 +73,22 @@ To invoke an AWS lambda function you can use
urllib3.exceptions.ReadTimeoutError:
AWSHTTPSConnectionPool(host='lambda.us-east-1.amazonaws.com', port=443): Read
timed out. (read timeout=60)
- If you encounter this issue, configure :ref:`howto/connection:aws` to
allow ``botocore`` / ``boto3`` to use long connections with
- timeout or keep-alive settings in the Connection Extras field
+ If you encounter this issue, you need to provide `botocore.config.Config
<https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html>`__
+ for use long connections with timeout or keep-alive settings
+
+ By provide **botocore_config** as operator parameter
Review Comment:
```suggestion
If you encounter this issue, you need to provide `botocore.config.Config
<https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html>`__
to use long connections with timeout or keep-alive settings.
By providing **botocore_config** as an operator parameter
```
##########
docs/apache-airflow-providers-amazon/connections/aws.rst:
##########
@@ -127,6 +127,10 @@ Extra (optional)
* ``verify``: Whether or not to verify SSL certificates.
+ * ``False`` - do not validate SSL certificates.
+ * **path/to/cert/bundle.pem** - A filename of the CA cert bundle to
uses. You can specify this argument
+ if you want to use a different CA cert bundle than the one used by
botocore.
+
Review Comment:
```suggestion
* ``False`` - Do not validate SSL certificates.
* **path/to/cert/bundle.pem** - A filename of the CA cert bundle to
use. You can specify this argument
if you want to use a different CA cert bundle than the one used by
botocore.
```
##########
docs/apache-airflow-providers-amazon/_partials/generic_parameters.rst:
##########
@@ -0,0 +1,69 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+aws_conn_id
+ Reference to :ref:`Amazon Web Services Connection <howto/connection:aws>`
ID.
+ If this parameter is set to ``None`` then the default boto3 behaviour is
used without lookup connection.
+ Otherwise use credentials stored into the Connection. Default:
``aws_default``
+
+region_name
+ AWS Region Name. If this parameter is set to ``None`` or omitted then
**region_name** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of connection value. Default:
``None``
+
+verify
+ Whether or not to verify SSL certificates.
+
+ * ``False`` - do not validate SSL certificates.
+ * **path/to/cert/bundle.pem** - A filename of the CA cert bundle to uses.
You can specify this argument
+ if you want to use a different CA cert bundle than the one used by
botocore.
+
+ If this parameter is set to ``None`` or omitted then **verify** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of from connection value. Default:
``None``
+
+botocore_config
+ Use provided dictionary to construct a `botocore.config.Config`_.
+ This configuration will able to use for
:ref:`howto/connection:aws:avoid-throttling-exceptions`, configure timeouts and
etc.
+
+ .. code-block:: python
+ :caption: Example, for more detail about parameters please have a look
`botocore.config.Config`_
+
+ {
+ "signature_version": "unsigned",
+ "s3": {
+ "us_east_1_regional_endpoint": True,
+ },
+ "retries": {
+ "mode": "standard",
+ "max_attempts": 10,
+ },
+ "connect_timeout": 300,
+ "read_timeout": 300,
+ "tcp_keepalive": True,
+ }
+
+ If this parameter is set to ``None`` or omitted then **config_kwargs** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of connection value. Default:
``None``
+
+ .. note::
+ Empty dictionary ``{}`` uses for construct default botocore Config
+ and will overwrite connection configuration for
`botocore.config.Config`_
Review Comment:
Check that I've got this one right, I had to do a bit more interpretation
here:
```suggestion
.. note::
Specifying an empty dictionary, ``{}``, will overwrite the
connection configuration for `botocore.config.Config`_
```
##########
docs/apache-airflow-providers-amazon/_partials/generic_parameters.rst:
##########
@@ -0,0 +1,69 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+aws_conn_id
+ Reference to :ref:`Amazon Web Services Connection <howto/connection:aws>`
ID.
+ If this parameter is set to ``None`` then the default boto3 behaviour is
used without lookup connection.
+ Otherwise use credentials stored into the Connection. Default:
``aws_default``
+
+region_name
+ AWS Region Name. If this parameter is set to ``None`` or omitted then
**region_name** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of connection value. Default:
``None``
Review Comment:
```suggestion
AWS Region Name. If this parameter is set to ``None`` or omitted then
**region_name** from
:ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will be used.
Otherwise use the specified value instead of the connection value.
Default: ``None``
```
##########
docs/apache-airflow-providers-amazon/_partials/generic_parameters.rst:
##########
@@ -0,0 +1,69 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+aws_conn_id
+ Reference to :ref:`Amazon Web Services Connection <howto/connection:aws>`
ID.
+ If this parameter is set to ``None`` then the default boto3 behaviour is
used without lookup connection.
+ Otherwise use credentials stored into the Connection. Default:
``aws_default``
+
+region_name
+ AWS Region Name. If this parameter is set to ``None`` or omitted then
**region_name** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of connection value. Default:
``None``
+
+verify
+ Whether or not to verify SSL certificates.
+
+ * ``False`` - do not validate SSL certificates.
+ * **path/to/cert/bundle.pem** - A filename of the CA cert bundle to uses.
You can specify this argument
+ if you want to use a different CA cert bundle than the one used by
botocore.
+
+ If this parameter is set to ``None`` or omitted then **verify** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of from connection value. Default:
``None``
+
+botocore_config
+ Use provided dictionary to construct a `botocore.config.Config`_.
+ This configuration will able to use for
:ref:`howto/connection:aws:avoid-throttling-exceptions`, configure timeouts and
etc.
+
+ .. code-block:: python
+ :caption: Example, for more detail about parameters please have a look
`botocore.config.Config`_
+
+ {
+ "signature_version": "unsigned",
+ "s3": {
+ "us_east_1_regional_endpoint": True,
+ },
+ "retries": {
+ "mode": "standard",
+ "max_attempts": 10,
+ },
+ "connect_timeout": 300,
+ "read_timeout": 300,
+ "tcp_keepalive": True,
+ }
+
+ If this parameter is set to ``None`` or omitted then **config_kwargs** from
+ :ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will use.
+ Otherwise use specified value instead of connection value. Default:
``None``
Review Comment:
```suggestion
If this parameter is set to ``None`` or omitted then **config_kwargs**
from
:ref:`AWS Connection Extra Parameter
<howto/connection:aws:configuring-the-connection>` will be used.
Otherwise use the specified value instead of the connection value.
Default: ``None``
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
