[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration
[ https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998549#comment-16998549 ] ASF subversion and git services commented on AIRFLOW-6083: -- Commit 9aeb4196a817c32e3e5f51966d2fff82ecdeb2fa in airflow's branch refs/heads/v1-10-test from Daumis102 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9aeb419 ] [AIRFLOW-6083] Adding ability to pass custom configuration to lambda client. (#6678) There was no way to pass custom configuration from AwsLambdaHook to lambda client. This was an issue when for example default lambda timeout was too short. (cherry picked from commit c4c635df6906f56e01724573923e19763bb0da62) > AwsLambdaHook is not accepting non-default configuration > > > Key: AIRFLOW-6083 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6083 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.6 >Reporter: Daumantas Pagojus >Assignee: Daumantas Pagojus >Priority: Blocker > Fix For: 1.10.7 > > > Hello. > While using Airflow we have come across a problem with AwsLambdaHook. > We are using this hook to launch Lambda function which takes a while to > complete (around 5-6 minutes). However, since lambda is invoked through boto3 > client, which has default timeout set to [60 > seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/], > our Airflow interface shows this lambda failing, even though Lambda finishes > successfully checking at AWS console. This also causes another side effect: > Since boto3 thinks that Lambda has timed out, it automatically spawns another > instance, which also times out and this chain lasts 5 times, spawning 5 > Lambdas and all these Lambdas show as failed in Airflow interface, while they > actually succeed. > > This can be solved by passing in custom configuration when creating a boto3 > client, however, it is not possible to do that when creating AwsLambdaHook as > it does not take in this parameter. > However, we see that AwsLambdaHook inherits and uses AwsHook's function > (get_client_type) to get the boto3 client and this function accepts > configuration parameter (which defaults to None), but it is never passed to > it from the Lambda's hook, which could be easily achieved and would fix the > bug we are facing at the moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration
[ https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998465#comment-16998465 ] ASF subversion and git services commented on AIRFLOW-6083: -- Commit 186ce3ba7842b2576b1a314a8c1761b647e87a24 in airflow's branch refs/heads/v1-10-test from Daumis102 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=186ce3b ] [AIRFLOW-6083] Adding ability to pass custom configuration to lambda client. (#6678) There was no way to pass custom configuration from AwsLambdaHook to lambda client. This was an issue when for example default lambda timeout was too short. (cherry picked from commit c4c635df6906f56e01724573923e19763bb0da62) > AwsLambdaHook is not accepting non-default configuration > > > Key: AIRFLOW-6083 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6083 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.6 >Reporter: Daumantas Pagojus >Assignee: Daumantas Pagojus >Priority: Blocker > Fix For: 1.10.7 > > > Hello. > While using Airflow we have come across a problem with AwsLambdaHook. > We are using this hook to launch Lambda function which takes a while to > complete (around 5-6 minutes). However, since lambda is invoked through boto3 > client, which has default timeout set to [60 > seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/], > our Airflow interface shows this lambda failing, even though Lambda finishes > successfully checking at AWS console. This also causes another side effect: > Since boto3 thinks that Lambda has timed out, it automatically spawns another > instance, which also times out and this chain lasts 5 times, spawning 5 > Lambdas and all these Lambdas show as failed in Airflow interface, while they > actually succeed. > > This can be solved by passing in custom configuration when creating a boto3 > client, however, it is not possible to do that when creating AwsLambdaHook as > it does not take in this parameter. > However, we see that AwsLambdaHook inherits and uses AwsHook's function > (get_client_type) to get the boto3 client and this function accepts > configuration parameter (which defaults to None), but it is never passed to > it from the Lambda's hook, which could be easily achieved and would fix the > bug we are facing at the moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration
[ https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998432#comment-16998432 ] ASF subversion and git services commented on AIRFLOW-6083: -- Commit dfe396ed85a0f4fd31d10add6e7d04a68aef58b2 in airflow's branch refs/heads/v1-10-test from Daumis102 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=dfe396e ] [AIRFLOW-6083] Adding ability to pass custom configuration to lambda client. (#6678) There was no way to pass custom configuration from AwsLambdaHook to lambda client. This was an issue when for example default lambda timeout was too short. (cherry picked from commit c4c635df6906f56e01724573923e19763bb0da62) > AwsLambdaHook is not accepting non-default configuration > > > Key: AIRFLOW-6083 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6083 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.6 >Reporter: Daumantas Pagojus >Assignee: Daumantas Pagojus >Priority: Blocker > Fix For: 1.10.7 > > > Hello. > While using Airflow we have come across a problem with AwsLambdaHook. > We are using this hook to launch Lambda function which takes a while to > complete (around 5-6 minutes). However, since lambda is invoked through boto3 > client, which has default timeout set to [60 > seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/], > our Airflow interface shows this lambda failing, even though Lambda finishes > successfully checking at AWS console. This also causes another side effect: > Since boto3 thinks that Lambda has timed out, it automatically spawns another > instance, which also times out and this chain lasts 5 times, spawning 5 > Lambdas and all these Lambdas show as failed in Airflow interface, while they > actually succeed. > > This can be solved by passing in custom configuration when creating a boto3 > client, however, it is not possible to do that when creating AwsLambdaHook as > it does not take in this parameter. > However, we see that AwsLambdaHook inherits and uses AwsHook's function > (get_client_type) to get the boto3 client and this function accepts > configuration parameter (which defaults to None), but it is never passed to > it from the Lambda's hook, which could be easily achieved and would fix the > bug we are facing at the moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration
[ https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998317#comment-16998317 ] ASF GitHub Bot commented on AIRFLOW-6083: - ashb commented on pull request #6678: [AIRFLOW-6083] Add custom configuration to AWS lambda client URL: https://github.com/apache/airflow/pull/6678 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > AwsLambdaHook is not accepting non-default configuration > > > Key: AIRFLOW-6083 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6083 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.6 >Reporter: Daumantas Pagojus >Assignee: Daumantas Pagojus >Priority: Blocker > > Hello. > While using Airflow we have come across a problem with AwsLambdaHook. > We are using this hook to launch Lambda function which takes a while to > complete (around 5-6 minutes). However, since lambda is invoked through boto3 > client, which has default timeout set to [60 > seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/], > our Airflow interface shows this lambda failing, even though Lambda finishes > successfully checking at AWS console. This also causes another side effect: > Since boto3 thinks that Lambda has timed out, it automatically spawns another > instance, which also times out and this chain lasts 5 times, spawning 5 > Lambdas and all these Lambdas show as failed in Airflow interface, while they > actually succeed. > > This can be solved by passing in custom configuration when creating a boto3 > client, however, it is not possible to do that when creating AwsLambdaHook as > it does not take in this parameter. > However, we see that AwsLambdaHook inherits and uses AwsHook's function > (get_client_type) to get the boto3 client and this function accepts > configuration parameter (which defaults to None), but it is never passed to > it from the Lambda's hook, which could be easily achieved and would fix the > bug we are facing at the moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration
[ https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998319#comment-16998319 ] ASF subversion and git services commented on AIRFLOW-6083: -- Commit c4c635df6906f56e01724573923e19763bb0da62 in airflow's branch refs/heads/master from Daumis102 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c4c635d ] [AIRFLOW-6083] Adding ability to pass custom configuration to lambda client. (#6678) There was no way to pass custom configuration from AwsLambdaHook to lambda client. This was an issue when for example default lambda timeout was too short. > AwsLambdaHook is not accepting non-default configuration > > > Key: AIRFLOW-6083 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6083 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.6 >Reporter: Daumantas Pagojus >Assignee: Daumantas Pagojus >Priority: Blocker > > Hello. > While using Airflow we have come across a problem with AwsLambdaHook. > We are using this hook to launch Lambda function which takes a while to > complete (around 5-6 minutes). However, since lambda is invoked through boto3 > client, which has default timeout set to [60 > seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/], > our Airflow interface shows this lambda failing, even though Lambda finishes > successfully checking at AWS console. This also causes another side effect: > Since boto3 thinks that Lambda has timed out, it automatically spawns another > instance, which also times out and this chain lasts 5 times, spawning 5 > Lambdas and all these Lambdas show as failed in Airflow interface, while they > actually succeed. > > This can be solved by passing in custom configuration when creating a boto3 > client, however, it is not possible to do that when creating AwsLambdaHook as > it does not take in this parameter. > However, we see that AwsLambdaHook inherits and uses AwsHook's function > (get_client_type) to get the boto3 client and this function accepts > configuration parameter (which defaults to None), but it is never passed to > it from the Lambda's hook, which could be easily achieved and would fix the > bug we are facing at the moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration
[ https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987709#comment-16987709 ] Bjorn Olsen commented on AIRFLOW-6083: -- This should be resolved with AIRFLOW-6072 After this PR gets merged you could use the "Extra" part of the Airflow Connection to pass additional config to the boto3 client / resource / session as you please. > AwsLambdaHook is not accepting non-default configuration > > > Key: AIRFLOW-6083 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6083 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.6 >Reporter: Daumantas Pagojus >Assignee: Daumantas Pagojus >Priority: Blocker > > Hello. > While using Airflow we have come across a problem with AwsLambdaHook. > We are using this hook to launch Lambda function which takes a while to > complete (around 5-6 minutes). However, since lambda is invoked through boto3 > client, which has default timeout set to [60 > seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/], > our Airflow interface shows this lambda failing, even though Lambda finishes > successfully checking at AWS console. This also causes another side effect: > Since boto3 thinks that Lambda has timed out, it automatically spawns another > instance, which also times out and this chain lasts 5 times, spawning 5 > Lambdas and all these Lambdas show as failed in Airflow interface, while they > actually succeed. > > This can be solved by passing in custom configuration when creating a boto3 > client, however, it is not possible to do that when creating AwsLambdaHook as > it does not take in this parameter. > However, we see that AwsLambdaHook inherits and uses AwsHook's function > (get_client_type) to get the boto3 client and this function accepts > configuration parameter (which defaults to None), but it is never passed to > it from the Lambda's hook, which could be easily achieved and would fix the > bug we are facing at the moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration
[
https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983486#comment-16983486
]
ASF GitHub Bot commented on AIRFLOW-6083:
-
Daumis102 commented on pull request #6678: [AIRFLOW-6083] Adding custom
configuration to lambda client
URL: https://github.com/apache/airflow/pull/6678
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-6083) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6083
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
My PR adds ability to provide Lambda client with custom configuration.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: It does not need testing as the functionality that
was already implemented in function get_client_type was defaulting config param
to None, which I also do in case no config is provided to AwsLambdaHook
constructor. If config is provided, then the only thing I do is pass it to the
client creation function, which already has everything else implemented.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AwsLambdaHook is not accepting non-default configuration
>
>
> Key: AIRFLOW-6083
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6083
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
>Affects Versions: 1.10.6
>Reporter: Daumantas Pagojus
>Assignee: Daumantas Pagojus
>Priority: Blocker
>
> Hello.
> While using Airflow we have come across a problem with AwsLambdaHook.
> We are using this hook to launch Lambda function which takes a while to
> complete (around 5-6 minutes). However, since lambda is invoked through boto3
> client, which has default timeout set to [60
> seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/],
> our Airflow interface shows this lambda failing, even though Lambda finishes
> successfully checking at AWS console. This also causes another side effect:
> Since boto3 thinks that Lambda has timed out, it automatically spawns another
> instance, which also times out and this chain lasts 5 times, spawning 5
> Lambdas and all these Lambdas show as failed in Airflow interface, while they
> actually succeed.
>
> This can be solved by passing in custom configuration when creating a boto3
> client, however, it is not possible to do that when creating AwsLambdaHook as
> it does not take in this parameter.
> However, we see that AwsLambdaHook inherits and uses AwsHook's function
> (get_client_type) to get the boto3 client and this function accepts
> configuration parameter (which defaults to None), but it is
[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration
[ https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983485#comment-16983485 ] ASF GitHub Bot commented on AIRFLOW-6083: - Daumis102 commented on pull request #6677: AIRFLOW-6083 - Custom Lambda client parameters URL: https://github.com/apache/airflow/pull/6677 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > AwsLambdaHook is not accepting non-default configuration > > > Key: AIRFLOW-6083 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6083 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.6 >Reporter: Daumantas Pagojus >Assignee: Daumantas Pagojus >Priority: Blocker > > Hello. > While using Airflow we have come across a problem with AwsLambdaHook. > We are using this hook to launch Lambda function which takes a while to > complete (around 5-6 minutes). However, since lambda is invoked through boto3 > client, which has default timeout set to [60 > seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/], > our Airflow interface shows this lambda failing, even though Lambda finishes > successfully checking at AWS console. This also causes another side effect: > Since boto3 thinks that Lambda has timed out, it automatically spawns another > instance, which also times out and this chain lasts 5 times, spawning 5 > Lambdas and all these Lambdas show as failed in Airflow interface, while they > actually succeed. > > This can be solved by passing in custom configuration when creating a boto3 > client, however, it is not possible to do that when creating AwsLambdaHook as > it does not take in this parameter. > However, we see that AwsLambdaHook inherits and uses AwsHook's function > (get_client_type) to get the boto3 client and this function accepts > configuration parameter (which defaults to None), but it is never passed to > it from the Lambda's hook, which could be easily achieved and would fix the > bug we are facing at the moment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration
[
https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983482#comment-16983482
]
ASF GitHub Bot commented on AIRFLOW-6083:
-
Daumis102 commented on pull request #6677: AIRFLOW-6083 - Custom Lambda client
parameters
URL: https://github.com/apache/airflow/pull/6677
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-6083) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6083
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
My PR adds ability to provide Lambda client with custom configuration.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
It does not need testing as the functionality that was already implemented
in function get_client_type was defaulting config param to None, which I also
do in case no config is provided to AwsLambdaHook constructor. If config is
provided, then the only thing I do is pass it to the client creation function,
which already has everything else implemented.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AwsLambdaHook is not accepting non-default configuration
>
>
> Key: AIRFLOW-6083
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6083
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
>Affects Versions: 1.10.6
>Reporter: Daumantas Pagojus
>Assignee: Daumantas Pagojus
>Priority: Blocker
>
> Hello.
> While using Airflow we have come across a problem with AwsLambdaHook.
> We are using this hook to launch Lambda function which takes a while to
> complete (around 5-6 minutes). However, since lambda is invoked through boto3
> client, which has default timeout set to [60
> seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/],
> our Airflow interface shows this lambda failing, even though Lambda finishes
> successfully checking at AWS console. This also causes another side effect:
> Since boto3 thinks that Lambda has timed out, it automatically spawns another
> instance, which also times out and this chain lasts 5 times, spawning 5
> Lambdas and all these Lambdas show as failed in Airflow interface, while they
> actually succeed.
>
> This can be solved by passing in custom configuration when creating a boto3
> client, however, it is not possible to do that when creating AwsLambdaHook as
> it does not take in this parameter.
> However, we see that AwsLambdaHook inherits and uses AwsHook's function
> (get_client_type) to get the boto3 client and this function accepts
> configuration parameter (which defaults to None), but it is never passed
