Bjorn Olsen commented on AIRFLOW-6083:

This should be resolved with AIRFLOW-6072

After this PR gets merged you could use the "Extra" part of the Airflow 
Connection to pass additional config to the boto3 client / resource / session 
as you please.

> AwsLambdaHook is not accepting non-default configuration
> --------------------------------------------------------
>                 Key: AIRFLOW-6083
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6083
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: aws
>    Affects Versions: 1.10.6
>            Reporter: Daumantas Pagojus
>            Assignee: Daumantas Pagojus
>            Priority: Blocker
> Hello.
> While using Airflow we have come across a problem with AwsLambdaHook.
> We are using this hook to launch Lambda function which takes a while to 
> complete (around 5-6 minutes). However, since lambda is invoked through boto3 
> client, which has default timeout set to [60 
> seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/],
>  our Airflow interface shows this lambda failing, even though Lambda finishes 
> successfully checking at AWS console. This also causes another side effect: 
> Since boto3 thinks that Lambda has timed out, it automatically spawns another 
> instance, which also times out and this chain lasts 5 times, spawning 5 
> Lambdas and all these Lambdas show as failed in Airflow interface, while they 
> actually succeed.
> This can be solved by passing in custom configuration when creating a boto3 
> client, however, it is not possible to do that when creating AwsLambdaHook as 
> it does not take in this parameter.
> However, we see that AwsLambdaHook inherits and uses AwsHook's function 
> (get_client_type) to get the boto3 client and this function accepts 
> configuration parameter (which defaults to None), but it is never passed to 
> it from the Lambda's hook, which could be easily achieved and would fix the 
> bug we are facing at the moment.

This message was sent by Atlassian Jira

Reply via email to