burakovsky commented on issue #20186: URL: https://github.com/apache/airflow/issues/20186#issuecomment-1016411468
I had a similar problem and the issue was with Airflow IAM permissions. According to the [official documentation](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/iam-execution-role.html), your Airflow user (or role) must have `emr-containers:StartJobRun` permissions to run EMR job in EKS. But for Airflow, it's also required to have `emr-containers:DescribeJobRun` (and optional `emr-containers:CancelJobRun` for job canceling). The final IAM policy which works for me looks like this: ``` { "Version": "2012-10-17" "Statement": [ { "Sid": "" "Effect": "Allow", "Action": [ "emr-containers:StartJobRun" ], "Resource": "arn:aws:emr-containers:REGION:AWS_ACCOUNT_ID:/virtualclusters/VIRTUAL_CLUSTER_ID", "Condition": { "StringEquals": { "emr-containers:ExecutionRoleArn": "EXECUTION_ROLE_ARN" } } }, { "Sid": "" "Effect": "Allow", "Action": [ "emr-containers:DescribeJobRun", "emr-containers:CancelJobRun" ], "Resource": "arn:aws:emr-containers:REGION:AWS_ACCOUNT_ID:/virtualclusters/VIRTUAL_CLUSTER_ID/*", } ] } ``` Also, if you send EMR job logs to CloudWatch as is configured in the [example](https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.2.0/_modules/airflow/providers/amazon/aws/example_dags/example_emr_eks_job.html), be sure, that CloudWatch log group is created or your execution role has permissions to create new CloudWatch log group. I tested it on Airflow 2.1.4, probably for the latest version it works a bit differently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
