pankajastro opened a new issue, #24318:
URL: https://github.com/apache/airflow/issues/24318
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
_No response_
### Apache Airflow version
2.3.2 (latest released)
### Operating System
os
### Deployment
Other
### Deployment details
_No response_
### What happened
EmrCreateJobFlowOperator currently have two params for connection
`emr_conn_id` and `aws_conn_id`. So it works only when I set `aws_conn_id`
containing credentials and an empty `emr_conn_id` and it does not work in the
below case
- when I set both aws_conn_id and emr_conn_id in the operator and both
connection contains credentials i.e it has aws_access_key_id and other params
in airflow connection extra
```
Unknown parameter in input: "aws_access_key_id", must be one of: Name,
LogUri, LogEncryptionKmsKeyId, AdditionalInfo, AmiVersion, ReleaseLabel,
Instances, Steps, BootstrapActions, SupportedProducts, NewSupportedProducts,
Applications, Configurations, VisibleToAllUsers, JobFlowRole, ServiceRole,
Tags, SecurityConfiguration, AutoScalingRole, ScaleDownBehavior, CustomAmiId,
EbsRootVolumeSize, RepoUpgradeOnBoot, KerberosAttributes, StepConcurrencyLevel,
ManagedScalingPolicy, PlacementGroupConfigs, AutoTerminationPolicy,
OSReleaseLabel
```
- when I set both aws_conn_id and emr_conn_id in the operator and only
emr_conn_id connection contains credentials i.e it has aws_access_key_id and
other params in airflow connection extra
```
[2022-06-07, 20:49:19 UTC] {taskinstance.py:1826} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/opt/airflow/airflow/providers/amazon/aws/operators/emr.py", line
324, in execute
response = emr.create_job_flow(job_flow_overrides)
File "/opt/airflow/airflow/providers/amazon/aws/hooks/emr.py", line 87, in
create_job_flow
response = self.get_conn().run_job_flow(**job_flow_overrides)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
895, in _make_api_call
operation_model, request_dict, request_context
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
917, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line
116, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line
195, in _send_request
request = self.create_request(request_dict, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line
134, in create_request
operation_name=operation_model.name,
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 412,
in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 256,
in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 239,
in _emit
response = handler(**kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line
103, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line
187, in sign
auth.add_auth(request)
File "/usr/local/lib/python3.7/site-packages/botocore/auth.py", line 405,
in add_auth
raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
```
- When I set only aws_conn_id in the operator and it contains credentials
```
Traceback (most recent call last):
File "/opt/airflow/airflow/providers/amazon/aws/operators/emr.py", line
324, in execute
response = emr.create_job_flow(job_flow_overrides)
File "/opt/airflow/airflow/providers/amazon/aws/hooks/emr.py", line 90, in
create_job_flow
emr_conn = self.get_connection(self.emr_conn_id)
File "/opt/airflow/airflow/hooks/base.py", line 67, in get_connection
conn = Connection.get_connection_from_secrets(conn_id)
File "/opt/airflow/airflow/models/connection.py", line 430, in
get_connection_from_secrets
raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
```
- When I set only emr_conn_id in the operator and it contains credentials
```
[2022-06-07, 20:49:19 UTC] {taskinstance.py:1826} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/opt/airflow/airflow/providers/amazon/aws/operators/emr.py", line
324, in execute
response = emr.create_job_flow(job_flow_overrides)
File "/opt/airflow/airflow/providers/amazon/aws/hooks/emr.py", line 87, in
create_job_flow
response = self.get_conn().run_job_flow(**job_flow_overrides)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
895, in _make_api_call
operation_model, request_dict, request_context
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
917, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line
116, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line
195, in _send_request
request = self.create_request(request_dict, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line
134, in create_request
operation_name=operation_model.name,
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 412,
in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 256,
in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 239,
in _emit
response = handler(**kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line
103, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line
187, in sign
auth.add_auth(request)
File "/usr/local/lib/python3.7/site-packages/botocore/auth.py", line 405,
in add_auth
raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
```
### What you think should happen instead
It should work with even one connection id i.e with aws_conn_id or
emr_conn_id and should not fail even emr_conn_id has credentials
### How to reproduce
- Create EmrCreateJobFlowOperator and pass both aws_conn_id and emr_conn_id
or
- Create EmrCreateJobFlowOperator and pass aws_conn_id or emr_conn_id
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
