Taragolis commented on code in PR #26687:
URL: https://github.com/apache/airflow/pull/26687#discussion_r985138469
##########
airflow/providers/amazon/aws/hooks/emr.py:
##########
@@ -77,22 +81,97 @@ def get_cluster_id_by_name(self, emr_cluster_name: str,
cluster_states: list[str
def create_job_flow(self, job_flow_overrides: dict[str, Any]) -> dict[str,
Any]:
"""
- Creates a job flow using the config from the EMR connection.
- Keys of the json extra hash may have the arguments of the boto3
- run_job_flow method.
- Overrides for this config may be passed as the job_flow_overrides.
+ Create and start running a new cluster (job flow).
+
+ This method uses ``EmrHook.emr_conn_id`` to receive the initial Amazon
EMR cluster configuration.
+ If ``EmrHook.emr_conn_id`` is empty or the connection does not exist,
then an empty initial
+ configuration is used.
+
+ :param job_flow_overrides: Is used to overwrite the parameters in the
initial Amazon EMR configuration
+ cluster. The resulting configuration will be used in the boto3 emr
client run_job_flow method.
+
+ .. seealso::
+ - :ref:`Amazon Elastic MapReduce Connection <howto/connection:emr>`
+ - `API RunJobFlow
<https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html>`_
+ - `boto3 emr client run_job_flow method
<https://boto3.amazonaws.com/v1/documentation/\
+
api/latest/reference/services/emr.html#EMR.Client.run_job_flow>`_.
"""
- try:
- emr_conn = self.get_connection(self.emr_conn_id)
- config = emr_conn.extra_dejson.copy()
- except AirflowNotFoundException:
- config = {}
+ config = {}
+ if self.emr_conn_id:
Review Comment:
Actually for this purpose usually use `get_conn()` however we can not
overwrite this method because it uses for obtain AWS credentials
(`aws_conn_id`).
We could create this method, but only use in one place. Current
implementation not contain any complex logic, so personally I do not see any
benefits with this separate method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
