LaPetiteSouris commented on code in PR #26687:
URL: https://github.com/apache/airflow/pull/26687#discussion_r985073631
##########
airflow/providers/amazon/aws/hooks/emr.py:
##########
@@ -77,22 +81,97 @@ def get_cluster_id_by_name(self, emr_cluster_name: str,
cluster_states: list[str
def create_job_flow(self, job_flow_overrides: dict[str, Any]) -> dict[str,
Any]:
"""
- Creates a job flow using the config from the EMR connection.
- Keys of the json extra hash may have the arguments of the boto3
- run_job_flow method.
- Overrides for this config may be passed as the job_flow_overrides.
+ Create and start running a new cluster (job flow).
+
+ This method uses ``EmrHook.emr_conn_id`` to receive the initial Amazon
EMR cluster configuration.
+ If ``EmrHook.emr_conn_id`` is empty or the connection does not exist,
then an empty initial
+ configuration is used.
+
+ :param job_flow_overrides: Is used to overwrite the parameters in the
initial Amazon EMR configuration
+ cluster. The resulting configuration will be used in the boto3 emr
client run_job_flow method.
+
+ .. seealso::
+ - :ref:`Amazon Elastic MapReduce Connection <howto/connection:emr>`
+ - `API RunJobFlow
<https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html>`_
+ - `boto3 emr client run_job_flow method
<https://boto3.amazonaws.com/v1/documentation/\
+
api/latest/reference/services/emr.html#EMR.Client.run_job_flow>`_.
"""
- try:
- emr_conn = self.get_connection(self.emr_conn_id)
- config = emr_conn.extra_dejson.copy()
- except AirflowNotFoundException:
- config = {}
+ config = {}
+ if self.emr_conn_id:
Review Comment:
Neat-picking: Should we have a separate function ?
```python
def _validate_params_emr_conn_id(emr_conn_id: str):
```
IMHO it does increase readability by ditching a few level of indentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
