feluelle commented on a change in pull request #9594:
URL: https://github.com/apache/airflow/pull/9594#discussion_r455648080
##########
File path: airflow/providers/amazon/aws/hooks/sagemaker.py
##########
@@ -786,6 +825,33 @@ def list_training_jobs(
)
return results
+ def list_processing_jobs(self, **kwargs) -> List[Dict]: # noqa: D402
+ """
+ This method wraps boto3's list_processing_jobs(). All arguments should
be provided via kwargs.
+ Note boto3 expects these in CamelCase format, for example:
+
+ .. code-block:: python
+
+ list_processing_jobs(NameContains="myjob", StatusEquals="Failed")
+
+ .. seealso::
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.list_processing_jobs
+
+ :param kwargs: (optional) kwargs to boto3's list_training_jobs method
+ :return: results of the list_processing_jobs request
+ """
+
+ config = {}
+ config.update(kwargs)
+ list_processing_jobs_request =
partial(self.get_conn().list_processing_jobs, **config)
+ # Unset MaxResults, we'll use the SageMakerHook's internal method for
iteratively fetching results
+ max_results = kwargs["MaxResults"]
+ del kwargs["MaxResults"]
+ results = self._list_request(
+ list_processing_jobs_request, "ProcessingJobSummaries",
max_results=max_results
+ )
Review comment:
1. Note that you need to check if this key exists before accessing it as
it is optional `kwargs["MaxResults"]`
2. Do you really need to remove this? Because it will be automatically set
in `_list_request`.
So my suggestion:
```suggestion
results = self._list_request(
list_processing_jobs_request, "ProcessingJobSummaries",
max_results=kwargs.get("MaxResults")
)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
