ashokballolli commented on code in PR #77069:
URL: https://github.com/apache/airflow/pull/77069#discussion_r4569400647
##########
providers/google/src/airflow/providers/google/cloud/hooks/dataflow.py:
##########
@@ -1540,6 +1582,31 @@ async def list_jobs(
page_result: ListJobsAsyncPager = await
client.list_jobs(request=request)
return page_result
+ async def get_latest_job_by_name(
+ self,
+ job_name: str,
+ project_id: str | None = PROVIDE_PROJECT_ID,
+ location: str | None = DEFAULT_DATAFLOW_LOCATION,
+ ) -> Job | None:
+ """
+ Get the latest created job with the given name, in any state.
+
+ Job names are not unique in Dataflow, so if more than one job has this
name, the newest one
+ is returned.
+
+ :param job_name: Name of the job. It is converted to lowercase because
Dataflow job names
+ are always lowercase.
+ :param project_id: Optional. The Google Cloud project ID in which the
job was started.
+ If set to None or missing, the default project_id from the Google
Cloud connection is used.
+ :param location: Optional. The location of the Dataflow job (for
example europe-west1).
+ :return: The job, or None if no job has this name.
+ """
+ job_name = job_name.lower()
+ jobs = await self.list_jobs(project_id=project_id, location=location)
+ matching_jobs = [job async for job in jobs if job.name == job_name]
+ # create_time is typed as protobuf Timestamp, but proto-plus returns
it as a datetime.
+ return max(matching_jobs, key=lambda job: cast("datetime",
job.create_time), default=None)
Review Comment:
Done here also. max() cannot be used on the async pager, so I changed it to
a simple loop which keeps the latest job, there is no list now. create_time is
already a datetime here, so the string sorting issue was not there in this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
