ashokballolli commented on code in PR #77069:
URL: https://github.com/apache/airflow/pull/77069#discussion_r4619332884


##########
providers/google/src/airflow/providers/google/cloud/hooks/dataflow.py:
##########
@@ -460,13 +461,33 @@ def fetch_job_id(self) -> str | None:
             self.log.info("Google Cloud DataFlow job not available yet..")
         return self._job_id
 
-    def _fetch_all_jobs(self, jobs_filter: str | None = None) -> list[dict]:
+    def fetch_latest_job_by_name(self, job_name: str) -> dict | None:
+        """
+        Return the latest created job with the given name, in any state.
+
+        Dataflow API does not support getting a single job by name, so the 
jobs in the location
+        are listed with the name passed to the list request.
+
+        :param job_name: Name of the job. It is converted to lowercase because 
Dataflow job names
+            are always lowercase.
+        :return: The job, or None if no job has this name.
+        """
+        job_name = job_name.lower()
+        return max(
+            (job for job in self._fetch_all_jobs(name=job_name) if 
job.get("name") == job_name),

Review Comment:
   No, it is not needed. I checked the list call with `name` on a real project: 
it returns only the job with that exact name, a prefix or a different case 
gives nothing. So I removed the check and the result of `_fetch_all_jobs` goes 
straight to `max()` now.
   
   ---
   Drafted-by: Claude Code (Opus 5.5); reviewed by @ashokballolli before posting
   



##########
providers/google/src/airflow/providers/google/cloud/hooks/dataflow.py:
##########
@@ -1602,10 +1609,17 @@ async def get_latest_job_by_name(
         :return: The job, or None if no job has this name.
         """
         job_name = job_name.lower()
-        jobs = await self.list_jobs(project_id=project_id, location=location)
-        matching_jobs = [job async for job in jobs if job.name == job_name]
-        # create_time is typed as protobuf Timestamp, but proto-plus returns 
it as a datetime.
-        return max(matching_jobs, key=lambda job: cast("datetime", 
job.create_time), default=None)
+        jobs = await self.list_jobs(project_id=project_id, location=location, 
name=job_name)
+        latest_job: Job | None = None
+        async for job in jobs:
+            if job.name != job_name:
+                continue
+            # create_time is typed as protobuf Timestamp, but proto-plus 
returns it as a datetime.
+            if latest_job is None or cast("datetime", job.create_time) > cast(
+                "datetime", latest_job.create_time
+            ):
+                latest_job = job
+        return latest_job

Review Comment:
   Sorry about that. I changed it to the loop to avoid the intermediate list, 
because `max()` cannot take the async pager directly. Looking at it again, it 
made the code harder to read for very little gain. I went back to the earlier 
version with the list comprehension and removed the name check, same as in the 
sync one. The async client gives the same exact-name result, I checked that one 
on the real project too.
   
   ---
   Drafted-by: Claude Code (Opus 5.5); reviewed by @ashokballolli before posting
   



##########
providers/google/tests/unit/google/cloud/hooks/test_dataflow.py:
##########
@@ -1939,6 +1959,19 @@ async def test_list_jobs(self, initialize_client_mock, 
hook, make_mock_awaitable
         initialize_client_mock.assert_called_once()
         client.list_jobs.assert_called_once_with(request=request)
 
+    @pytest.mark.asyncio
+    @mock.patch(DATAFLOW_STRING.format("AsyncDataflowHook.initialize_client"))
+    async def test_list_jobs_with_name(self, initialize_client_mock, hook):

Review Comment:
   Agreed, a separate test was not needed. I removed `test_list_jobs_with_name` 
and parametrized `test_list_jobs` with `name` (`None` and `"test-job"`) 
instead. I also removed the jobs with other names from the 
`get_latest_job_by_name` test data, since the API does the name filtering now.
   
   ---
   Drafted-by: Claude Code (Opus 5.5); reviewed by @ashokballolli before posting
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to