DrYampy commented on issue #36963: URL: https://github.com/apache/airflow/issues/36963#issuecomment-2669566660
IF you want to change the [CloudRunExecuteJobOperator](https://github.com/apache/airflow/blob/0adb5ad1cbee03309ffeffa91ec61722723713f1/providers/google/src/airflow/providers/google/cloud/operators/cloud_run.py#L352) to actually fetch and print the logs - this does so after completion: ``` def _wait_for_operation(self, operation: 'operation.Operation'): try: return operation.result(timeout=self.timeout_seconds) except Exception: error = operation.exception(timeout=self.timeout_seconds) raise AirflowException(error) finally: # This was a process to fetch logs from Google Cloud Logging - it hits the GCP project quota for 60 logging calls/min # This can be raised with qupta management, however I opted for just a link to the logs in both Datadog and Logging from google.cloud import logging_v2 client = logging_v2.Client(project=self.project_id) execution_name = operation.metadata.name.split('/')[-1] print(f"Isolated Execution ID: {execution_name}") # Define log filter for the specific Cloud Run job execution (run) filter_str = f""" resource.type="cloud_run_job" resource.labels.project_id="{self.project_id}" resource.labels.location="{self.region}" resource.labels.job_name="{self.job_name}" labels."run.googleapis.com/execution_name"="{execution_name}" """ print(f"Full Filter String:\n{filter_str}") # Execute the query to fetch logs print(f"EXECUTION LOGS BELOW!") log_entries = [] entries = client.list_entries(filter_=filter_str) for entry in entries: if hasattr(entry, "payload"): if isinstance(entry.payload, str): log_entries.append(entry.payload) elif isinstance(entry.payload, OrderedDict): log_entries.append(entry.payload["status"]["message"]) else: log_entries.append( "CORNER: Entry Payload is not a String or OrderedDict:" ) log_entries.append(entry.payload) else: log_entries.append("CORNER: No Payload in Log Entry:") log_entries.append(entry) # Print log entries to Airflow task logs for log_entry in log_entries: print(log_entry) ``` But again, I dont recommend this bevcause if you consider using this operator at scale where you need to fetch more than 60 executions a minute, youre gonna run afoul of the log api quota, and you will need to request a raise on it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
