Srabasti commented on issue #32107: URL: https://github.com/apache/airflow/issues/32107#issuecomment-1610523420
Thanks for sharing your concern! To clarify further details, I have a Cloud Composer DAG, that is running Dataflow jobs to check if a file exists in GCS, and then loads into BigQuery using Dataflow, and then moves to processed folder. After the Composer DAG has been triggered, the generic error "Exception: DataFlow failed with return code 1" is shown in Composer log. Ideally, once Dataflow job is triggered, it should be showing the success/failure logs in Dataflow. However, no logs are shown in Dataflow. The log from above is from Composer task that failed. Since the log is not specific enough to determine the Root Cause. as a result had to go through details in Cloud Logging, and found the reason is that the file is getting deleted. If the log is specific and shows a few lines displayed of the standard error from Dataflow command run, this will help users give a context and figure out the easily. However since the error message is generic in Composer, it is difficult to figure out the RCA. Hence the request to display a couple of lines above this generic output, so it gives context to the user, for both Airflow versions. Please let me know for any questions! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
