jason810496 opened a new pull request, #56869: URL: https://github.com/apache/airflow/pull/56869
related: #56811 ## Why After #56811, we can ensure there are 6 files present in S3 (the same amount of tasks in `example_xcom_test` Dag) before validating source `task_log_sources` and `log_files`. However the any CI with `Additional PROD image tests / Remote logging tests with PROD image / test-e2e-integration-tests` is still failing. IMO, the **main root cause is the `/opt/airflow/logs/` prefix in `task_log_sources`, which make the `any(source in log_files for source in task_log_sources)` condition always fail**. Example fail runs: - https://github.com/apache/airflow/actions/runs/18640518241/job/53139796492#logs - https://github.com/apache/airflow/actions/runs/18631048833/job/53116606752?pr=56848 ``` =========================== short test summary info ============================ FAILED tests/airflow_e2e_tests/remote_log_tests/test_remote_logging.py::TestRemoteLogging::test_remote_logging_s3 - AssertionError: None of the log sources ['/opt/airflow/logs/dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=bash_pull/attempt=1.log'] were found in S3 bucket logs ['s3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=bash_pull/attempt=1.log', 's3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=bash_push/attempt=1.log', 's3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=pull_value_from_bash_push/attempt=1.log', 's3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=puller/attempt=1.log', 's3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=push/attempt=1.log', 's3://test-airflow- logs/dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=push_by_returning/attempt=1.log'] assert False + where False = any(<generator object TestRemoteLogging.test_remote_logging_s3.<locals>.<genexpr> at 0x7fe6880d4c10>) =================== 1 failed, 1 passed, 1 warning in 42.88s ==================== ``` There are already 6 files in `log_files` (which _should_ be success) but the test still fail in above traceback. ## What - Remove `/opt/airflow/logs/` prefix from `task_log_sources` - `task_log_sources` format before fix: - `/opt/airflow/logs/dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=bash_pull/attempt=1.log` - `task_log_sources` format after fix: - `dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=bash_pull/attempt=1.log` - `log_files` format: - `s3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-20T03:24:32.261538+00:00/task_id=bash_pull/attempt=1.log` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
