seanghaeli commented on issue #76640:
URL: https://github.com/apache/airflow/issues/76640#issuecomment-6965274871

   Once `upload()` reports the result, the triggerer's watcher log files 
(#76527) can use it in five places that guess today, in 
`airflow/utils/log/watcher_log.py` and `airflow/jobs/triggerer_job_runner.py`:
   
   - 
[`WatcherLogs.upload_rotated`](https://github.com/apache/airflow/blob/426dafc3e73028d5544345b847a64c0ef940842d/airflow-core/src/airflow/utils/log/watcher_log.py#L329):
 a rotated file is sent again once 
([`WatcherLogs._upload_rotated_again`](https://github.com/apache/airflow/blob/426dafc3e73028d5544345b847a64c0ef940842d/airflow-core/src/airflow/utils/log/watcher_log.py#L320),
 same file) if its upload raised, except with OSS, or if S3, GCS, WASB or OSS 
left its lines.
   - 
[`WatcherLogs._upload_orphaned_trigger_log_dir`](https://github.com/apache/airflow/blob/426dafc3e73028d5544345b847a64c0ef940842d/airflow-core/src/airflow/utils/log/watcher_log.py#L450):
 the same rule for the files a dead or stopped triggerer left behind; those it 
judges failed stay for the next triggerer that starts on that disk.
   - 
[`WatcherLogs.upload_rotated`](https://github.com/apache/airflow/blob/426dafc3e73028d5544345b847a64c0ef940842d/airflow-core/src/airflow/utils/log/watcher_log.py#L329)
 again: the next rotation deletes the previous file after its upload and one 
retry, whether or not it was stored.
   - 
[`WatcherLoggingFactory.upload_to_remote`](https://github.com/apache/airflow/blob/426dafc3e73028d5544345b847a64c0ef940842d/airflow-core/src/airflow/jobs/triggerer_job_runner.py#L638)
 / 
[`WatcherLogs.upload_and_close`](https://github.com/apache/airflow/blob/426dafc3e73028d5544345b847a64c0ef940842d/airflow-core/src/airflow/utils/log/watcher_log.py#L372):
 the upload when a watcher leaves the triggerer, or when the triggerer stops, 
isn't retried. A file it failed to store keeps its lines, so they go with that 
watcher's next upload if it comes back to this triggerer, or else to the next 
triggerer that starts on that disk.
   - 
[`upload_watcher_log_file`](https://github.com/apache/airflow/blob/426dafc3e73028d5544345b847a64c0ef940842d/airflow-core/src/airflow/utils/log/watcher_log.py#L183):
 `OSSRemoteLogIO.upload` drops the result of `oss_write`, so for OSS it calls 
`oss_write` directly and empties the file only when the write succeeded. Once 
`upload()` reports success, replace that direct `oss_write` call with 
`upload()`'s result.
   
   These pick the store by handler class name 
([`_EMPTIES_STORED_FILE_HANDLERS`](https://github.com/apache/airflow/blob/426dafc3e73028d5544345b847a64c0ef940842d/airflow-core/src/airflow/utils/log/watcher_log.py#L148),
 
[`_LINES_LEFT_MEAN_FAILED_HANDLERS`](https://github.com/apache/airflow/blob/426dafc3e73028d5544345b847a64c0ef940842d/airflow-core/src/airflow/utils/log/watcher_log.py#L152)
 and the lists next to them). That has to stay as the fallback for providers 
that don't return a result yet.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to