vandonr-amz opened a new pull request, #30886: URL: https://github.com/apache/airflow/pull/30886
there was several problems with the current implementation: - it was only fetching 1 response from the logs, so if the job was producing a lot of logs, it'd lag further and further back in time (and possibly never catch up) - if the job was marked as failed, it was jumping from one stream to an other without warning, and started filtering on exceptions and errors, providing a very weird experience to someone who'd use those to debug - upon changing log streams, the continuation token was not reset, so that behavior was not working anyway (you cannot use a continuation token from a stream when reading from another) - upon checking the different log streams available (/output and /error), I realized that it was not clear cut at all. A stack trace could go in /output, and some INFO logs were in /error, so I think it has value to have both streams if one wants to understand what's happening. Regarding the fact that we now display both streams, I hesitated between interleaving messages from both, sorting by timestamp, or leaving them separated. I ended up choosing to have them separated to keep the experience consistent with what the user would see in cloudwatch, but I'd be happy to change that to chronological order if people think it's better. Also: added it to the system test (for better visibility for users) + added utest cc @ferruzzi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
