jason810496 commented on PR #45129: URL: https://github.com/apache/airflow/pull/45129#issuecomment-2562218329
Finally fixed the tests! This is the first (and likely the largest) PR for resolving OOM issues when reading large logs in the webserver. Further PRs will only focus on refactoring each provider, as listed in the TODO tasks in #45079. Even though the providers haven't yet been refactored to support stream-based log reading, the compatibility utility will transform the old `read` log method (which returns the entire list of logs) into a stream-based approach. Once all providers are refactored to use stream-based reading, the compatibility utility can be removed. For the testing part: Since the CI will run provider compatibility tests for versions 2.9.3 and 2.10.3, my approach is to copy the old test cases related to log reading into new stream-based tests. I’ve added the `mark_test_for_old_read_log_method` and `mark_test_for_stream_based_read_log_method` pytest decorators to selectively skip the corresponding test runs. From my perspective, this approach is simpler and minimizes changes to the original test logic. Additionally, tests marked with `mark_test_for_old_read_log_method` can be safely removed once all providers migrate to stream-based reading. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
