potiuk commented on issue #46657: URL: https://github.com/apache/airflow/issues/46657#issuecomment-2663288463
> From my perspective, this shouldn't be a major issue (at least from the API side; I'm unsure about the UI side). The task_log_reader.read_log_stream method is actually a generator, which is also how the legacy UI retrieves logs from the text/plain log response. BTW. We've also discussed it before and **maybe** some optimizations can be (I am repeating myself) using some of the existing json parsers. There are a few - notably the super fast rust jiter which is part of Pydantic and offers partial json parsing / with streaming https://docs.rs/jiter/latest/jiter/. This has the feature of being able to parse incomplete JSON so we could chunk logs and use RANGE HTTP requests very efficiently without loading whole log in memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
