killua1zoldyck commented on issue #31105: URL: https://github.com/apache/airflow/issues/31105#issuecomment-1627849348
> Hmm, does it mean you’re doing the stream merge on the client (browser) side? Currently the merging is happening in the server side but it reads all the logs into memory and merging them before truncating already read parts. I have changed the reading and merging based on what was already read and this we get using the metadata. > I haven’t seen this API before but I wonder if there’s a way to supplement the full_content: false param with a reverse: true? I don't think it has a reverse: true parameter. https://github.com/apache/airflow/blob/main/airflow/api_connexion/endpoints/log_endpoint.py . If we know the total character length of our logs before hand we can send that as a token. This too would read the whole file and then truncate the end so the memory would still be an issue. > Is your goal with the updated UI to support full pagination through all the logs, or just show the most recent ~1,000 lines? I was thinking full pagination through all the logs till the end. The one I have implemented now does that by reading 50 lines from a file at a time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
