howardyoo commented on PR #40802: URL: https://github.com/apache/airflow/pull/40802#issuecomment-2462366769
I understand. So we can revert this no problem. In that case, would it be okay to make this done in some async way, such that it will not block the scheduler loop, but can execute to get the logs? On Thu, Nov 7, 2024 at 7:56 AM Ash Berlin-Taylor ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In airflow/jobs/scheduler_job_runner.py > <https://github.com/apache/airflow/pull/40802#discussion_r1832724193>: > > > + if conf.has_option("traces", "otel_task_log_event") and conf.getboolean( > + "traces", "otel_task_log_event" > + ): > + from airflow.utils.log.log_reader import TaskLogReader > + > + task_log_reader = TaskLogReader() > + if task_log_reader.supports_read: > + metadata: dict[str, Any] = {} > + logs, metadata = task_log_reader.read_log_chunks(ti, ti.try_number, metadata) > + if ti.hostname in dict(logs[0]): > + message = str(dict(logs[0])[ti.hostname]).replace("\\n", "\n") > + while metadata["end_of_log"] is False: > + logs, metadata = task_log_reader.read_log_chunks( > + ti, ti.try_number - 1, metadata > + ) > + if ti.hostname in dict(logs[0]): > + message = message + str(dict(logs[0])[ti.hostname]).replace("\\n", "\n") > + if span.is_recording(): > + span.add_event( > + name="task_log", > + attributes={ > + "message": message, > + "metadata": str(metadata), > + }, > + ) > > @howardyoo <https://github.com/howardyoo> @ferruzzi > <https://github.com/ferruzzi> This is a huge no-no. The scheduler cannot > do any processing that will block the main scheduling loop for so long, and > going and reading all of the logs is going to block the scheduler loop for > a noticable time. > > THis block needs reverting I'm afraid -- it is not a feature that can > exist in the scheduler. > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/airflow/pull/40802#pullrequestreview-2421054357>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHZNLLVWTA67MDPGFO4QMXDZ7NWQVAVCNFSM6AAAAABK5KFCK2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDIMRRGA2TIMZVG4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
