KevinYang21 commented on a change in pull request #7141: [AIRFLOW-6544] add
log_id to end-of-file mark and also add an index config for logs
URL: https://github.com/apache/airflow/pull/7141#discussion_r367675716
##########
File path: airflow/utils/log/es_task_handler.py
##########
@@ -255,7 +256,9 @@ def close(self):
# Mark the end of file using end of log mark,
# so we know where to stop while auto-tailing.
- self.handler.stream.write(self.end_of_log_mark)
+ if self.write_stdout:
+ print()
+ self.handler.emit(logging.makeLogRecord({'msg': self.end_of_log_mark}))
Review comment:
It means the last line would be something like `[2020-01-16 07:58:32,712]
{es_task_handler.py:XXX} INFO [end_of_log_mark]` and thus made the reader
unable to understand it.
I'm a bit lost how did this removed the log id from the end_of_log_mark.
Isn't the log_id we constructed in this file only for log fetching? My
understanding is that the log_id is determined when we upload the log, e.g.
when we pipe stdout to logstash or when we upload file through filebeat to
logstash.
Maybe I was understanding this wrong and there is indeed a bug. In that case
I would agree on spliting this change into two PRs for sanity purpose.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
