potiuk commented on code in PR #24236:
URL: https://github.com/apache/airflow/pull/24236#discussion_r894755846
##########
dev/breeze/src/airflow_breeze/commands/testing_commands.py:
##########
@@ -112,6 +129,93 @@ def docker_compose_tests(
sys.exit(return_code)
+class MonitoringThread(Thread):
+ """Thread class with a stop() method. The thread itself has to check
+ regularly for the stopped() condition."""
+
+ def __init__(self, title: str, file_name: str):
+ super().__init__(target=self.peek_percent_at_last_lines_of_file,
daemon=True)
+ self._stop_event = Event()
+ self.title = title
+ self.file_name = file_name
+
+ def peek_percent_at_last_lines_of_file(self) -> None:
+ max_line_length = 400
+ matcher = re.compile(r"^.*\[([^\]]*)\]$")
+ while not self.stopped():
+ if os.path.exists(self.file_name):
+ try:
+ with open(self.file_name, 'rb') as temp_f:
+ temp_f.seek(-(max_line_length * 2), os.SEEK_END)
Review Comment:
Beause I want to find last two lines in the file very efficiently without
reading the whole file in memory and without loosing time on reading the whole
file. from the beginning (it can be 20-30MB file so it might take few seconds
to read it and it will consume I/O and memory)
I really do not want to read the whole file to find the last two lines -
this is all I care about and I can survive if I won't succeed - at worst I will
not print progress. No big deal.. And it's generally a difficult problem to
solve, because in order to be sure you'd have to read the whole file to be
absolutely sure to print last two lines. For example if you have big file with
two lines - each of them 10000 chars, you would have to print 20000 characters
to prin the line.
So I am cheating "a bit" here. I assume the line is not longer than X. I go
to the END of the file (SEEK.END), then i go back 2 *X and read only those
maximum 2*X characters. This is much faster and takes much less memory than
reading the whole file nd being "sure" on the other hand, I KNOW our lines are
not too long in most cases and actually I only care about two last lines if
they contain Pytest output:
```
test1
................................................................................
[20%]
test2 ......................................................................
```
So I KNOW the lines I am interested at are not long. And in case I have
longer lines - too bad (the output does nothing if ti does not find the [X%] in
the output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
