amoghrajesh commented on code in PR #70513:
URL: https://github.com/apache/airflow/pull/70513#discussion_r3657365829
##########
providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -319,9 +324,8 @@ def __init__(
self._driver_id: str | None = None
self._driver_status: str | None = None
self._spark_exit_code: int | None = None
- # Last few lines of the spark-submit process's own stdout/stderr, so
failure
- # exceptions can include the actual root cause instead of just an exit
code.
- self._last_submit_log_lines: deque[str] = deque(maxlen=20)
+ self._last_submit_log_lines: deque[str] = deque(maxlen=500)
Review Comment:
Default reverted to `deque(maxlen=20)` as per comment above and added a test
to actually exercise the 500-line cap once the anchor fires and confirms it
caps and evicts from the front like any bounded deque, rather than being an
untested number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
