kadubhumika commented on PR #73947: URL: https://github.com/apache/airflow/pull/73947#issuecomment-5908033651
Hi I have successfully investigated, resolved, and submitted a patch for this log truncation bug. What was happening: The new Airflow 3.0+ remote logging utility path inside OpensearchRemoteLogIO.read() was executing a single query hook to _os_read(). Because _os_read() is hardcoded to a maximum page threshold window (size=1000), any task instance producing over 1,000 log records had its lines 1001+ completely dropped from the UI viewer once local logs were cleared. How it's fixed: I have refactored read() to implement an automated page-aggregating loop. Instead of requesting higher page boundaries with from_ changes (which causes deep pagination performance penalties on the OpenSearch cluster), the method now uses attrgetter to read the tracking ID of the final row in a batch. It slides the query window dynamically to pull subsequent sheets where "gt": last_seen_offset until the log index matches are fully exhausted. I also added a comprehensive mock unit test simulating a split multi-page JSON response payload (1,500 lines combined), and the regression suite passes flawlessly. Please review the PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
