minyeamer commented on PR #71558:
URL: https://github.com/apache/airflow/pull/71558#issuecomment-5463332406
@bbovenzi
Thanks for the detailed review. I interpreted the four main concerns as
scalability and
data-flow issues rather than only UI adjustments.
Since b13f993, excluding merge commits, I made the following changes:
- Replaced client-side pagination and request fan-out with a dedicated
streaming endpoint.
- Removed the unbounded "All Dag runs" option and enforced a maximum of
5,000 runs on the backend.
- Moved Dag and Dag run filtering into the backend query construction.
- Aggregate only the selected Day or Week view on the server.
- Stream results in batches so the UI can render progressively.
- Updated the filter implementation after the shared FilterBar changes were
merged.
- Kept the private OpenAPI contract and generated client aligned with the
backend route.
The current flow is:
```mermaid
flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
```
The browser now receives only the filtered data needed for the selected
view, while the backend controls the result size and aggregation cost.
I initially intended this change to remain UI-only and did not plan to add a
backend API. Because the review identified the scalability limitations of the
original approach, I needed to first understand Airflow's backend API
structure, streaming session lifecycle, authorization filters, and OpenAPI
generation flow. I also spent additional time reviewing the AI-assisted code
carefully, which is why the follow-up changes took about 5 days.
The implementation is now focused on the production-scale concerns from the
review while preserving the existing Airflow UI/API patterns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
