uplsh580 commented on PR #61171: URL: https://github.com/apache/airflow/pull/61171#issuecomment-3812890789
## Performance Consideration The current implementation filters import errors in-memory after fetching paginated results. This ensures we include import errors from files that didn't create any DAGs. **Trade-offs:** **Current approach (in-memory filtering):** - ✅ Simpler query logic - ✅ Works correctly for the use case - ⚠️ May fetch more data than needed if most import errors are filtered out - ⚠️ Pagination may return fewer items than requested (but still accurate for UI) **Alternative approach (DB-level filtering with LEFT JOIN + EXISTS):** - ✅ Only fetches authorized items - ✅ More accurate pagination - ⚠️ More complex query - ⚠️ Requires indexes on `(bundle_name, relative_fileloc)` and `(bundle_name, dag_id)` for optimal performance **Questions for reviewers:** - Does the in-memory filtering approach align with similar patterns in the codebase? - Are there any concerns with the current implementation approach? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
