uranusjr opened a new pull request, #73118: URL: https://github.com/apache/airflow/pull/73118
Some AIP-85 changes I mentioned I was thinking when reviewing #72369. Discovery now filters purely by a source's identity rather than its contents. `find_file_dag_definitions` and `list_dag_definitions` match on extension and `.airflowignore`, skip `__pycache__`, and prefer a .py over a side-by-side .pyc, without reading any file or taking a safe_mode flag. ZipImporter now processes one member at a time instead of a whole archive. It enumerates individual members during discovery, applying the same `__pycache__` and .py/.pyc rules and the ZipSlip defence there, and imports a single member per call, rather than listing whole .zip files and walking their members inside `import_definition`. Modules are loaded from a definition's bytes. Source and bytecode are read through the definition with custom loaders in place of the file-backed SourceFileLoader and SourcelessFileLoader, so archive members are no longer extracted to a temporary file to be scanned or imported and .py and .pyc members load in place. Accordingly, `might_contain_dag` now takes a definition rather than a path, and PythonDagImporter is parameterized over the file and zip-member definition types it handles. Tests cover identity-only discovery, import-time skipping for both files and zip members, .py/.pyc dedup, .pyc import for a file and a member, and corrupt-bytecode handling. cc @dilnazanlid -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
