bhavesh200628-debug opened a new pull request, #62633: URL: https://github.com/apache/airflow/pull/62633
Description Closes #62622 Currently, _sync_to_local_dir_delete_stale_local_files in the S3 hook uses Path.iterdir(), which only iterates through immediate children. If a user has a nested DAG structure (e.g., dags/team_a/my_dag.py), stale files inside those subfolders are ignored and never deleted. This PR replaces the flat iteration with a recursive helper function that: Traverses subdirectories bottom-up – ensures nested files are handled before their parent folders. Deletes stale files – removes items not present in the resolved current_s3_keys set. Safely cleans up subdirectories – removes folders only once they become empty. Testing Ensured backward compatibility by guarding with local_dir.exists(). Verified that only empty directories are removed. Confirmed Path.resolve() is used to handle path comparisons accurately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
