nabuskey opened a new issue, #57176: URL: https://github.com/apache/airflow/issues/57176
### Apache Airflow version 3.1.0 ### If "Other Airflow 2/3 version" selected, which one? _No response_ ### What happened? When I configure DAG bundle from S3 according to the documentation, it fails with `IsADirectoryError`. This occurs because it does not ignore prefix objects in https://github.com/apache/airflow/blob/44f8479b1597e8882c8985c601b4e40abbc024bb/providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py#L1776-L1785 `bucket.objects()` returns prefixes as well. ``` >>> objects = bucket.objects.filter(Prefix='dags/') >>> for obj in objects: ... print(obj) ... s3.ObjectSummary(bucket_name='data-on-eks-airflow-20251023154905245200000001', key='dags/') s3.ObjectSummary(bucket_name='data-on-eks-airflow-20251023154905245200000001', key='dags/simple.py') ``` ### What you think should happen instead? It should be able to retrieve DAGs without errors. ### How to reproduce Setup standalone: ```bash uv pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" uv pip install apache-airflow-providers-amazon ``` Update `airflow.cfg` to use S3 as a source for DAG bundles ``` [dag_processor] dag_bundle_config_list = [ { "name": "dags-s3", "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle", "kwargs": { "bucket_name": "some-bucket", "prefix": "dags/" } } ] ``` Put DAGs in s3: ``` aws s3 ls --recursive data-on-eks-airflow-20251023154905245200000001/dags/ 2025-10-23 15:49:08 0 dags/ 2025-10-23 16:18:07 1000 dags/simple.py 2025-10-23 18:54:56 0 dags/test1/ 2025-10-23 18:56:06 1007 dags/test1/simple-test.py ``` Run airflow: ``` airflow standalone ``` Fails with: ``` dag-processor | File "/tmp/airflow/.venv/lib/python3.13/site-packages/s3transfer/utils.py", line 299, in rename_file dag-processor | rename_file(current_filename, new_filename) dag-processor | ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ dag-processor | IsADirectoryError: [Errno 21] Is a directory: '/tmp/airflow/dag_bundles/dags-s3.64d6feDD' -> '/tmp/airflow/dag_bundles/dags-s3' ``` ### Operating System Ubuntu 24.04 ### Versions of Apache Airflow Providers ``` apache-airflow-providers-amazon 9.15.0 apache-airflow-providers-common-compat 1.7.4 apache-airflow-providers-common-io 1.6.3 apache-airflow-providers-common-sql 1.28.1 apache-airflow-providers-http 5.3.4 apache-airflow-providers-smtp 2.2.1 apache-airflow-providers-standard 1.8.0 ``` it fails with: ``` dag-processor | IsADirectoryError: [Errno 21] Is a directory: '/tmp/airflow/dag_bundles/dags-s3.64d6feDD' -> '/tmp/airflow/dag_bundles/dags-s3' ``` ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
