yvesmarinho opened a new issue, #32727:
URL: https://github.com/apache/airflow/issues/32727
### Apache Airflow version
2.6.3
### What happened
We were unable to filter the file using part of the name and extension with
the match_glob option.
I'm using the apache-airflow-providers-google==10.4.0 package and I still
don't get the result as it appears in the documentation
### What you think should happen instead
In the API, the match_glob parameter is not being used.
### How to reproduce
bucketName = "test-integration-dev"
client = "test_gcp_conn_2"
filePreffix = "20230717"
extensionType = '**/*/.csv'
bucketPath = 'teste_include/input'
gcs_list_objects_operator = GCSListObjectsOperator(task_id=nId,
bucket=bucketName, prefix=bucketPath, match_glob=filePreffix + extensionType ,
gcp_conn_id=client).execute(kwargs)
expected outcome: ["20230717-Report-Custom.csv"]
### Operating System
docker
### Versions of Apache Airflow Providers
apache-airflow-providers-google==10.4.0
### Deployment
Docker-Compose
### Deployment details
Generate new image called airflow-custom:2.6.3, based on Dockerfile file
below.
Dockerfile:
FROM apache/airflow:2.6.3-python3.9
RUN pip install pip -U \
&& pip install mysql-connector-python 'apache-airflow[celery]' \
apache-airflow-providers-mysql apache-airflow-providers-slack \
apache-airflow-providers-google==10.4.0 apache-airflow-providers-sftp \
airflow-provider-rabbitmq wget pika pexpect cron-validator
docker-compose.yaml:
version: '3'
services:
rabbitmq-teste:
image: rabbitmq:3-management
hostname: rabbitmq.airflow-teste.local
container_name: airflow-rabbitmq-teste
healthcheck:
test: rabbitmq-diagnostics -q ping
interval: 30s
timeout: 30s
retries: 3
ports:
- 5673:5672
- 15673:15672
# command:
# - /bin/sh
# - -c
# - |
# echo "baeldung"
# echo "docker"
# exit 1
networks:
- app-network-teste
restart: always
volumes:
- type: bind
source: ./storage/rabbitmq1/rabbitmq.config
target: /etc/rabbitmq/conf.d/10-defautls.conf
- type: bind
source: ./storage/rabbitmq1/definitions.json
target: /etc/rabbitmq/conf.d/definitions.json
- type: bind
source: ./storage/rmq-start1.sh
target: /usr/local/bin/rmq-start.sh
- type: bind
source: ./storage/rabbitmq1
target: /var/lib/rabbitmq
environment:
- TZ='America/Sao_Paulo'
- RABBITMQ_ERLANG_COOKIE="Z3THAKa-ef+d14xu5L2e"
- RABBITMQ_ERLG_COOKIE="Z3THAKa-ef+d14xu5L2e"
- ERLANG_COOKIE="Z3THAKa-ef+d14xu5L2e"
airflow-scheduler-teste:
image: airflow-custom:2.6.3
container_name: airflow-scheduler-teste
networks:
- app-network-teste
command: scheduler
volumes:
- type: bind
source: ./airflow.cfg
target: /opt/airflow/airflow.cfg
- type: bind
source: ./dags
target: /opt/airflow/dags
- type: bind
source: ./logs
target: /opt/airflow/logs
- type: bind
source: ./plugins
target: /opt/airflow/plugins
- type: bind
source: ./tmp
target: /opt/airflow/tmp
- type: bind
source: ./utils
target: /opt/airflow/utils
- type: bind
source: ./certify
target: /opt/airflow/certify:rw
- type: bind
source: /opt
target: /opt
restart: always
user: "${AIRFLOW_UID:-50000}:0"
airflow-webserver-teste:
image: airflow-custom:2.6.3
container_name: airflow-webserver-teste
networks:
- app-network-teste
command:
- webserver
ports:
- 8081:8080
volumes:
- type: bind
source: ./airflow.cfg
target: /opt/airflow/airflow.cfg
- type: bind
source: ./dags
target: /opt/airflow/dags
- type: bind
source: ./logs
target: /opt/airflow/logs
- type: bind
source: ./plugins
target: /opt/airflow/plugins
- type: bind
source: ./tmp
target: /opt/airflow/tmp
- type: bind
source: ./utils
target: /opt/airflow/utils
- type: bind
source: ./certify
target: /opt/airflow/certify
- type: bind
source: /opt
target: /opt
restart: always
user: "${AIRFLOW_UID:-50000}:0"
healthcheck:
test:
[
"CMD",
"curl",
"--fail",
"http://localhost:8080/health";
]
interval: 10s
timeout: 10s
retries: 5
airflow-worker-teste:
image: airflow-custom:2.6.3
container_name: airflow-worker-teste
networks:
- app-network-teste
command: celery worker
volumes:
- type: bind
source: ./airflow.cfg
target: /opt/airflow/airflow.cfg
- type: bind
source: ./dags
target: /opt/airflow/dags
- type: bind
source: ./logs
target: /opt/airflow/logs
- type: bind
source: ./plugins
target: /opt/airflow/plugins
- type: bind
source: ./tmp
target: /opt/airflow/tmp
- type: bind
source: ./utils
target: /opt/airflow/utils
- type: bind
source: ./certify
target: /opt/airflow/certify
- type: bind
source: /opt
target: /opt
restart: always
user: "${AIRFLOW_UID:-50000}:0"
airflow-flower-teste:
image: airflow-custom:2.6.3
container_name: airflow-flower-teste
hostname: flower.airflow.local
networks:
- app-network-teste
command: celery flower
ports:
- 5556:5555
volumes:
- type: bind
source: ./airflow.cfg
target: /opt/airflow/airflow.cfg
- type: bind
source: ./dags
target: /opt/airflow/dags
- type: bind
source: ./logs
target: /opt/airflow/logs
- type: bind
source: ./plugins
target: /opt/airflow/plugins
- type: bind
source: ./tmp
target: /opt/airflow/tmp
- type: bind
source: ./utils
target: /opt/airflow/utils
- type: bind
source: ./certify
target: /opt/airflow/certify
- type: bind
source: /opt
target: /opt
restart: always
user: "${AIRFLOW_UID:-50000}:0"
airflow-cli-teste:
image: airflow-custom:2.6.3
container_name: airflow-cli-teste
networks:
- app-network-teste
environment:
- CONNECTION_CHECK_MAX_COUNT="0"
command:
- bash
- -c
- airflow
volumes:
- type: bind
source: ./airflow.cfg
target: /opt/airflow/airflow.cfg
- type: bind
source: ./dags
target: /opt/airflow/dags
- type: bind
source: ./logs
target: /opt/airflow/logs
- type: bind
source: ./plugins
target: /opt/airflow/plugins
- type: bind
source: ./tmp
target: /opt/airflow/tmp
- type: bind
source: ./utils
target: /opt/airflow/utils
- type: bind
source: ./certify
target: /opt/airflow/certify
- type: bind
source: /opt
target: /opt
restart: always
user: "${AIRFLOW_UID:-50000}:0"
profiles:
- debug
networks:
app-network-teste:
external: true
### Anything else
Database is hosted on another server
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
