lohithasree opened a new issue, #32044:
URL: https://github.com/apache/airflow/issues/32044
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
We migrated from Local Executor to Celery Executor where using RabbitMQ as a
broker.
With Local executor, remote logging is working as expected. Able to read logs
**Airflow Version: 2.6.1
Celery Version: 5.3.0**
apache-airflow-providers-common-sql==1.1.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-postgres==5.2.0
apache-airflow-providers-celery==3.2.0
**Airflow Config for Remote Logging:**
base_log_folder = /opt/airflow/logs
remote_logging = True
remote_log_conn_id = google_cloud_default
delete_local_logs = True
google_key_path =
remote_base_log_folder = gs://bucket-name/logs
remote_task_handler_kwargs =
encrypt_s3_logs = False
logging_level = INFO
celery_logging_level =
fab_logging_level = WARNING
logging_config_class =
colored_console_log = False
colored_log_format = [%(blue)s%(asctime)s%(reset)s]
{%(blue)s%(filename)s:%(reset)s%(lineno)d} %(log_color)s%(levelname)s%(reset)s
- %(log_color)s%(message)s%(reset)s
colored_formatter_class =
airflow.utils.log.colored_log.CustomTTYColoredFormatter
log_format = [%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s -
%(message)s
simple_log_format = %(asctime)s %(levelname)s - %(message)s
dag_processor_log_target = file
dag_processor_log_format = [%(asctime)s] [SOURCE:DAG_PROCESSOR]
{%(filename)s:%(lineno)d} %(levelname)s - %(message)s
log_formatter_class = airflow.utils.log.timezone_aware.TimezoneAware
secret_mask_adapter =
task_log_prefix_template =
log_filename_template = dag_id={{ ti.dag_id }}/run_id={{ ti.run_id
}}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index
}}/{% endif %}attempt={{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
dag_processor_manager_log_location =
/opt/airflow/logs/dag_processor_manager/dag_processor_manager.log
task_log_reader = task
extra_logger_names =
worker_log_server_port = 8793
trigger_log_server_port = 8794
file_task_handler_new_folder_permissions = 0o775
file_task_handler_new_file_permissions = 0o664
when we trigger dags, dags are going into queue state and then failed state
by stating unable to read the remote logs like below error.
**Celery Config:**
celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_prefetch_multiplier = 1
worker_enable_remote_control = True
broker_url =
amqps://username:[email protected]:5671/host_name;amqps://username:[email protected]:5671/host_name;amqps://username:[email protected]:5671/host_name
flower_host = 0.0.0.0
flower_url_prefix = flower
flower_port = 5555
flower_basic_auth = QW3Q3uWPJqloOhEWv:pvmGvSATFxBMuD4El
sync_parallelism = 0
celery_config_options =
airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
ssl_active = True
ssl_key = /opt/airflow/cert/ddsdev-airflow.uk.hsbc.key
ssl_cert = /opt/airflow/cert/ddsdev-airflow.uk.hsbc.crt
ssl_cacert =
pool = prefork
operation_timeout = 1.0
task_track_started = True
task_publish_max_retries = 3
worker_precheck = False
result_backend = db+postgresql://username:password@localhost:5432/database
Accessing Airflow webserver and Flower UI using NGINX reverse proxy
***** Unable to read remote log from
gs://**********/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log
*** 404 GET
https://storage.googleapis.com/download/storage/v1/b/*********/o/logs%2Fdag_id%3DDDS_catalog_update%2Frun_id%3Dmanual__2023-06-20T15%3A22%3A58.216561%2B00%3A00%2Ftask_id%3Dinsert_dds_ct_start_ts%2Fattempt%3D1.log?alt=media:
No such object:
*********88/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log:
('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK:
200>, <HTTPStatus.PARTIAL_CONTENT: 206>) *** Could not read served logs:
Request URL is missing an 'http://' or 'https://' protocol.**
We have verified the object path manually, Its existing. But from airflow we
are getting error like above.
We have verified all the permissions and config.
Can you please help us to resolve.
### What you think should happen instead
I expected I could read the task log on UI
### How to reproduce
While I run any DAGs, the log always shows up this.
***** Unable to read remote log from
gs://**********/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log
*** 404 GET
https://storage.googleapis.com/download/storage/v1/b/*********/o/logs%2Fdag_id%3DDDS_catalog_update%2Frun_id%3Dmanual__2023-06-20T15%3A22%3A58.216561%2B00%3A00%2Ftask_id%3Dinsert_dds_ct_start_ts%2Fattempt%3D1.log?alt=media:
No such object:
*********88/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log:
('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK:
200>, <HTTPStatus.PARTIAL_CONTENT: 206>) *** Could not read served logs:
Request URL is missing an 'http://' or 'https://' protocol.**
The log files have already exists on worker and store on GCS.
### Operating System
RHEL 7
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
