armandleopold opened a new issue #12484:
URL: https://github.com/apache/airflow/issues/12484


   **Description**
   
   There is not enough documentation about the **execution_date** format for 
the elasticsearch **log_id** creation from custom logging pipelines.
   
   > from [Airflow Configuration 
Reference](https://airflow.apache.org/docs/stable/configurations-ref.html#log-id-template)
   
       - name: AIRFLOW__ELASTICSEARCH__LOG_ID_TEMPLATE
         value: "{dag_id}-{task_id}-{execution_date}-{try_number}"
   
   This function : 
[_clean_execution_date](https://github.com/apache/airflow/blob/master/airflow/providers/elasticsearch/log/es_task_handler.py#L123)
 declares a required format that needs to be meet in the **log_id** in order 
for airflow to fetch the logs from elasticsearch.
   
   **Use case / motivation**
   
   In a local kubernetes environment with the **KubernetesExecutor** & 
**KubernetesPodOperator**, I would like to log the logs from my pods to an 
elasticsearch from which i can fetch them to the Airflow UI.
   
   Unfortunately, everywhere in the UI , the displayed date format is not what 
is queried from Elasticsearch 
   * Displayed format : `2020-11-19T12:57:33.605561+00:00`
   * Queried format : `2020_11_19T12_57_33_605561`
   
   I have searched for so many things for so long, i have loose 1 week of work 
because of this lack of documentation for my (i admit, very specific) case.
   
   I suggess to add the required format here : [configuration 
reference](https://airflow.apache.org/docs/stable/configurations-ref.html#log-id-template)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to