[
https://issues.apache.org/jira/browse/AIRFLOW-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799108#comment-16799108
]
ASF subversion and git services commented on AIRFLOW-3737:
----------------------------------------------------------
Commit e8c395093d06946d1be5ee072116bc2994dad0a7 in airflow's branch
refs/heads/master from Paul Bramhall
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e8c3950 ]
[AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names (#4636)
Kubernetes has a 63char limit on label values, so when running either a
subdag or a dag with a name with a length longer than 63 chars, the pod
creation process will fail with the following error:
[2019-01-21 08:07:05,337] {rest.py:219} DEBUG - response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable
to parse requirement: invalid label value:
\"very_long_dag_name.very_long_task_name\": must be no more than 63
characters","reason":"BadRequest","code":400}
Annotations however, allow for a relatively unrestricted length when
storing them against pods, and as such would be better suited in this
situation.
Generate deterministic random data for dag_id and task_id
To ensure testing consistency, when calling the '_gen_random_string()'
function, we want to set the seed dependant on the string length and
iteration within the `cases.extend` loop.
This allows random values to be created (as required) but ensuring
consistency of these values during each and every time we run tests
against the code.
> Kubernetes executor cannot handle long dag/task names
> -----------------------------------------------------
>
> Key: AIRFLOW-3737
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3737
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.2
> Reporter: afusr
> Priority: Minor
> Fix For: 1.10.3
>
>
> When using the Kubernetes Executor, if the combined DAG and Task name is
> longer than 63 characters the scheduler is unable to create a pod and stops
> processing further dags. For example the logs show the following:
> (edit: our issue appears to be related to using the subdag operator, the
> string which is 67 chars is the full subdag title)
> {noformat}
> [2019-01-21 08:07:05,337] {rest.py:219} DEBUG - response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable
> to parse requirement: invalid label value:
> \"very_long_dag_name.very_long_task_name\": must be no more than 63
> characters","reason":"BadRequest","code":400}
> [2019-01-21 08:07:05,389] {dag_processing.py:579} WARNING - Ending without
> manager process.
> [2019-01-21 08:07:05,390] {jobs.py:1510} INFO - Exited execute loop
> {noformat}
> The error appears to be returned by the k8s api.
> Perhaps related to the code here?
> https://github.com/apache/airflow/blob/a7e369f082025fb089d909a6978106bb7cd24798/airflow/contrib/executors/kubernetes_executor.py#L499-L517
> This was tested using 1.10.2RC2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
