[ https://issues.apache.org/jira/browse/AIRFLOW-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Riccomini resolved AIRFLOW-1696. -------------------------------------- Resolution: Fixed > GCP Dataproc Operator fails due to '+' in Airflow version > --------------------------------------------------------- > > Key: AIRFLOW-1696 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1696 > Project: Apache Airflow > Issue Type: Bug > Reporter: Trevor Edwards > Assignee: Trevor Edwards > Fix For: 1.10.0 > > > Since Dataproc operator attaches the Airflow version as one of its labels, > and dataproc labels cannot include the character '+', dataproc operator > currently fails with the following error: > {code:none} > [2017-10-09 19:28:48,035] {base_task_runner.py:115} INFO - Running: ['bash', > '-c', u'airflow run smokey-dataproc start_cluster 2017-10-08T00:00:00 > --job_id 6 --raw -sd DAGS_FOLDER/dataa.py'] > [2017-10-09 19:28:49,041] {base_task_runner.py:98} INFO - Subtask: > [2017-10-09 19:28:49,041] {__init__.py:45} INFO - Using executor LocalExecutor > [2017-10-09 19:28:49,139] {base_task_runner.py:98} INFO - Subtask: > [2017-10-09 19:28:49,139] {models.py:187} INFO - Filling up the DagBag from > /home/airflow/dags/dataa.py > [2017-10-09 19:28:49,258] {base_task_runner.py:98} INFO - Subtask: Cluster > name: smoke-cluster-aa05845a-1b60-4543-94d0-f7dfddb90ee0 > [2017-10-09 19:28:49,258] {base_task_runner.py:98} INFO - Subtask: > [2017-10-09 19:28:49,257] {dataproc_operator.py:267} INFO - Creating cluster: > smoke-cluster-aa05845a-1b60-4543-94d0-f7dfddb90ee0 > [2017-10-09 19:28:49,265] {base_task_runner.py:98} INFO - Subtask: > [2017-10-09 19:28:49,265] {gcp_api_base_hook.py:82} INFO - Getting connection > using a JSON key file. > [2017-10-09 19:28:59,909] {base_task_runner.py:98} INFO - Subtask: > [2017-10-09 19:28:59,906] {models.py:1564} ERROR - <HttpError 400 when > requesting > https://dataproc.googleapis.com/v1/projects/cloud-airflow-test/regions/global/clusters?alt=json > returned "Multiple validation errors: > [2017-10-09 19:28:59,910] {base_task_runner.py:98} INFO - Subtask: - Not a > valid value: "v1-10-0dev0+incubating". Only lowercase letters, numbers, and > dashes are allowed. The value must start with lowercase letter or number and > end with a lowercase letter or number. > [2017-10-09 19:28:59,910] {base_task_runner.py:98} INFO - Subtask: - User > label value must conform to '[\p{Ll}\p{Lo}\p{N}_-]{0,63}' pattern"> > [2017-10-09 19:28:59,911] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2017-10-09 19:28:59,911] {base_task_runner.py:98} INFO - Subtask: File > "/home/airflow/incubator-airflow/airflow/models.py", line 1462, in > _run_raw_task > [2017-10-09 19:28:59,911] {base_task_runner.py:98} INFO - Subtask: result > = task_copy.execute(context=context) > [2017-10-09 19:28:59,912] {base_task_runner.py:98} INFO - Subtask: File > "/home/airflow/incubator-airflow/airflow/contrib/operators/dataproc_operator.py", > line 300, in execute > [2017-10-09 19:28:59,912] {base_task_runner.py:98} INFO - Subtask: raise e > [2017-10-09 19:28:59,913] {base_task_runner.py:98} INFO - Subtask: HttpError: > <HttpError 400 when requesting > https://dataproc.googleapis.com/v1/projects/cloud-airflow-test/regions/global/clusters?alt=json > returned "Multiple validation errors: > [2017-10-09 19:28:59,914] {base_task_runner.py:98} INFO - Subtask: - Not a > valid value: "v1-10-0dev0+incubating". Only lowercase letters, numbers, and > dashes are allowed. The value must start with lowercase letter or number and > end with a lowercase letter or number. > [2017-10-09 19:28:59,914] {base_task_runner.py:98} INFO - Subtask: - User > label value must conform to '[\p{Ll}\p{Lo}\p{N}_-]{0,63}' pattern"> > [2017-10-09 19:28:59,915] {base_task_runner.py:98} INFO - Subtask: > [2017-10-09 19:28:59,909] {models.py:1585} INFO - Marking task as UP_FOR_RETRY > [2017-10-09 19:28:59,978] {base_task_runner.py:98} INFO - Subtask: > [2017-10-09 19:28:59,977] {models.py:1613} ERROR - <HttpError 400 when > requesting > https://dataproc.googleapis.com/v1/projects/cloud-airflow-test/regions/global/clusters?alt=json > returned "Multiple validation errors: > [2017-10-09 19:28:59,978] {base_task_runner.py:98} INFO - Subtask: - Not a > valid value: "v1-10-0dev0+incubating". Only lowercase letters, numbers, and > dashes are allowed. The value must start with lowercase letter or number and > end with a lowercase letter or number. > [2017-10-09 19:28:59,979] {base_task_runner.py:98} INFO - Subtask: - User > label value must conform to '[\p{Ll}\p{Lo}\p{N}_-]{0,63}' pattern"> > [2017-10-09 19:28:59,979] {base_task_runner.py:98} INFO - Subtask: Traceback > (most recent call last): > [2017-10-09 19:28:59,979] {base_task_runner.py:98} INFO - Subtask: File > "/usr/local/bin/airflow", line 6, in <module> > [2017-10-09 19:28:59,980] {base_task_runner.py:98} INFO - Subtask: > exec(compile(open(__file__).read(), __file__, 'exec')) > [2017-10-09 19:28:59,980] {base_task_runner.py:98} INFO - Subtask: File > "/home/airflow/incubator-airflow/airflow/bin/airflow", line 27, in <module> > [2017-10-09 19:28:59,981] {base_task_runner.py:98} INFO - Subtask: > args.func(args) > [2017-10-09 19:28:59,981] {base_task_runner.py:98} INFO - Subtask: File > "/home/airflow/incubator-airflow/airflow/bin/cli.py", line 394, in run > [2017-10-09 19:28:59,982] {base_task_runner.py:98} INFO - Subtask: > pool=args.pool, > [2017-10-09 19:28:59,982] {base_task_runner.py:98} INFO - Subtask: File > "/home/airflow/incubator-airflow/airflow/utils/db.py", line 50, in wrapper > [2017-10-09 19:28:59,982] {base_task_runner.py:98} INFO - Subtask: result > = func(*args, **kwargs) > [2017-10-09 19:28:59,983] {base_task_runner.py:98} INFO - Subtask: File > "/home/airflow/incubator-airflow/airflow/models.py", line 1462, in > _run_raw_task > [2017-10-09 19:28:59,983] {base_task_runner.py:98} INFO - Subtask: result > = task_copy.execute(context=context) > [2017-10-09 19:28:59,984] {base_task_runner.py:98} INFO - Subtask: File > "/home/airflow/incubator-airflow/airflow/contrib/operators/dataproc_operator.py", > line 300, in execute > [2017-10-09 19:28:59,984] {base_task_runner.py:98} INFO - Subtask: raise e > [2017-10-09 19:28:59,984] {base_task_runner.py:98} INFO - Subtask: > googleapiclient.errors.HttpError: <HttpError 400 when requesting > https://dataproc.googleapis.com/v1/projects/cloud-airflow-test/regions/global/clusters?alt=json > returned "Multiple validation errors: > [2017-10-09 19:28:59,985] {base_task_runner.py:98} INFO - Subtask: - Not a > valid value: "v1-10-0dev0+incubating". Only lowercase letters, numbers, and > dashes are allowed. The value must start with lowercase letter or number and > end with a lowercase letter or number. > [2017-10-09 19:28:59,985] {base_task_runner.py:98} INFO - Subtask: - User > label value must conform to '[\p{Ll}\p{Lo}\p{N}_-]{0,63}' pattern"> > {code} > A fix similar to > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L243 > may be applied. -- This message was sent by Atlassian JIRA (v6.4.14#64029)