Jon Davies created AIRFLOW-2741:
-----------------------------------
Summary: Example Kubernetes pod operator doesn't work
Key: AIRFLOW-2741
URL: https://issues.apache.org/jira/browse/AIRFLOW-2741
Project: Apache Airflow
Issue Type: Bug
Reporter: Jon Davies
I'm trying to make the DAG from this blog post work:
https://www.techatbloomberg.com/blog/airflow-on-kubernetes/
But it fails with:
{code}
airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375]
{kubernetes_executor.py:579} INFO - self.running: {(u'kubernetes_sample',
u'run_this_first', datetime.datetime(2018, 7, 10, 13, 0, 17, 332338,
tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow run
kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd
/root/airflow/dags'}
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
+ airflow run kubernetes_sample run_this_first
2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:57,667] {settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=1800
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:57,919] {__init__.py:51} INFO - Using executor LocalExecutor
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:58,023] {models.py:257} INFO - Filling up the DagBag from
/root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:58,029] {models.py:348} INFO - File
/usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py assumed
to contain no DAGs. Skipping.
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
Traceback (most recent call last):
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/bin/airflow", line 32, in <module>
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
args.func(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74,
in wrapper
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
return f(*args, **kwargs)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 475,
in run
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
dag = get_dag(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 146,
in get_dag
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
'parse.'.format(args.dag_id))
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
airflow.exceptions.AirflowException: dag_id could not be found:
kubernetes_sample. Either the dag did not exist or it failed to parse.
{code}
...note that the DAG isn't in my Docker image, I just cp'ed it to the
webserver/scheduler as the blog post did.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
