[
https://issues.apache.org/jira/browse/AIRFLOW-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Davies updated AIRFLOW-2741:
--------------------------------
Description:
I'm trying to make the DAG from this blog post work:
https://www.techatbloomberg.com/blog/airflow-on-kubernetes/
But it fails with:
{code}
airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375]
{kubernetes_executor.py:579} INFO - self.running: {(u'kubernetes_sample',
u'run_this_first', datetime.datetime(2018, 7, 10, 13, 0, 17, 332338,
tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow run
kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd
/root/airflow/dags'}
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
+ airflow run kubernetes_sample run_this_first
2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:57,667] {settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=1800
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:57,919] {__init__.py:51} INFO - Using executor LocalExecutor
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:58,023] {models.py:257} INFO - Filling up the DagBag from
/root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:58,029] {models.py:348} INFO - File
/usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py assumed
to contain no DAGs. Skipping.
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
Traceback (most recent call last):
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/bin/airflow", line 32, in <module>
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
args.func(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74,
in wrapper
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
return f(*args, **kwargs)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 475,
in run
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
dag = get_dag(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 146,
in get_dag
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
'parse.'.format(args.dag_id))
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
airflow.exceptions.AirflowException: dag_id could not be found:
kubernetes_sample. Either the dag did not exist or it failed to parse.
{code}
...note that the DAG isn't in my Docker image, I just cp'ed it to the
webserver/scheduler as the blog post did in "Step 5: Upload a test document".
was:
I'm trying to make the DAG from this blog post work:
https://www.techatbloomberg.com/blog/airflow-on-kubernetes/
But it fails with:
{code}
airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375]
{kubernetes_executor.py:579} INFO - self.running: {(u'kubernetes_sample',
u'run_this_first', datetime.datetime(2018, 7, 10, 13, 0, 17, 332338,
tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow run
kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd
/root/airflow/dags'}
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
+ airflow run kubernetes_sample run_this_first
2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:57,667] {settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=1800
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:57,919] {__init__.py:51} INFO - Using executor LocalExecutor
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:58,023] {models.py:257} INFO - Filling up the DagBag from
/root/airflow/dags
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
[2018-07-10 13:00:58,029] {models.py:348} INFO - File
/usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py assumed
to contain no DAGs. Skipping.
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
Traceback (most recent call last):
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/bin/airflow", line 32, in <module>
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
args.func(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74,
in wrapper
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
return f(*args, **kwargs)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 475,
in run
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
dag = get_dag(args)
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 146,
in get_dag
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
'parse.'.format(args.dag_id))
airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
airflow.exceptions.AirflowException: dag_id could not be found:
kubernetes_sample. Either the dag did not exist or it failed to parse.
{code}
...note that the DAG isn't in my Docker image, I just cp'ed it to the
webserver/scheduler as the blog post did.
> Example Kubernetes pod operator doesn't work
> --------------------------------------------
>
> Key: AIRFLOW-2741
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2741
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Jon Davies
> Priority: Major
>
> I'm trying to make the DAG from this blog post work:
>
> https://www.techatbloomberg.com/blog/airflow-on-kubernetes/
> But it fails with:
> {code}
> airflow-test/airflow-5cc9cf9f99-v6rtd[scheduler]: [2018-07-10 13:00:41,375]
> {kubernetes_executor.py:579} INFO - self.running: {(u'kubernetes_sample',
> u'run_this_first', datetime.datetime(2018, 7, 10, 13, 0, 17, 332338,
> tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None))): u'airflow run
> kubernetes_sample run_this_first 2018-07-10T13:00:17.332338+00:00 --local -sd
> /root/airflow/dags'}
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> + airflow run kubernetes_sample run_this_first
> 2018-07-10T13:00:17.332338+00:00 --local -sd /root/airflow/dags
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> [2018-07-10 13:00:57,667] {settings.py:174} INFO - setting.configure_orm():
> Using pool settings. pool_size=5, pool_recycle=1800
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> [2018-07-10 13:00:57,919] {__init__.py:51} INFO - Using executor
> LocalExecutor
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> [2018-07-10 13:00:58,023] {models.py:257} INFO - Filling up the DagBag from
> /root/airflow/dags
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> [2018-07-10 13:00:58,029] {models.py:348} INFO - File
> /usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py
> assumed to contain no DAGs. Skipping.
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> Traceback (most recent call last):
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> File "/usr/local/bin/airflow", line 32, in <module>
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> args.func(args)
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line
> 74, in wrapper
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> return f(*args, **kwargs)
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line
> 475, in run
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> dag = get_dag(args)
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line
> 146, in get_dag
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> 'parse.'.format(args.dag_id))
> airflow-test/kubernetessamplerunthisfirst-9be9b8e6d3b84b949fdd9cc7ba3263c3[base]:
> airflow.exceptions.AirflowException: dag_id could not be found:
> kubernetes_sample. Either the dag did not exist or it failed to parse.
> {code}
> ...note that the DAG isn't in my Docker image, I just cp'ed it to the
> webserver/scheduler as the blog post did in "Step 5: Upload a test document".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
