I tested master branch by putting the following in my requirements.txt:
git+https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
and did a pip install -r requirements.txt
When I started the airflow webserver, I saw deprecation warnings. I
put some additional debugging in models.py to through an exception so that I
could see the
full stacktrace:
[2018-05-29 14:00:34,419] {models.py:307} ERROR - Failed to import:
/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/example_dags/example_kubernetes_operator.py
Traceback (most recent call last):
File
"/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py", line
304, in process_file
m = imp.load_source(mod_name, filepath)
File
"/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/example_dags/example_kubernetes_operator.py",
line 53, in <module>
dag=dag)
File
"/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/utils/decorators.py",
line 98, in wrapper
result = func(*args, **kwargs)
File
"/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py", line
2308, in __init__
raise Exception("Invalid use of args or kwargs")
Exception: Invalid use of args or kwargs
If looks like example_kubernetes_operator.py, this code is the source of the
exception:
k = KubernetesPodOperator(
namespace='default',
image="ubuntu:16.04",
cmds=["bash", "-cx"],
arguments=["echo", "10"],
labels={"foo": "bar"},
name="airflow-test-pod",
in_cluster=False,
task_id="task",
get_logs=True,
dag=dag)
Without my extra debugging, the deprecation warning looks like this:
[2018-05-29 14:06:27,567] {example_kubernetes_operator.py:30} WARNING - Could
not import KubernetesPodOperator
/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py:2315:
PendingDeprecationWarning: Invalid arguments were passed to BaseOperator.
Support for passing such arguments will be dropped in Airflow 2.0. Invalid
arguments were:
*args: ()
**kwargs: {'name': 'airflow-test-pod', 'image': 'ubuntu:16.04', 'labels':
{'foo': 'bar'}, 'namespace': 'default', 'cmds': ['bash', '-cx'], 'arguments':
['echo', '10'], 'in_cluster': False, 'get_logs': True}
category=PendingDeprecationWarning
What is the correct fix for this? It looks like a lot of operators pass in
arguments which are not
processed by BaseOperator, and thus trip over this deprecation warning.
--
Craig
