Hi there,
Does anyone have experience in running CgroupTaskRunner? I tested it on our
staging cluster but kept getting errors:
Traceback (most recent call last):
File
"/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/jobs.py",
line 2540, in _execute
self.task_runner.start()
File
"/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/contrib/task_runner/cgroup_task_runner.py",
line 154, in start
mem_cgroup_node.controller.limit_in_bytes = self._mem_mb_limit * 1024 *
1024
File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/interfaces.py",
line 63, in __set__
return instance.set_property(self.filename, value)
File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/controllers.py",
line 63, in set_property
return f.write(str(value))
OSError: [Errno 28] No space left on device
Actually it succeeded in creating memory group but failed later creating
cpu group in
https://github.com/apache/airflow/blob/c3939c8e721870d263997e7aeaebc28e678d544b/airflow/contrib/task_runner/cgroup_task_runner.py#L139-#L156.
However, If I switch the order, it could create cpu group and then failed
creating memory group. So it sounds to me that it can only succeed in
creating the first cgroup configuration. I did some researches online but
relevant information is little. Any help is appreciated!
FYI, this is the system version that our Airflow is running on:
NAME="Ubuntu"
VERSION="14.04.6 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.6 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/";
SUPPORT_URL="http://help.ubuntu.com/";
Thanks,
Junda
