Have you checked the obvious: Do you have disk space left?
If you do it's possible you've hit a limit on the number of cgroups - either cos there are too many running at the same time, or because something isn't cleaning them up. -ash > On 5 Apr 2019, at 19:44, Junda Yang <[email protected]> wrote: > > Hi there, > > Does anyone have experience in running CgroupTaskRunner? I tested it on our > staging cluster but kept getting errors: > > Traceback (most recent call last): > > File > "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/jobs.py", > line 2540, in _execute > > self.task_runner.start() > > File > "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/contrib/task_runner/cgroup_task_runner.py", > line 154, in start > > mem_cgroup_node.controller.limit_in_bytes = self._mem_mb_limit * 1024 * > 1024 > > File > "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/interfaces.py", > line 63, in __set__ > > return instance.set_property(self.filename, value) > > File > "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/controllers.py", > line 63, in set_property > > return f.write(str(value)) > > OSError: [Errno 28] No space left on device > > > Actually it succeeded in creating memory group but failed later creating > cpu group in > https://github.com/apache/airflow/blob/c3939c8e721870d263997e7aeaebc28e678d544b/airflow/contrib/task_runner/cgroup_task_runner.py#L139-#L156. > However, If I switch the order, it could create cpu group and then failed > creating memory group. So it sounds to me that it can only succeed in > creating the first cgroup configuration. I did some researches online but > relevant information is little. Any help is appreciated! > > FYI, this is the system version that our Airflow is running on: > > NAME="Ubuntu" > > VERSION="14.04.6 LTS, Trusty Tahr" > > ID=ubuntu > > ID_LIKE=debian > > PRETTY_NAME="Ubuntu 14.04.6 LTS" > > VERSION_ID="14.04" > > HOME_URL="http://www.ubuntu.com/"; > > SUPPORT_URL="http://help.ubuntu.com/"; > > Thanks, > Junda
