Yeah. There is enough disk space left on the machine. The latter theory sounds reasonable. Going to verify that. Thanks Ash!
> On Apr 5, 2019, at 11:47, Ash Berlin-Taylor <[email protected]> wrote: > > Have you checked the obvious: > > Do you have disk space left? > > If you do it's possible you've hit a limit on the number of cgroups - either > cos there are too many running at the same time, or because something isn't > cleaning them up. > > -ash > >> On 5 Apr 2019, at 19:44, Junda Yang <[email protected]> wrote: >> >> Hi there, >> >> Does anyone have experience in running CgroupTaskRunner? I tested it on our >> staging cluster but kept getting errors: >> >> Traceback (most recent call last): >> >> File >> "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/jobs.py", >> line 2540, in _execute >> >> self.task_runner.start() >> >> File >> "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/contrib/task_runner/cgroup_task_runner.py", >> line 154, in start >> >> mem_cgroup_node.controller.limit_in_bytes = self._mem_mb_limit * 1024 * >> 1024 >> >> File >> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/interfaces.py", >> line 63, in __set__ >> >> return instance.set_property(self.filename, value) >> >> File >> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/controllers.py", >> line 63, in set_property >> >> return f.write(str(value)) >> >> OSError: [Errno 28] No space left on device >> >> >> Actually it succeeded in creating memory group but failed later creating >> cpu group in >> https://github.com/apache/airflow/blob/c3939c8e721870d263997e7aeaebc28e678d544b/airflow/contrib/task_runner/cgroup_task_runner.py#L139-#L156. >> However, If I switch the order, it could create cpu group and then failed >> creating memory group. So it sounds to me that it can only succeed in >> creating the first cgroup configuration. I did some researches online but >> relevant information is little. Any help is appreciated! >> >> FYI, this is the system version that our Airflow is running on: >> >> NAME="Ubuntu" >> >> VERSION="14.04.6 LTS, Trusty Tahr" >> >> ID=ubuntu >> >> ID_LIKE=debian >> >> PRETTY_NAME="Ubuntu 14.04.6 LTS" >> >> VERSION_ID="14.04" >> >> HOME_URL="http://www.ubuntu.com/"; >> >> SUPPORT_URL="http://help.ubuntu.com/"; >> >> Thanks, >> Junda >
