Hi all,
I have strange error since last week on my koji. All regen-repo tasks are
failed with error LockError: [Errno 37] No locks available
Koji works perfectly since January month. Directory /mnt/koji is a nfs mount
but i can touch a new file on it. Database is ok too.
I have two builders, and error appear on both.
Any idea ?
== Client log ==
koji regen-repo centos-6-build
Regenerating repo for tag centos-6-build
Watching tasks (this may be safely interrupted)...
4888 newRepo (centos-6-build): free
4888 newRepo (centos-6-build): free -> open (builder-vbox2.b2pweb.com)
4889 createrepo (x86_64): free
4890 createrepo (i386): free
4889 createrepo (x86_64): free -> open (builder-vbox2.b2pweb.com)
4890 createrepo (i386): free -> open (builder-vbox2.b2pweb.com)
4889 createrepo (x86_64): open (builder-vbox2.b2pweb.com) -> FAILED:
LockError: [Errno 37] No locks available
0 free 2 open 0 done 1 failed
4888 newRepo (centos-6-build): open (builder-vbox2.b2pweb.com) -> FAILED:
LockError: [Errno 37] No locks available
0 free 1 open 0 done 2 failed
4890 createrepo (i386): open (builder-vbox2.b2pweb.com) -> FAILED:
LockError: [Errno 37] No locks available
0 free 0 open 0 done 3 failed
4888 newRepo (centos-6-build) failed
== Builder log ==
/usr/sbin/kojid --verbose --fg
2013-11-15 14:46:47,576 [INFO] {12485} koji.build:73 Starting up
2013-11-15 14:47:03,363 [INFO] {12485} koji.TaskManager:1078 Attempting to
take task 4891
2013-11-15 14:47:03,711 [INFO] {12485} koji.TaskManager:679 pids: {4891:
12568}
2013-11-15 14:47:03,775 [INFO] {12485} koji.TaskManager:681 open task:
{'waiting': None, 'id': 4891, 'weight': 0.10000000000000001}
2013-11-15 14:47:19,053 [INFO] {12485} koji.TaskManager:679 pids: {4891:
12568}
2013-11-15 14:47:19,117 [INFO] {12485} koji.TaskManager:681 open task:
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001}
2013-11-15 14:47:19,312 [INFO] {12485} koji.TaskManager:1078 Attempting to
take task 4892
2013-11-15 14:47:19,662 [INFO] {12485} koji.TaskManager:679 pids: {4891:
12568, 4892: 12636}
2013-11-15 14:47:19,727 [INFO] {12485} koji.TaskManager:681 open task:
{'waiting': None, 'id': 4892, 'weight': 1.5}
2013-11-15 14:47:19,728 [INFO] {12485} koji.TaskManager:681 open task:
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001}
2013-11-15 14:47:19,925 [INFO] {12485} koji.TaskManager:1078 Attempting to
take task 4893
2013-11-15 14:47:20,277 [INFO] {12485} koji.TaskManager:679 pids: {4891:
12568, 4892: 12636, 4893: 12638}
2013-11-15 14:47:20,342 [INFO] {12485} koji.TaskManager:681 open task:
{'waiting': None, 'id': 4893, 'weight': 1.5}
2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task:
{'waiting': None, 'id': 4892, 'weight': 1.5}
2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task:
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001}
2013-11-15 14:47:27,662 [WARNING] {12636} koji.TaskManager:1181 TRACEBACK:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in
runTask
response = (handler.run(),)
File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run
return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in
call_with_argcheck
return func(*args, **kwargs)
File "/usr/sbin/kojid", line 2845, in handler
self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
File "/usr/sbin/kojid", line 2895, in create_local_repo
status = log_output(self.session, cmd[0], cmd, logfile,
self.getUploadDir(), logerror=True)
File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in
log_output
incremental_upload(session, remotename, outfd, uploadpath)
File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in
incremental_upload
fast_incremental_upload(session, fname, fd, path, retries, logger)
File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in
fast_incremental_upload
result = session.rawUpload(contents, offset, path, fname, overwrite=True)
File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in
__call__
return self.__func(self.__name,args,opts)
File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in
_callMethod
raise err
LockError: [Errno 37] No locks available
2013-11-15 14:47:33,658 [WARNING] {12638} koji.TaskManager:1181 TRACEBACK:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in
runTask
response = (handler.run(),)
File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run
return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in
call_with_argcheck
return func(*args, **kwargs)
File "/usr/sbin/kojid", line 2845, in handler
self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
File "/usr/sbin/kojid", line 2895, in create_local_repo
status = log_output(self.session, cmd[0], cmd, logfile,
self.getUploadDir(), logerror=True)
File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in
log_output
incremental_upload(session, remotename, outfd, uploadpath)
File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in
incremental_upload
fast_incremental_upload(session, fname, fd, path, retries, logger)
File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in
fast_incremental_upload
result = session.rawUpload(contents, offset, path, fname, overwrite=True)
File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in
__call__
return self.__func(self.__name,args,opts)
File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in
_callMethod
raise err
LockError: [Errno 37] No locks available
2013-11-15 14:47:35,625 [INFO] {12485} koji.TaskManager:679 pids: {4891:
12568, 4892: 12636, 4893: 12638}
2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:681 open task:
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True}
2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:693 Waking up task:
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True}
2013-11-15 14:47:35,691 [INFO] {12485} koji.TaskManager:846 Task 4892 (pid
12636) exited with status 0
2013-11-15 14:47:35,715 [INFO] {12485} koji.TaskManager:991 Expiring
subsession 7036 (task 4892)
2013-11-15 14:47:35,780 [INFO] {12485} koji.TaskManager:846 Task 4893 (pid
12638) exited with status 0
2013-11-15 14:47:35,803 [INFO] {12485} koji.TaskManager:991 Expiring
subsession 7037 (task 4893)
2013-11-15 14:47:36,640 [WARNING] {12568} koji.TaskManager:1181 TRACEBACK:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in
runTask
response = (handler.run(),)
File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run
return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in
call_with_argcheck
return func(*args, **kwargs)
File "/usr/sbin/kojid", line 2811, in handler
results = self.wait(subtasks.values(), all=True, failany=True)
File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 234, in wait
return dict(self.session.host.taskWaitResults(self.id,subtasks))
File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in
__call__
return self.__func(self.__name,args,opts)
File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in
_callMethod
raise err
LockError: [Errno 37] No locks available
2013-11-15 14:47:51,152 [INFO] {12485} koji.TaskManager:679 pids: {4891:
12568}
2013-11-15 14:47:51,215 [INFO] {12485} koji.TaskManager:846 Task 4891 (pid
12568) exited with status 0
2013-11-15 14:47:51,239 [INFO] {12485} koji.TaskManager:991 Expiring
subsession 7035 (task 4891)
