Re: [galaxy-dev] Bug in handling user job limits (due to dynamic job runner code?)
Hi Lance I was just running into the same issue as part of upgrading our production server from release 2012_10_05 to 2013_02_08 (unfortunately, I didn't catch it last week, when I did our test server, as the job limit was not set there). I added a little comment to your trello ticket, but I thought it might be useful to know for a the wider community. The user job limit has been very handy for us. And I am sure other Galaxy installations rely on this as well. Regards, Hans-Rudolf On 01/18/2013 11:28 PM, Lance Parsons wrote: Just an update on this issue. Upon further investigation, it looks like the dynamic job runner code in commit 6f3b4e8 broke this. I haven't been able to parse through everything going on yet to propose a fix, but I've discovered two things: 1) Moving the self.__clear_user_job_count() call outside of the if block, does indeed fix that issue 2) Fixing that leads the __check_user_jobs method to fail since it does not check `self.track_jobs_in_database`, but instead assumes that jobs are in the database, resulting in a different exception regarding invalid columns. Lance Parsons wrote: I submitted a trello ticket [(https://trello.com/c/6vxkqdjT) regarding this, but wanted to make sure I brought it to someone's attention (it's causing me some queue issues with my instance of galaxy). When registered_user_job_limit and anonymous_user_job_limit are set in universe.wsgi jobs cannot be run, instead the following error occurs: galaxy.jobs.handler ERROR 2012-12-04 12:44:51,869 failure running job 22396 Traceback (most recent call last): File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 183, in __monitor_step job_state = self.__check_if_ready_to_run( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 253, in __check_if_ready_to_run state = self.__check_user_jobs( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 274, in __check_user_jobs if not self.user_job_count: AttributeError: 'JobHandlerQueue' object has no attribute 'user_job_count' Commenting out the lines for registered_user_job_limit and anonymous_user_job_limit in universe.wsgi allows job to be queue one again. It looks like this is due to the fact that `self.__clear_user_job_count()` on line 159 of `handler.py` is only called when jobs are tracked in the database. If jobs are not tracked in the database (as in my case), the error occurs. Perhaps the fix would be to simply move the call outside the `if` block. It appears this was broken in the 2012-11-13 revision (73e05bc). ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug in handling user job limits (due to dynamic job runner code?)
Hi Hans, Thanks for bumping this thread. The issue has been fixed in development and will be included in the next release. --nate On Mar 4, 2013, at 5:48 AM, Hans-Rudolf Hotz wrote: Hi Lance I was just running into the same issue as part of upgrading our production server from release 2012_10_05 to 2013_02_08 (unfortunately, I didn't catch it last week, when I did our test server, as the job limit was not set there). I added a little comment to your trello ticket, but I thought it might be useful to know for a the wider community. The user job limit has been very handy for us. And I am sure other Galaxy installations rely on this as well. Regards, Hans-Rudolf On 01/18/2013 11:28 PM, Lance Parsons wrote: Just an update on this issue. Upon further investigation, it looks like the dynamic job runner code in commit 6f3b4e8 broke this. I haven't been able to parse through everything going on yet to propose a fix, but I've discovered two things: 1) Moving the self.__clear_user_job_count() call outside of the if block, does indeed fix that issue 2) Fixing that leads the __check_user_jobs method to fail since it does not check `self.track_jobs_in_database`, but instead assumes that jobs are in the database, resulting in a different exception regarding invalid columns. Lance Parsons wrote: I submitted a trello ticket [(https://trello.com/c/6vxkqdjT) regarding this, but wanted to make sure I brought it to someone's attention (it's causing me some queue issues with my instance of galaxy). When registered_user_job_limit and anonymous_user_job_limit are set in universe.wsgi jobs cannot be run, instead the following error occurs: galaxy.jobs.handler ERROR 2012-12-04 12:44:51,869 failure running job 22396 Traceback (most recent call last): File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 183, in __monitor_step job_state = self.__check_if_ready_to_run( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 253, in __check_if_ready_to_run state = self.__check_user_jobs( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 274, in __check_user_jobs if not self.user_job_count: AttributeError: 'JobHandlerQueue' object has no attribute 'user_job_count' Commenting out the lines for registered_user_job_limit and anonymous_user_job_limit in universe.wsgi allows job to be queue one again. It looks like this is due to the fact that `self.__clear_user_job_count()` on line 159 of `handler.py` is only called when jobs are tracked in the database. If jobs are not tracked in the database (as in my case), the error occurs. Perhaps the fix would be to simply move the call outside the `if` block. It appears this was broken in the 2012-11-13 revision (73e05bc). ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug in handling user job limits (due to dynamic job runner code?)
Thanks Nate, it's much appreciated. I'm glad I was able to help out a bit, just sorry I wasn't able to provide a complete fix (though maybe that's for the best if there is a rewrite underway). Lance Nate Coraor wrote: Hi Lance, I'm rewriting much of this code, I should have all of it committed (including this fix) some time this week. Thanks for figuring out exactly what's going on here. --nate On Jan 18, 2013, at 5:28 PM, Lance Parsons wrote: Just an update on this issue. Upon further investigation, it looks like the dynamic job runner code in commit 6f3b4e8 broke this. I haven't been able to parse through everything going on yet to propose a fix, but I've discovered two things: 1) Moving the self.__clear_user_job_count() call outside of the if block, does indeed fix that issue 2) Fixing that leads the __check_user_jobs method to fail since it does not check `self.track_jobs_in_database`, but instead assumes that jobs are in the database, resulting in a different exception regarding invalid columns. Lance Parsons wrote: I submitted a trello ticket [(https://trello.com/c/6vxkqdjT) regarding this, but wanted to make sure I brought it to someone's attention (it's causing me some queue issues with my instance of galaxy). When registered_user_job_limit and anonymous_user_job_limit are set in universe.wsgi jobs cannot be run, instead the following error occurs: galaxy.jobs.handler ERROR 2012-12-04 12:44:51,869 failure running job 22396 Traceback (most recent call last): File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 183, in __monitor_step job_state = self.__check_if_ready_to_run( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 253, in __check_if_ready_to_run state = self.__check_user_jobs( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 274, in __check_user_jobs if not self.user_job_count: AttributeError: 'JobHandlerQueue' object has no attribute 'user_job_count' Commenting out the lines for registered_user_job_limit and anonymous_user_job_limit in universe.wsgi allows job to be queue one again. It looks like this is due to the fact that `self.__clear_user_job_count()` on line 159 of `handler.py` is only called when jobs are tracked in the database. If jobs are not tracked in the database (as in my case), the error occurs. Perhaps the fix would be to simply move the call outside the `if` block. It appears this was broken in the 2012-11-13 revision (73e05bc). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug in handling user job limits (due to dynamic job runner code?)
Hi Lance, I'm rewriting much of this code, I should have all of it committed (including this fix) some time this week. Thanks for figuring out exactly what's going on here. --nate On Jan 18, 2013, at 5:28 PM, Lance Parsons wrote: Just an update on this issue. Upon further investigation, it looks like the dynamic job runner code in commit 6f3b4e8 broke this. I haven't been able to parse through everything going on yet to propose a fix, but I've discovered two things: 1) Moving the self.__clear_user_job_count() call outside of the if block, does indeed fix that issue 2) Fixing that leads the __check_user_jobs method to fail since it does not check `self.track_jobs_in_database`, but instead assumes that jobs are in the database, resulting in a different exception regarding invalid columns. Lance Parsons wrote: I submitted a trello ticket [(https://trello.com/c/6vxkqdjT) regarding this, but wanted to make sure I brought it to someone's attention (it's causing me some queue issues with my instance of galaxy). When registered_user_job_limit and anonymous_user_job_limit are set in universe.wsgi jobs cannot be run, instead the following error occurs: galaxy.jobs.handler ERROR 2012-12-04 12:44:51,869 failure running job 22396 Traceback (most recent call last): File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 183, in __monitor_step job_state = self.__check_if_ready_to_run( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 253, in __check_if_ready_to_run state = self.__check_user_jobs( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 274, in __check_user_jobs if not self.user_job_count: AttributeError: 'JobHandlerQueue' object has no attribute 'user_job_count' Commenting out the lines for registered_user_job_limit and anonymous_user_job_limit in universe.wsgi allows job to be queue one again. It looks like this is due to the fact that `self.__clear_user_job_count()` on line 159 of `handler.py` is only called when jobs are tracked in the database. If jobs are not tracked in the database (as in my case), the error occurs. Perhaps the fix would be to simply move the call outside the `if` block. It appears this was broken in the 2012-11-13 revision (73e05bc). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug in handling user job limits (due to dynamic job runner code?)
Just an update on this issue. Upon further investigation, it looks like the dynamic job runner code in commit 6f3b4e8 broke this. I haven't been able to parse through everything going on yet to propose a fix, but I've discovered two things: 1) Moving the self.__clear_user_job_count() call outside of the if block, does indeed fix that issue 2) Fixing that leads the __check_user_jobs method to fail since it does not check `self.track_jobs_in_database`, but instead assumes that jobs are in the database, resulting in a different exception regarding invalid columns. Lance Parsons wrote: I submitted a trello ticket [(https://trello.com/c/6vxkqdjT) regarding this, but wanted to make sure I brought it to someone's attention (it's causing me some queue issues with my instance of galaxy). When registered_user_job_limit and anonymous_user_job_limit are set in universe.wsgi jobs cannot be run, instead the following error occurs: galaxy.jobs.handler ERROR 2012-12-04 12:44:51,869 failure running job 22396 Traceback (most recent call last): File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 183, in __monitor_step job_state = self.__check_if_ready_to_run( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 253, in __check_if_ready_to_run state = self.__check_user_jobs( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 274, in __check_user_jobs if not self.user_job_count: AttributeError: 'JobHandlerQueue' object has no attribute 'user_job_count' Commenting out the lines for registered_user_job_limit and anonymous_user_job_limit in universe.wsgi allows job to be queue one again. It looks like this is due to the fact that `self.__clear_user_job_count()` on line 159 of `handler.py` is only called when jobs are tracked in the database. If jobs are not tracked in the database (as in my case), the error occurs. Perhaps the fix would be to simply move the call outside the `if` block. It appears this was broken in the 2012-11-13 revision (73e05bc). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
