Hey Devs, Just wanted to get some input on our to plan to implement the queue throttling feature.
Batch Queue Throttling: - in Orchestrator, the current submit() function in GFACPassiveJobSubmitter publishes jobs to rabbitmq immediately - instead of publishing immediately we should pass the messages to a new component, call it BatchQueueClass. - we need a new component BatchQueueClass to periodically check to see when we can unload jobs to submit Adding BatchQueueClass - setup a new table(s) to contain compute resource names and their corresponding queues' current job numbers and maximum job limits - data models in airavata have information on maximum job submission limits for a queue but no data on how many jobs are currently running - the current job number will effectively act as a counter, which will be incremented when a job is submitted, and when a job is completed - once that is done, BatchQueueClass needs to periodically check new table to see if the user's requested queue's current job number < queue job limit. If it is then we can pop jobs off and submit them until we hit the job limit; if not, then we wait until the we're under the job limit. How does this sound? Doug
