Workflow Manager Client - Add Connection Limit Option
-----------------------------------------------------
Key: OODT-383
URL: https://issues.apache.org/jira/browse/OODT-383
Project: OODT
Issue Type: New Feature
Components: workflow manager
Environment: centOS 5/6
Reporter: Cameron Goodale
Assignee: Cameron Goodale
Priority: Minor
When using the wmgr-client to run thousands of jobs it is pretty easy to
overwhelm the xml-rpc connection pool to the workflow manager. I was using a
simple python script to submit 10K jobs and the workflow manager couldn't
handle the jobs quickly enough and many jobs were dropped as a result.
One fix I implemented in my Python code was to use lsof to check the number of
ESTABLISHED connections to the workflow manager. If the workflow manager had
more than say 30 connections, my program would go to sleep and try submitting
jobs later.
I would like to enhance the wmgr-client shell script with an option to limit
the number of connections to the wmgr, by default this limit would not be set.
If the connection limit is reached the wmgr-client would sleep for 10 seconds,
and re-check the number of connections. This loop would continue until the
number of connections dropped below the specified limit. Once the connection
count drops below the target number, the wmgr-client would resume submitting
jobs to the wmgr.
On my production server I was using lsof to gather the number of connections to
the wmgr. I am not sure if we can always rely on lsof being installed on all
machines, so we might need to use a more universal method (maybe in Java).
here is the lsof command I used with some grep and wc sprinkled in:
{{/usr/sbin/lsof -i :9001 | grep ESTABLISHED | wc}}
This assumes you are running wmgr on localhost:9001 and lsof is installed at
/usr/sbin/lsof
Any other thoughts or ideas to work this out would be appreciated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
