Thread configuration
All, We've just run through an upgrade on a server group from 7.0.1 to 7.1.0 (latest patch of each). The primary upgrade ran fine. Secondary seemed to run through fine as well, until users started logging on this morning, when it went slow. Turns out that the 7.1.0 installer amended the thread configuration on the secondary. Previous : Private-RPC-Socket: 390601 1 15 Private-RPC-Socket: 390620 6 35 Private-RPC-Socket: 390625 1 40 Private-RPC-Socket: 390635 6 45 New : Private-RPC-Socket: 390601 1 15 Private-RPC-Socket: 390620 6 35 Private-RPC-Socket: 390625 1 40 Private-RPC-Socket: 390635 6 45 Private-RPC-Socket: 390620 2 2 Private-RPC-Socket: 390635 2 2 390620 and 390635 were effectively replaced. Turns out the same thing happened on our test server group environment, but we never had more than a couple of users to check on performance. Commented out the two new lines and performance returned to normal. Our production servers are running solaris, and both are running 32 cores. Which - according to the old theory - means that our thread config is still incorrect. Does the min=3xCPUs, max=5xCPUs still count for much? Is the calculation based on nn x Number of cores or nn x number of CPUs? Regards Dave Barber ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Thread configuration
Dave, My understanding was 3X for Fast, 5X for List...BOTH are MAX values...neither are Min values. The idea is that it's a 'calculated' maximum for your hardware for optimum performance. What the minimum should be is entirely up to your system. If your system is working will with 6 min and 35/45 max respective...that's good. I personally like to set the min to somewhere around 2, then 'let the system run' and see where the thread counts are at in general, and adjust my min's appropriately. The idea behind having a min/max is that starting a thread takes time, so you don't necessarily want to be waiting for that thread to start during load...so you want your min to be high enough to handle your 'standard' configuration, but not so high as to have wasted resources. I have always considered the max to be a 'to handle overflow' type of setting...you want to allow your system to grow within certain parameters, but not unlimited. On Mon, Nov 4, 2013 at 7:55 AM, Dave Barber daddy.bar...@gmail.com wrote: ** All, We've just run through an upgrade on a server group from 7.0.1 to 7.1.0 (latest patch of each). The primary upgrade ran fine. Secondary seemed to run through fine as well, until users started logging on this morning, when it went slow. Turns out that the 7.1.0 installer amended the thread configuration on the secondary. Previous : Private-RPC-Socket: 390601 1 15 Private-RPC-Socket: 390620 6 35 Private-RPC-Socket: 390625 1 40 Private-RPC-Socket: 390635 6 45 New : Private-RPC-Socket: 390601 1 15 Private-RPC-Socket: 390620 6 35 Private-RPC-Socket: 390625 1 40 Private-RPC-Socket: 390635 6 45 Private-RPC-Socket: 390620 2 2 Private-RPC-Socket: 390635 2 2 390620 and 390635 were effectively replaced. Turns out the same thing happened on our test server group environment, but we never had more than a couple of users to check on performance. Commented out the two new lines and performance returned to normal. Our production servers are running solaris, and both are running 32 cores. Which - according to the old theory - means that our thread config is still incorrect. Does the min=3xCPUs, max=5xCPUs still count for much? Is the calculation based on nn x Number of cores or nn x number of CPUs? Regards Dave Barber _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Thread configuration
Many thanks for that. It was interesting to see how the one working server coped with the load this morning. Typically around 600 users per server, the load was all directed to one and while it slowed down, it didn't slow down by that much (opening an incident went from 7 secs to about 12). I think that our current configuration (aside from the 7.1.0 thread changes) is probably sufficient. Some of our users would question current application performance, but complaining about systems is what keeps them happy. Regards Dave On 4 November 2013 15:11, LJ LongWing lj.longw...@gmail.com wrote: ** Dave, My understanding was 3X for Fast, 5X for List...BOTH are MAX values...neither are Min values. The idea is that it's a 'calculated' maximum for your hardware for optimum performance. What the minimum should be is entirely up to your system. If your system is working will with 6 min and 35/45 max respective...that's good. I personally like to set the min to somewhere around 2, then 'let the system run' and see where the thread counts are at in general, and adjust my min's appropriately. The idea behind having a min/max is that starting a thread takes time, so you don't necessarily want to be waiting for that thread to start during load...so you want your min to be high enough to handle your 'standard' configuration, but not so high as to have wasted resources. I have always considered the max to be a 'to handle overflow' type of setting...you want to allow your system to grow within certain parameters, but not unlimited. On Mon, Nov 4, 2013 at 7:55 AM, Dave Barber daddy.bar...@gmail.comwrote: ** All, We've just run through an upgrade on a server group from 7.0.1 to 7.1.0 (latest patch of each). The primary upgrade ran fine. Secondary seemed to run through fine as well, until users started logging on this morning, when it went slow. Turns out that the 7.1.0 installer amended the thread configuration on the secondary. Previous : Private-RPC-Socket: 390601 1 15 Private-RPC-Socket: 390620 6 35 Private-RPC-Socket: 390625 1 40 Private-RPC-Socket: 390635 6 45 New : Private-RPC-Socket: 390601 1 15 Private-RPC-Socket: 390620 6 35 Private-RPC-Socket: 390625 1 40 Private-RPC-Socket: 390635 6 45 Private-RPC-Socket: 390620 2 2 Private-RPC-Socket: 390635 2 2 390620 and 390635 were effectively replaced. Turns out the same thing happened on our test server group environment, but we never had more than a couple of users to check on performance. Commented out the two new lines and performance returned to normal. Our production servers are running solaris, and both are running 32 cores. Which - according to the old theory - means that our thread config is still incorrect. Does the min=3xCPUs, max=5xCPUs still count for much? Is the calculation based on nn x Number of cores or nn x number of CPUs? Regards Dave Barber _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Java plugin server thread configuration question
Many thanks to John Baker from Java Systems Solutions for a work-around to this via modification to the plugin code (see below and I hope I incorporated it correctly :-). I would still like to know if it is possible to configure per plugin threads via the pluginsvr.conf configuration or other mechanism. This is for an ARS 7.5 / 7.6 environment. Thanks, Eric Example singleton plugin: -- package com.company.samples; import java.util.List; import com.bmc.arsys.api.ARException; import com.bmc.arsys.api.Value; import com.bmc.arsys.pluginsvr.*; import com.bmc.arsys.pluginsvr.plugins.ARFilterAPIPluggable; import com.bmc.arsys.pluginsvr.plugins.ARPluggable; import com.bmc.arsys.pluginsvr.plugins.ARFilterAPIPlugin; import com.bmc.arsys.pluginsvr.plugins.ARPluginContext; import com.bmc.arsys.*; import org.apache.log4j.Logger; import com.company.someotherstuff*; /** * A class representing a singletone Plugin service. * This will run as a java plugin under the Remedy Java plugin framework * as a single instance instead of x instances as defined by arpluginsvr coreThreads * * Useful when the plugin calls it's own management mechanism for work distribution * or a poller mechanism that should only be initialized once. * */ public class MyPlugin extends ARFilterAPIPlugin { // initiate logger private static final Logger logger = Logger.getLogger(MyPlugin.class); // for our singleton private static MyPlugin myplug; public MyPlugin() {}; /* Initialize the Remedy plugin */ public void initialize(ARPluginContext context) throws ARException { logger.info(Started plugin init); super.initialize(context); /* check if it's already available, if not it's safe to init the code for other stuff to do */ if(myplug == null){ myplug = getInstance(); //initialize the stuff we want to run outside of current thread initializeApp(context); } } public void terminate(ARPluginContext context) throws ARException { logger.info(Terminating context); //cleanup routine here for graceful shutdown super.terminate(context); } /* start it (the plugin) */ public static void main(String[] args) { //'cause it's not ultimately useful at this juncture to accept params at startup... if(args!= null){ logger.warn(This plugin does not accept command line messaging...); } } /* initialize the meat of the plugin outside of the plugin initialization thread so the init process can complete and we aren't mucking things up (according to the docs) */ private void initializeApp(ARPluginContext context) throws ARException{ //class for storing config variables in memory instead of dealing with an overload of i/o ConfigParams cp = new ConfigParams(); try{ logger.info(calling init...); cp.initialize(); // load config file parameters into mem Thread t = new Thread(new Poller()); //thread our poller mechanism t.start(); //start the poller } catch (RuntimeException rte){ terminate(context); } } /* we aren't allowing filter api calls to this plugin because the plugin polls remedy asynchronously for all work to process */ public ListValue filterAPICall(ARPluginContext context, ListValue in) throws ARException { return null; } /* we aren't allowing event calls to this plugin */ public void onEvent(ARPluginContext context, int arg1) throws ARException { //do nothing } /* get an instance of this plugin */ private synchronized static MyPlugin getInstance() { if (myplug==null) myplug = new MyPlugin(); return plug; } } Previous message: Is it possible to configure individual plugins within a plugin server to use their own configuration for threads? I.E. if there are multiple plugins within pluginsvr_config.xml, can each have their own designated number of threads or is it only possible that each uses the numCoreThreads setting for the overall plugin server? If they need to be different from the numCoreThreads designated by the pluginserver, is there any way to handle outside of running under a different plugin server instance, and if not, can multiple plugin servers be run on the same server with a single instance of Remedy? (see below for example plugin server config) Thanks in advance for any thoughts on this... Kind Regards, Eric Roys GSSI Verizon Business Example plugin server config file... pluginsvr_config portmyPort/port regPortMapperfalse/regPortMapper encryptionPolicy2/encryptionPolicy publicKeyAlg4/publicKeyAlg publicKeyExpiry86400/publicKeyExpiry dataEncryptionAlg1/dataEncryptionAlg dataKeyExpiry2700/dataKeyExpiry numCoreThreads5/numCoreThreads
Java plugin server thread configuration question
Eric, Well done, although there is one minor change that's required: private static final byte[] LOCK= new byte[0]; public void initialize(ARPluginContext context) throws ARException { ... synchronized(LOCK) { if(myplug == null){ myplug = getInstance(); ... } } } The lock is required because multiple threads can access that method, and you only want one to create the MyPlugin object. As a further improvement, I'd use a different object for your singleton, i.e. separate it from the plugin. Finally, do remember that in a multi-threaded model, using a singleton is a bottleneck. John -- Single Sign On for the AR System http://www.javasystemsolutions.com/jss/ssoplugin ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: Where the Answers Are
Re: Java plugin server thread configuration question
Thanks, John. Duly noted. Again, appreciate the wisdom! Without the gory details, I believe any bottle neck here, will have little if any affect on the requirements and design for my particular need, but I'll keep searching for a better alternative. -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of John Baker Sent: Friday, October 29, 2010 4:08 PM To: arslist@ARSLIST.ORG Subject: Java plugin server thread configuration question Eric, Well done, although there is one minor change that's required: private static final byte[] LOCK= new byte[0]; public void initialize(ARPluginContext context) throws ARException { ... synchronized(LOCK) { if(myplug == null){ myplug = getInstance(); ... } } } The lock is required because multiple threads can access that method, and you only want one to create the MyPlugin object. As a further improvement, I'd use a different object for your singleton, i.e. separate it from the plugin. Finally, do remember that in a multi-threaded model, using a singleton is a bottleneck. John -- Single Sign On for the AR System http://www.javasystemsolutions.com/jss/ssoplugin ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: Where the Answers Are
Java plugin server thread configuration question
Is it possible to configure individual plugins within a plugin server to use their own configuration for threads? I.E. if there are multiple plugins within pluginsvr_config.xml, can each have their own designated number of threads or is it only possible that each uses the numCoreThreads setting for the overall plugin server? If they need to be different from the numCoreThreads designated by the pluginserver, is there any way to handle outside of running under a different plugin server instance, and if not, can multiple plugin servers be run on the same server with a single instance of Remedy? (see below for example plugin server config) Thanks in advance for any thoughts on this... Kind Regards, Eric Roys GSSI Verizon Business Example plugin server config file... pluginsvr_config portmyPort/port regPortMapperfalse/regPortMapper encryptionPolicy2/encryptionPolicy publicKeyAlg4/publicKeyAlg publicKeyExpiry86400/publicKeyExpiry dataEncryptionAlg1/dataEncryptionAlg dataKeyExpiry2700/dataKeyExpiry numCoreThreads5/numCoreThreads numSelectorThreads2/numSelectorThreads workQueueMonitorLogInterval0/workQueueMonitorLogInterval workQueueTaskThreshold5/workQueueTaskThreshold plugins plugin namePLUG00/name !-- start 10 threads ? -- pathelement type=locationsome jar location/pathelement classnamesome class name/classname /plugin plugin namePLUG01/name !-- start 1 threads ? -- pathelement type=locationsome jar location/pathelement classnamesome class name/classname /plugin plugin namePLUG02/name !-- start 5 threads ? -- pathelement type=locationsome jar location/pathelement classnamesome class name/classname /plugin plugin namePLUG03/name !-- start 7 threads ?-- pathelement type=locationsome jar location/pathelement classnamesome class name/classname /plugin /plugins /pluginsvr_config ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: Where the Answers Are
Thread Configuration Questions
All: My client has questions regarding where to configure thread usage. Anyone with this expertise? 1. How can we tell how many threads the servers are currently configured to use? (NOTE: Not AREA threads - we have already made updates to those to reduce the TCP connections) 2. Where is thread configuration information stored? 3. Are there a minimum/maximum number of threads that the servers are configured to use? 4. Are there an initial one-to-one relationship between connections and threads? In other words, does each new connection (from a unique IP) start by using only one new thread or does each connection start with multiple threads? 5. Do the number of threads stay the same after established for that connection, or do they increase depending on the requests being made to the server? Thanks in advance for any help you can give! -LisaD - Lisa W [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Thread-Configuration-Questions-tp18945859p18945859.html Sent from the ARS (Action Request System) mailing list archive at Nabble.com. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: Where the Answers Are
Re: Thread Configuration Questions
Lisa, I have to assume that you know how and where to define Fast/List threads (question #1), and that I perhaps misunderstood your real question. Let me share some deeper information on what I believe are your other questions. 1. The minimum number of defined threads will always start and be available, whether in use or not. 2. The threads are stored in the ar.conf/ar.cfg file, which is read and cached at AR System startup. You can turn on thread-level logging to see how many are actually in use at any time. 3. The Min/Max numbers are in the docs - Configuration, I think. 4/5. I'll answer this one based on how a thread processes transactions. My knowledge here might be old, but it's the last thing I heard. Thread #1 will queue up to 5 transaction connections (appropriate to tx type). Once tx #6 comes along, it is bumped to the next thread in line. If there are no other active threads available, AR System will start one unless it is already at its defined maximum number. This continues until the max # of defined threads for that tx type (List/Fast) has been reached. At that point, timeouts will likely occur. I don't think the threads are IP/client specific (i.e. sticky), but I could be wrong about that. I think it's just dealing with the raw transactions. Hope this is what you're looking for! Rick On Tue, Aug 12, 2008 at 7:50 AM, LisaD [EMAIL PROTECTED] wrote: All: My client has questions regarding where to configure thread usage. Anyone with this expertise? 1. How can we tell how many threads the servers are currently configured to use? (NOTE: Not AREA threads - we have already made updates to those to reduce the TCP connections) 2. Where is thread configuration information stored? 3. Are there a minimum/maximum number of threads that the servers are configured to use? 4. Are there an initial one-to-one relationship between connections and threads? In other words, does each new connection (from a unique IP) start by using only one new thread or does each connection start with multiple threads? 5. Do the number of threads stay the same after established for that connection, or do they increase depending on the requests being made to the server? Thanks in advance for any help you can give! -LisaD - Lisa W [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Thread-Configuration-Questions-tp18945859p18945859.html Sent from the ARS (Action Request System) mailing list archive at Nabble.com. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: Where the Answers Are
Re: Thread Configuration Questions
Rick - thank you! That is exactly what I was looking for. From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Rick Cook Sent: Tuesday, August 12, 2008 10:16 AM To: arslist@ARSLIST.ORG Subject: Re: Thread Configuration Questions ** Lisa, I have to assume that you know how and where to define Fast/List threads (question #1), and that I perhaps misunderstood your real question. Let me share some deeper information on what I believe are your other questions. 1. The minimum number of defined threads will always start and be available, whether in use or not. 2. The threads are stored in the ar.conf/ar.cfg file, which is read and cached at AR System startup. You can turn on thread-level logging to see how many are actually in use at any time. 3. The Min/Max numbers are in the docs - Configuration, I think. 4/5. I'll answer this one based on how a thread processes transactions. My knowledge here might be old, but it's the last thing I heard. Thread #1 will queue up to 5 transaction connections (appropriate to tx type). Once tx #6 comes along, it is bumped to the next thread in line. If there are no other active threads available, AR System will start one unless it is already at its defined maximum number. This continues until the max # of defined threads for that tx type (List/Fast) has been reached. At that point, timeouts will likely occur. I don't think the threads are IP/client specific (i.e. sticky), but I could be wrong about that. I think it's just dealing with the raw transactions. Hope this is what you're looking for! Rick On Tue, Aug 12, 2008 at 7:50 AM, LisaD [EMAIL PROTECTED] wrote: All: My client has questions regarding where to configure thread usage. Anyone with this expertise? 1. How can we tell how many threads the servers are currently configured to use? (NOTE: Not AREA threads - we have already made updates to those to reduce the TCP connections) 2. Where is thread configuration information stored? 3. Are there a minimum/maximum number of threads that the servers are configured to use? 4. Are there an initial one-to-one relationship between connections and threads? In other words, does each new connection (from a unique IP) start by using only one new thread or does each connection start with multiple threads? 5. Do the number of threads stay the same after established for that connection, or do they increase depending on the requests being made to the server? Thanks in advance for any help you can give! -LisaD - Lisa W [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Thread-Configuration-Questions-tp18945859p18945859 .html Sent from the ARS (Action Request System) mailing list archive at Nabble.com. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: Where the Answers Are __Platinum Sponsor: www.rmsportal.com ARSlist: Where the Answers Are html___ This message is subject to and does not create or vary any contractual relationship between TuringSMI, SMI Technologies, SMI Telco, its subsidiaries or affiliates and you. Internet communications are not secure and therefore the TuringSMI Group does not accept any legal responsibility for the contents of this message. Any views or opinions expressed are those of the author. This message is intended for the addressee(s) only and its contents and any attached files are strictly confidential. If you have received it in error, please contact the sender on the number above. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: Where the Answers Are
Re: Thread Configuration Questions
One important consideration the Max nunmber of threads depends of the Hardware (CPU RAM) if the server ask to your OS Personnel to ask how many threads the server can accept for connections; also consider that the fast and list threads are for operations of search and write for all transactions due the database, exists other threads for alerts, and admin queue, if it's needed you can define a private queue with their own max min threads for making a balancing in the ARS Server, but always considering the hardware capacities. Best Regards Hugo Ruesga perotsystems® US 972.577.7000MX +52 (33) 3332.3868 P Please consider the environment before printing this email The information contained in and transferred with this electronic message is intended only for the recipient(s) designated above, it is protected by law and it may contain information which is privileged and confidential. If you are not the intended recipient, please do not read, copy, or use it, and do not disclose it to others. Please notify the sender of the delivery error by replying to this message, and then delete it from your system. Thank you. Date: Tue, 12 Aug 2008 16:20:57 +0100From: [EMAIL PROTECTED]: Re: Thread Configuration QuestionsTo: [EMAIL PROTECTED] Rick – thank you! That is exactly what I was looking for. From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Rick CookSent: Tuesday, August 12, 2008 10:16 AMTo: [EMAIL PROTECTED]: Re: Thread Configuration Questions ** Lisa, I have to assume that you know how and where to define Fast/List threads (question #1), and that I perhaps misunderstood your real question. Let me share some deeper information on what I believe are your other questions.1. The minimum number of defined threads will always start and be available, whether in use or not.2. The threads are stored in the ar.conf/ar.cfg file, which is read and cached at AR System startup. You can turn on thread-level logging to see how many are actually in use at any time.3. The Min/Max numbers are in the docs - Configuration, I think.4/5. I'll answer this one based on how a thread processes transactions. My knowledge here might be old, but it's the last thing I heard. Thread #1 will queue up to 5 transaction connections (appropriate to tx type). Once tx #6 comes along, it is bumped to the next thread in line. If there are no other active threads available, AR System will start one unless it is already at its defined maximum number. This continues until the max # of defined threads for that tx type (List/Fast) has been reached. At that point, timeouts will likely occur.I don't think the threads are IP/client specific (i.e. sticky), but I could be wrong about that. I think it's just dealing with the raw transactions.Hope this is what you're looking for!Rick On Tue, Aug 12, 2008 at 7:50 AM, LisaD [EMAIL PROTECTED] wrote: All: My client has questions regarding where to configure thread usage.Anyone with this expertise?1. How can we tell how many threads the servers are currently configured touse?(NOTE: Not AREA threads - we have already made updates to those to reducethe TCP connections)2. Where is thread configuration information stored?3. Are there a minimum/maximum number of threads that the servers areconfigured to use?4. Are there an initial one-to-one relationship between connections andthreads? In other words, does each new connection (from a unique IP) startby using only one new thread or does each connection start with multiplethreads?5. Do the number of threads stay the same after established for thatconnection, or do they increase depending on the requests being made to theserver?Thanks in advance for any help you can give!-LisaD-Lisa [EMAIL PROTECTED] this message in context: http://www.nabble.com/Thread-Configuration-Questions-tp18945859p18945859.htmlSent from the ARS (Action Request System) mailing list archive at Nabble.com.___UNSUBSCRIBE or access ARSlist Archives at www.arslist.orgPlatinum Sponsor: www.rmsportal.com ARSlist: Where the Answers Are __Platinum Sponsor: www.rmsportal.com ARSlist: Where the Answers Are html___ TuringSMI is a Platinum Sponsor of both BMC UserWorld Events Email Disclaimer This email has been sent from the TuringSMI Group This message is subject to and does not create or vary any contractual relationship between TuringSMI, SMI Technologies, SMI Telco, its subsidiaries or affiliates and you. Internet communications are not secure and therefore the TuringSMI Group does not accept any legal responsibility for the contents of this message. Any views or opinions expressed are those of the author. This message is intended for the addressee(s) only and its contents and any attached files are strictly confidential. If you have received it in error, please contact the sender on the number