Alex Rousskov wrote:
Hello,I would like to implement the following changes to add initial SMP support to Squid. The changes are relatively simple but should allow us to reap a lot of SMP benefits in many real deployments. I hope to submit the results for review and trunk inclusion in about 45 days. More sophisticated changes will follow, guided by real and lab performance data. The plan is based on the SMP-related discussions we had in the past few months. I factored in the apparent lack of developers available to tackle more ambitious designs, the current state of code, and our historical inability to handle huge projects without slipping off schedule and creating various disasters along the way. 1. Processes versus threads: The initial design is process- and not thread-based. Processes may be converted to threads and/or threads may be added to certain processes later. If we start with threads now, we may drown in problems caused by poor encapsulation of some of the major code pieces and thread-unreadiness of basic Squid libraries.
You are speaking of the implementing layer 1 of the 3-layer architecture from the wiki?
2. Building blocks: A Squid process dedicated to a subset of squid.conf options is a building block. The user can configure Squid to launch multiple such processes. Option subsets are likely to overlap a lot because many options would be the same for each process. In the initial implementation, _all_ subsets will be identical. In other words, all Squid processes will be identically configured and, hence, "do the same thing". The main Squid process will need to open http_port(s) and other listening sockets. It will either do it before forking child Squids or will pass open socket descriptors to child Squids via sendmsg(2). In the initial implementation, the admin will be able to specify which CPU core(s) should be used for Squid. Eventually, it would be possible to map individual building blocks to individual CPU cores. 3. What is expected to work: SMP-scalable performance on general workloads. For example, if you have 8 CPU cores, you can utilize all of them. Squid will behave as a single instance with respect to misses, reconfiguration, access logging, and mgr:info part of the cache manager interface. 4. Limitations: Caching in the initial implementation is not shared and not synchronized. Options that require exclusive, single-process access such as a single source port for HTCP queries will not be supported in SMP mode. Eventually, the associated functionality can be adjusted to work with multiple processes or threads.
Not a problem if handled the same way the listening ports are opened by master and passed in to children.
When logging via a pipe to a program, multiple program instances would be launched. I am sure other limitations will surface.
I see any situations needing full SMP support as probably choosing to use the TCP log daemon when it's ready. Its been designed with this multi-instance and multi-host installations in mind.
The file daemon can probably be migrated to a special localhost version of that under SMP conditions.
For the forking of child processes please begin the use of vfork(). All the 64-bit installations heard of recently are encountering memory and resource related problems which can be traced back to fork() and the amount of RAM squid uses.
It sounds like what we have discussed and agreed already. But hard to tell without code to read.
+1/-0 from me. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE23 Current Beta Squid 3.1.0.16
