Awesome, thanks! This is what I figured as well that the issue will
continue if we did not update the balancer. Let me look into it further and
see if we can figure it out how to update the balancer names to unique IDs
I didn't think about this and thought about splitting our mod_proxy conf
files using multiple vhosts. This wouldn't work as we use the same fill url
call https://server one.com for all services.
Below is what we're thinking of doing but I feel it will not solve the
problem and am worried that deleting the sephmore segments while the server
is running can cause issues in production.
Our plan to patch the problem below.
Update kernel parameters: kernel.sem= 500 32000 200 256 (inside of the
/etc/sysctl.conf ) (Completed in devlab).
3. Use root crontab job to remove unused semaphores (Not Completed) :
for x in $(ipcs -m | awk 'NR>1 {print $2}'); do ipcrm -m $x; done
4. Update shutdown apache script to remove .shm files during stopsvc (Not
Completed)
On Mon, Dec 7, 2020, 5:41 PM Yann Ylavic <[email protected]> wrote:
> On Mon, Dec 7, 2020 at 6:33 PM Wendell Hatcher
> <[email protected]> wrote:
> >
> > Yann, quick question would it be ok to have a cronjob running daily
> maybe at slow periods that runs a script to delete the sephmore segments
> using a script while apache is running? It shouldn't cause issues with the
> apache server processing data correct?
>
> I think that the error is about shared memories (SHM), not semaphores
> (your httpd -V output shows "-D APR_USE_PROC_PTHREAD_SERIALIZE" which
> indicates the use of pthread global mutexes, not backed by a system
> descriptor/inode, so nothing to cleanup really).
>
> The SHMs usually do have a descriptor or inode to delete somewhere on
> your system, but I don't think it's safe to do so while httpd is
> running.
> By the way, you didn't tell which SHM mechanism was used by the APR
> library on your system (which system? httpd/APR compiled manually?
> SysV, POSIX or other SHM mechanism? Where do your SHMs land on the
> filesystem?).
> It is not shown by httpd -V unfortunately (I added it to trunk lately,
> but it's not in any 2.4 version yet).
>
>
> > Second ndly, we want to run a script during our apache stop process that
> will deleting the files from the shm folder we are hoping this will
> reschedule le our hanging apache servers during stops and starts with a
> slotmem error.
>
> I'm not sure to understand this, deleting the SHMs could avoid
> conflicts temporarily but as soon as all the httpd instances are
> started again, the conflicts will show up again if they exist in the
> configuration files.
>
> One particular thing to look at are the URLs used in <Proxy
> balancer://[id]> declarations, in all the instances.
> Each balancer should have a unique [id], system wide, because a SHM
> will be created (system wide) based on this [id]. This is usually not
> an issue with a single httpd instance where uniqueness makes sense and
> balancer1,..,n is simply/often used, but when this first instance gets
> copied to create others then it breaks..
> You can use almost anything as [id], provided it's unique and the same
> [id] is used in the associated ProxyPass. It can be a UUID for
> example, a tool like `uuidgen` can generate one when a new balancer
> needs to be added.
>
> Hth,
> Yann.
>
