Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Wendell Hatcher]

Also, sorry I didn't see the APR question below.

httpd -V

Server version: Apache/2.4.46 (Unix)

Server built: Nov 24 2020 16:30:58

Server's Module Magic Number: 20120211:93

Server loaded: APR 1.7.0, APR-UTIL 1.6.1

Compiled using: APR 1.7.0, APR-UTIL 1.6.1

Architecture: 64-bit

Server MPM: worker

threaded: yes (fixed thread count)

forked: yes (variable process count)

Server compiled with

-D APR_HAS_SENDFILE

-D APR_HAS_MMAP

-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)

-D APR_USE_PROC_PTHREAD_SERIALIZE

-D APR_USE_PTHREAD_SERIALIZE

-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT

-D APR_HAS_OTHER_CHILD

-D AP_HAVE_RELIABLE_PIPED_LOGS

-D DYNAMIC_MODULE_LIMIT=256

-D HTTPD_ROOT="/home/rc/objs/ssws_ssws_rh6/linux/install"

-D SUEXEC_BIN="/home/rc/objs/ssws_ssws_rh6/linux/install/bin/suexec"

-D DEFAULT_PIDLOG="logs/httpd.pid"

-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"

-D DEFAULT_ERRORLOG="logs/error_log"

-D AP_TYPES_CONFIG_FILE="conf/mime.types"

-D SERVER_CONFIG_FILE="conf/httpd.conf"

svc...@web70.lab1.ariba.com>

On Mon, Dec 7, 2020, 5:41 PM Yann Ylavic  wrote:

> On Mon, Dec 7, 2020 at 6:33 PM Wendell Hatcher
>  wrote:
> >
> > Yann, quick question would it be ok to have a cronjob running daily
> maybe at slow periods that runs a script to delete the sephmore segments
> using a script while apache is running? It shouldn't cause issues with the
> apache server processing data correct?
>
> I think that the error is about shared memories (SHM), not semaphores
> (your httpd -V output shows "-D APR_USE_PROC_PTHREAD_SERIALIZE" which
> indicates the use of pthread global mutexes, not backed by a system
> descriptor/inode, so nothing to cleanup really).
>
> The SHMs usually do have a descriptor or inode to delete somewhere on
> your system, but I don't think it's safe to do so while httpd is
> running.
> By the way, you didn't tell which SHM mechanism was used by the APR
> library on your system (which system? httpd/APR compiled manually?
> SysV, POSIX or other SHM mechanism? Where do your SHMs land on the
> filesystem?).
> It is not shown by httpd -V unfortunately (I added it to trunk lately,
> but it's not in any 2.4 version yet).
>
>
> > Second ndly, we want to run a script during our apache stop process that
> will deleting the files from the shm folder we are hoping this will
> reschedule le our hanging apache servers during stops and starts with a
> slotmem error.
>
> I'm not sure to understand this, deleting the SHMs could avoid
> conflicts temporarily but as soon as all the httpd instances are
> started again, the conflicts will show up again if they exist in the
> configuration files.
>
> One particular thing to look at are the URLs used in  balancer://[id]> declarations, in all the instances.
> Each balancer should have a unique [id], system wide, because a SHM
> will be created (system wide) based on this [id]. This is usually not
> an issue with a single httpd instance where uniqueness makes sense and
> balancer1,..,n is simply/often used, but when this first instance gets
> copied to create others then it breaks..
> You can use almost anything as [id], provided it's unique and the same
> [id] is used in the associated ProxyPass. It can be a UUID for
> example, a tool like `uuidgen` can generate one when a new balancer
> needs to be added.
>
> Hth,
> Yann.
>

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Wendell Hatcher]

Awesome, thanks! This is what I figured as well that the issue will
continue if we did not update the balancer. Let me look into it further and
see if we can figure it out how to update the balancer names to unique IDs
I didn't think about this and thought about splitting our mod_proxy conf
files using multiple vhosts. This wouldn't work as we use the same fill url
call https://server one.com for all services.

Below is what we're thinking of doing but I feel it will not solve the
problem and am worried that deleting the sephmore segments while the server
is running can cause issues in production.

Our plan to patch the problem below.

Update kernel parameters: kernel.sem= 500 32000 200 256 (inside of the
/etc/sysctl.conf ) (Completed in devlab).

3. Use root crontab job to remove unused semaphores (Not Completed) :

for x in $(ipcs -m | awk 'NR>1 {print $2}'); do ipcrm -m $x; done


4. Update  shutdown apache script to remove .shm files during stopsvc (Not
Completed)




On Mon, Dec 7, 2020, 5:41 PM Yann Ylavic  wrote:

> On Mon, Dec 7, 2020 at 6:33 PM Wendell Hatcher
>  wrote:
> >
> > Yann, quick question would it be ok to have a cronjob running daily
> maybe at slow periods that runs a script to delete the sephmore segments
> using a script while apache is running? It shouldn't cause issues with the
> apache server processing data correct?
>
> I think that the error is about shared memories (SHM), not semaphores
> (your httpd -V output shows "-D APR_USE_PROC_PTHREAD_SERIALIZE" which
> indicates the use of pthread global mutexes, not backed by a system
> descriptor/inode, so nothing to cleanup really).
>
> The SHMs usually do have a descriptor or inode to delete somewhere on
> your system, but I don't think it's safe to do so while httpd is
> running.
> By the way, you didn't tell which SHM mechanism was used by the APR
> library on your system (which system? httpd/APR compiled manually?
> SysV, POSIX or other SHM mechanism? Where do your SHMs land on the
> filesystem?).
> It is not shown by httpd -V unfortunately (I added it to trunk lately,
> but it's not in any 2.4 version yet).
>
>
> > Second ndly, we want to run a script during our apache stop process that
> will deleting the files from the shm folder we are hoping this will
> reschedule le our hanging apache servers during stops and starts with a
> slotmem error.
>
> I'm not sure to understand this, deleting the SHMs could avoid
> conflicts temporarily but as soon as all the httpd instances are
> started again, the conflicts will show up again if they exist in the
> configuration files.
>
> One particular thing to look at are the URLs used in  balancer://[id]> declarations, in all the instances.
> Each balancer should have a unique [id], system wide, because a SHM
> will be created (system wide) based on this [id]. This is usually not
> an issue with a single httpd instance where uniqueness makes sense and
> balancer1,..,n is simply/often used, but when this first instance gets
> copied to create others then it breaks..
> You can use almost anything as [id], provided it's unique and the same
> [id] is used in the associated ProxyPass. It can be a UUID for
> example, a tool like `uuidgen` can generate one when a new balancer
> needs to be added.
>
> Hth,
> Yann.
>

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Yann Ylavic]

On Mon, Dec 7, 2020 at 6:33 PM Wendell Hatcher
 wrote:
>
> Yann, quick question would it be ok to have a cronjob running daily maybe at 
> slow periods that runs a script to delete the sephmore segments using a 
> script while apache is running? It shouldn't cause issues with the apache 
> server processing data correct?

I think that the error is about shared memories (SHM), not semaphores
(your httpd -V output shows "-D APR_USE_PROC_PTHREAD_SERIALIZE" which
indicates the use of pthread global mutexes, not backed by a system
descriptor/inode, so nothing to cleanup really).

The SHMs usually do have a descriptor or inode to delete somewhere on
your system, but I don't think it's safe to do so while httpd is
running.
By the way, you didn't tell which SHM mechanism was used by the APR
library on your system (which system? httpd/APR compiled manually?
SysV, POSIX or other SHM mechanism? Where do your SHMs land on the
filesystem?).
It is not shown by httpd -V unfortunately (I added it to trunk lately,
but it's not in any 2.4 version yet).


> Second ndly, we want to run a script during our apache stop process that will 
> deleting the files from the shm folder we are hoping this will reschedule le 
> our hanging apache servers during stops and starts with a slotmem error.

I'm not sure to understand this, deleting the SHMs could avoid
conflicts temporarily but as soon as all the httpd instances are
started again, the conflicts will show up again if they exist in the
configuration files.

One particular thing to look at are the URLs used in  declarations, in all the instances.
Each balancer should have a unique [id], system wide, because a SHM
will be created (system wide) based on this [id]. This is usually not
an issue with a single httpd instance where uniqueness makes sense and
balancer1,..,n is simply/often used, but when this first instance gets
copied to create others then it breaks..
You can use almost anything as [id], provided it's unique and the same
[id] is used in the associated ProxyPass. It can be a UUID for
example, a tool like `uuidgen` can generate one when a new balancer
needs to be added.

Hth,
Yann.

-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Wendell Hatcher]

Yann, quick question would it be ok to have a cronjob running daily maybe
at slow periods that runs a script to delete the sephmore segments using a
script while apache is running? It shouldn't cause issues with the apache
server processing data correct? Second ndly, we want to run a script during
our apache stop process that will deleting the files from the shm folder we
are hoping this will reschedule le our hanging apache servers during stops
and starts with a slotmem error.

-Wendell


On Sun, Dec 6, 2020, 6:45 PM Wendell Hatcher 
wrote:

> Will do! I appreciate the help with this as well.
>
> On Sun, Dec 6, 2020, 6:26 PM Yann Ylavic  wrote:
>
>> The logs from mod_slotmem_shm are the most interesting, so "LogLevel
>> slotmem_shm:debug" should be enough (to limit debug logging to this
>> module).
>> This should show which shared memories paths (i.e.
>> .../slotmem-shm-*.shm) are created/reused/attached by which instance,
>> and figure out what kind of duplicates there may be..
>>
>> On Sun, Dec 6, 2020 at 8:21 PM Wendell Hatcher
>>  wrote:
>> >
>> > Let me gather this information from one of our servers next week. I
>> will provide a little more background information tomorrow as well.
>> >
>> > On Sat, Dec 5, 2020, 7:08 PM Yann Ylavic  wrote:
>> >>
>> >> On Fri, Dec 4, 2020 at 7:45 PM Wendell Hatcher
>> >>  wrote:
>> >> >
>> >> > This has nothing to do with underlying SHM stuff and everything to
>> do with how multiple balancers, global and outside of virtual hosts, are
>> initialized and sometimes persisted. This was changed greatly in 2.4.29.You
>> should find the minimal set of balancers that reproduces the error and It
>> is likely related to duplicate virtual hosts or duplicate proxy "workers".
>> >>
>> >> Could you please provide the full error_log (with LogLevel debug) of
>> >> the httpd startup failure?
>> >> If there are multiple instances (and potential duplicates), the
>> >> error_log of all the instances may help too (with LogLevel debug,
>> >> still).
>> >>
>> >> Regards;
>> >> Yann.
>> >>
>> >> -
>> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
>> >> For additional commands, e-mail: users-h...@httpd.apache.org
>> >>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
>> For additional commands, e-mail: users-h...@httpd.apache.org
>>
>>

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Wendell Hatcher]

Will do! I appreciate the help with this as well.

On Sun, Dec 6, 2020, 6:26 PM Yann Ylavic  wrote:

> The logs from mod_slotmem_shm are the most interesting, so "LogLevel
> slotmem_shm:debug" should be enough (to limit debug logging to this
> module).
> This should show which shared memories paths (i.e.
> .../slotmem-shm-*.shm) are created/reused/attached by which instance,
> and figure out what kind of duplicates there may be..
>
> On Sun, Dec 6, 2020 at 8:21 PM Wendell Hatcher
>  wrote:
> >
> > Let me gather this information from one of our servers next week. I will
> provide a little more background information tomorrow as well.
> >
> > On Sat, Dec 5, 2020, 7:08 PM Yann Ylavic  wrote:
> >>
> >> On Fri, Dec 4, 2020 at 7:45 PM Wendell Hatcher
> >>  wrote:
> >> >
> >> > This has nothing to do with underlying SHM stuff and everything to do
> with how multiple balancers, global and outside of virtual hosts, are
> initialized and sometimes persisted. This was changed greatly in 2.4.29.You
> should find the minimal set of balancers that reproduces the error and It
> is likely related to duplicate virtual hosts or duplicate proxy "workers".
> >>
> >> Could you please provide the full error_log (with LogLevel debug) of
> >> the httpd startup failure?
> >> If there are multiple instances (and potential duplicates), the
> >> error_log of all the instances may help too (with LogLevel debug,
> >> still).
> >>
> >> Regards;
> >> Yann.
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
> >> For additional commands, e-mail: users-h...@httpd.apache.org
> >>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
> For additional commands, e-mail: users-h...@httpd.apache.org
>
>

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Yann Ylavic]

The logs from mod_slotmem_shm are the most interesting, so "LogLevel
slotmem_shm:debug" should be enough (to limit debug logging to this
module).
This should show which shared memories paths (i.e.
.../slotmem-shm-*.shm) are created/reused/attached by which instance,
and figure out what kind of duplicates there may be..

On Sun, Dec 6, 2020 at 8:21 PM Wendell Hatcher
 wrote:
>
> Let me gather this information from one of our servers next week. I will 
> provide a little more background information tomorrow as well.
>
> On Sat, Dec 5, 2020, 7:08 PM Yann Ylavic  wrote:
>>
>> On Fri, Dec 4, 2020 at 7:45 PM Wendell Hatcher
>>  wrote:
>> >
>> > This has nothing to do with underlying SHM stuff and everything to do with 
>> > how multiple balancers, global and outside of virtual hosts, are 
>> > initialized and sometimes persisted. This was changed greatly in 
>> > 2.4.29.You should find the minimal set of balancers that reproduces the 
>> > error and It is likely related to duplicate virtual hosts or duplicate 
>> > proxy "workers".
>>
>> Could you please provide the full error_log (with LogLevel debug) of
>> the httpd startup failure?
>> If there are multiple instances (and potential duplicates), the
>> error_log of all the instances may help too (with LogLevel debug,
>> still).
>>
>> Regards;
>> Yann.
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
>> For additional commands, e-mail: users-h...@httpd.apache.org
>>

-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Wendell Hatcher]

Let me gather this information from one of our servers next week. I will
provide a little more background information tomorrow as well.

On Sat, Dec 5, 2020, 7:08 PM Yann Ylavic  wrote:

> On Fri, Dec 4, 2020 at 7:45 PM Wendell Hatcher
>  wrote:
> >
> > This has nothing to do with underlying SHM stuff and everything to do
> with how multiple balancers, global and outside of virtual hosts, are
> initialized and sometimes persisted. This was changed greatly in 2.4.29.You
> should find the minimal set of balancers that reproduces the error and It
> is likely related to duplicate virtual hosts or duplicate proxy "workers".
>
> Could you please provide the full error_log (with LogLevel debug) of
> the httpd startup failure?
> If there are multiple instances (and potential duplicates), the
> error_log of all the instances may help too (with LogLevel debug,
> still).
>
> Regards;
> Yann.
>
> -
> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
> For additional commands, e-mail: users-h...@httpd.apache.org
>
>

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Yann Ylavic]

On Fri, Dec 4, 2020 at 7:45 PM Wendell Hatcher
 wrote:
>
> This has nothing to do with underlying SHM stuff and everything to do with 
> how multiple balancers, global and outside of virtual hosts, are initialized 
> and sometimes persisted. This was changed greatly in 2.4.29.You should find 
> the minimal set of balancers that reproduces the error and It is likely 
> related to duplicate virtual hosts or duplicate proxy "workers".

Could you please provide the full error_log (with LogLevel debug) of
the httpd startup failure?
If there are multiple instances (and potential duplicates), the
error_log of all the instances may help too (with LogLevel debug,
still).

Regards;
Yann.

-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Wendell Hatcher]

Yes, it is literal we are still seeing the issue. I saw a work around of
minimizing the proxy.conf file and adding vhosts to split up the calls.

*The https://bz.apache.org/bugzilla/show_bug.cgi?id=62044
 bug fix which was
recently updated with a work around on  09/2020. *



*Dasharath Masirkar 2020-09-02 09:11:57 UTC*

*- This issue can be workaround by configuring multiple instances of httpd
to handle such high vhost and proxy balancer configuration instead of
putting all in single instance. I have tested it and works for me.* We may
have an issue with having *so many items in our proxy balancer config file
and splitting it up with separate vhosts could solve the problem.*



This has nothing to do with underlying SHM stuff and *everything to do with
how multiple balancers, global and outside of virtual hosts, are
initialized and sometimes persisted. *This was changed greatly in
2.4.29.You should find the minimal set of balancers that reproduces the
error and It is likely related to duplicate virtual hosts or duplicate
proxy "workers".

https://bz.apache.org/bugzilla/show_bug.cgi?id=62044

On Wed, Aug 26, 2020 at 6:18 PM Wendell Hatcher <
wendellhatcher1...@gmail.com> wrote:

> >
> > Still seeing, as in a problem that you had both before and after
> > the update? Yes both before the update and after. This was during a
> shutdown. Why is it occurring during a bad shutdown? Yes, we have multiple
> instances running?
>
> Wendell
>
> > On Aug 26, 2020, at 6:14 PM, Nick Kew  wrote:
> >
> > On Wed, 26 Aug 2020 16:34:00 -0400
> > Wendell Hatcher  wrote:
> >
> >> Hello, I updated to the latest version of the APR and Apache but I am
> >> still seeing the following slotmem error.
> >
> > Still seeing, as in a problem that you had both before and after
> > the update? Yes both before the update and after. This was during a
> shutdown. Why is it occurring during a bad shutdown? Yes, we have multiple
> instances running?
> >
> >>  We run multiple apache
> >> services on a single box as well.
> >
> > Multiple instances of the server, all in the same filesystem?
> >
> >> error.2020.08.26:[Wed Aug 26 09:37:42.188705 2020]
> >> [slotmem_shm:error] [pid 125624:tid 140664905795392] (17)File exists:
> >> AH02611: create:
> >
> > That would most usually happen if a process doesn't shut down cleanly,
> > and so fails to clean up filesystem entries (e.g. a crash or kill).
> > But your "multiple services" comment leads me to wonder if you have
> > two or more instances in conflict.
> >
> >> apr_shm_create(lotmem-shm-p7bed9088_s4taskcxml_c3_0.shm) failed
> >
> > Is that a literal transcription?  I'm looking at the apparent
> > argument there!
> >
> > --
> > Nick Kew
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
> > For additional commands, e-mail: users-h...@httpd.apache.org
> >
>

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Wendell Hatcher]

> 
> Still seeing, as in a problem that you had both before and after
> the update? Yes both before the update and after. This was during a shutdown. 
> Why is it occurring during a bad shutdown? Yes, we have multiple instances 
> running? 

Wendell

> On Aug 26, 2020, at 6:14 PM, Nick Kew  wrote:
> 
> On Wed, 26 Aug 2020 16:34:00 -0400
> Wendell Hatcher  wrote:
> 
>> Hello, I updated to the latest version of the APR and Apache but I am
>> still seeing the following slotmem error.
> 
> Still seeing, as in a problem that you had both before and after
> the update? Yes both before the update and after. This was during a shutdown. 
> Why is it occurring during a bad shutdown? Yes, we have multiple instances 
> running? 
> 
>>  We run multiple apache
>> services on a single box as well.
> 
> Multiple instances of the server, all in the same filesystem?
> 
>> error.2020.08.26:[Wed Aug 26 09:37:42.188705 2020]
>> [slotmem_shm:error] [pid 125624:tid 140664905795392] (17)File exists:
>> AH02611: create:
> 
> That would most usually happen if a process doesn't shut down cleanly,
> and so fails to clean up filesystem entries (e.g. a crash or kill).
> But your "multiple services" comment leads me to wonder if you have
> two or more instances in conflict.
> 
>> apr_shm_create(lotmem-shm-p7bed9088_s4taskcxml_c3_0.shm) failed
> 
> Is that a literal transcription?  I'm looking at the apparent
> argument there!
> 
> -- 
> Nick Kew
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
> For additional commands, e-mail: users-h...@httpd.apache.org
> 

-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

Re: [users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Nick Kew]

On Wed, 26 Aug 2020 16:34:00 -0400
Wendell Hatcher  wrote:

> Hello, I updated to the latest version of the APR and Apache but I am
> still seeing the following slotmem error.

Still seeing, as in a problem that you had both before and after
the update?

>   We run multiple apache
> services on a single box as well.

Multiple instances of the server, all in the same filesystem?

>  error.2020.08.26:[Wed Aug 26 09:37:42.188705 2020]
> [slotmem_shm:error] [pid 125624:tid 140664905795392] (17)File exists:
> AH02611: create:

That would most usually happen if a process doesn't shut down cleanly,
and so fails to clean up filesystem entries (e.g. a crash or kill).
But your "multiple services" comment leads me to wonder if you have
two or more instances in conflict.

> apr_shm_create(lotmem-shm-p7bed9088_s4taskcxml_c3_0.shm) failed

Is that a literal transcription?  I'm looking at the apparent
argument there!

-- 
Nick Kew

-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

[users@httpd] slotmem error still occuring after updating to Apache 2.4.43 APR 1.7.0

[Wendell Hatcher]

Hello, I updated to the latest version of the APR and Apache but I am still
seeing the following slotmem error.  We run multiple apache services on a
single box as well.

 error.2020.08.26:[Wed Aug 26 09:37:42.188705 2020] [slotmem_shm:error]
[pid 125624:tid 140664905795392] (17)File exists: AH02611: create:
apr_shm_create(lotmem-shm-p7bed9088_s4taskcxml_c3_0.shm) failed

Server version: Apache/2.4.43 (Unix)
Server built:   Aug 11 2020 20:42:25
Server's Module Magic Number: 20120211:92
Server loaded:  APR 1.7.0, APR-UTIL 1.6.1
Compiled using: APR 1.7.0, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM: worker
  threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_PROC_PTHREAD_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
'
.kernel.sem = 500   32000   200 256