Thanks for reporting the problem Norm,
The patch was submitted.
Jean-Jacques [EMAIL PROTECTED] 10/24/04 5:08 PM
Greetings All,Just trying a 'build' of current 2.1 CVS on a Windows machine for NetWare and get the following error...Calling NWGNUmod_cachCompiling cache_util.cCompiling
);- (*new) = (char *)header;+ (*new) = header; [EMAIL PROTECTED] 10/25/04 10:32 AM
Jean-Jacques Clar wrote: Thanks for reporting the problem Norm, The patch was submitted.Is casting to (char *) really the right thing to do here? I mean why not just make new const so the cast isn't needed at all
All the directives for mod_mem_cache are preceded with an "M".
The ones for mod_cache are starting with a "c"
Should all the disk_cache directives be preceded with a "D" for
consistency and clarity?
I know it is going to break every user configuration, but before
the modules move out of
[EMAIL PROTECTED] 10/07/04 11:14 AM
--On Thursday, October 7, 2004 9:08 AM -0600 Jean-Jacques Clar [EMAIL PROTECTED] wrote:I *really* don't like the M prefix at all. I'd much prefer us to just spell the thing out: CacheMem* and CacheDisk* instead of MCache and/or DCache. I think a single letter
Norm,
If you can just do a cut and paste of the define
from the windows header into apr.hnw for today.
Tomorrow I will figure out what is the right thing to do for
NetWare.
Have a good day,
JJ [EMAIL PROTECTED] 09/23/04 4:18 PM
NormW wrote: Compiling network_io/win32/sendrecv.c ### mwccnlm
Should the cleanup field be removed from the object structure in
mod_cache.h? [EMAIL PROTECTED] 09/17/04 9:03 AM
stoddard 2004/09/17 08:03:08 Modified: modules/experimental Tag: APACHE_2_0_BRANCH mod_mem_cache.c Log: eliminate cleanup bit in favor of managing the object exclusively with the
Are there any reasons why that part of decrement_refcount() is not
needed anymore?
@@ -267,31 +271,9 @@ { cache_object_t *obj = (cache_object_t *) arg;- /* If obj-complete is not set, the cache update failed and the- * object needs to be removed from the cache then cleaned up.- */- if
Oups!, sorry I was few minutes slower than you Greg.
The overall patch looks good and much cleaner.
I would love to move my office next to both of you...
I will put the new code on my mp box and make sure
It does not break.
Thanks,
JJ [EMAIL PROTECTED] 09/16/04 9:50 AM
Bill Stoddard wrote: -
+1
Tried differentconfigurations in lab and changes are acting as expected.
Thanks to Greg and Bill S. for fixing the problem.
JJ [EMAIL PROTECTED] 09/16/04 11:39 AM
Adds back obj-complete check in decrement_refcount. Makes changes recommended by Greg.
[EMAIL PROTECTED] 09/15/04 12:41 PM
I'm still trying to learn the complete design though. I see that memcache_cache_free() can return with OBJECT_CLEANUP_BIT set, and that decrement_refcount() will react to it and free the memory if it is registered as a cleanup. Will decrement_refcount()
Can you help me/us understand the following a little better? If it is possible for two threads to simultaneously to have the same obj pointer and try to free the object, what prevents a seg fault* due to:- thread #1 gets a pointer to obj A, then looses its time slice due to the OS scheduler
I share your enjoyment.
Should I then go ahead and commit my patch to the 2.1 tree?Thanks,
JJ [EMAIL PROTECTED] 09/14/04 3:41 PM
Jean-Jacques Clar wrote: Can you help me/us understand the following a little better? Let's look at al the cases where the cleanup function is called or the object
This one is working as expected on my server. I tested most
of the paths and it looks fine. Added comments to the previous
path. Will like some double-checking and feedback if possible.
The race condition is fixed, no performance hit like with
the mutex patch, and no memory leak, from what I
[EMAIL PROTECTED] 09/10/04 7:33 PM
At 06:53 PM 9/10/2004, Jean-Jacques Clar wrote:I replaced the cleanup field with a bit set in refcount. This is done to prevent race conditions when refcount is accessed on two different threads/CPUS.+#define OBJECT_CLEANUP_BIT 0x00F00x00F0 isn't a bit
The following patch is a rework ofthe refcount and cleanup
fields in mod_mem_cache.c. I replaced the cleanup field with a bit set
in refcount. This is done to prevent race conditions when refcount
is accessed on two different threads/CPUS. The complete description
was done on thatmail thread
[EMAIL PROTECTED] 09/09/04 2:27 AM
As far as performance, I currently don't see any slow down on my box, but will run longer tests tomorrow morning. no worries about that hereLooking more closely at the implication of the submittedpatch,
I don't think we want to lock decrement_refcount() with
It should not be possibe for two threads to atomically decrement the refcount on the same object to 0.
I think there is a small window in there where it is possible to have the decs
happening on both CPUs one after the other making that bug possible in
decrement_refcount() and
I am usinga set of files I got from VeriTest
http://www.etestinglabs.com/benchmarks/webbench/default.asp
The tar file include about 6000 files of a total size around 60 MB.
Has you can see in the configuration MCacheSize and
MCacheMaxObjectCount force cached objects to be ejected
often. This
I find the possible need for the patch very disturbing; it implies tome a problem in the atomics code
I don't know.
I had the NetWare dec code reviewedby the people who
wrote/maintainour kernel, this is why I changed the
apr_atomic_dec(), and from what I understandthe
problem should be ours.
That's bad already. At any time when there are n threads with a"handle" to a cache object, refcount ought to be n in order to be ableto free it when refcount goes to zero.
Unless:
Condition 1) a mutex is held between the time that a thread gets ahandle to the cache object and when the thread
Testing 2.0 and 2.1 Head.
I am running a test that requires frequent ejections of cache entities:
max_cache_size and max_object_countare smaller than my sampling.
2 threads running at the concurrentlyon 2 CPUs.
Thread1(T1): an entry is ejected from the cache in cache_insert(),
resulting in a
I am seeing a double free happening on a removed
cache entry on my MP box.
It is a race condition that is happening because of
the way apr_atomic_dec is implemented on NetWare.
I should have a fix for it later today.
JJ
Index: core.c + char *w = strsep(p, ",");
strsep() seems to be platform dependant. That functiondoes not
exist on NetWare, and I don't think it exists on Windows.
It should at least be made anAPRfunction.
Here is the notes from the man page:
The strsep() function was introduced as a
3- overload detach field from cgi_exec_info_t in 2.0 to make sure no bump are needed, no backport entry needed in that case.Not sure what you mean there, any proposed backport to theAPACHE_2_0_BRANCH for httpd needs to be added to STATUS following thenormal procedure.The change done in 2.1
I'm writing an optimized caching module.
Would you please share with us some info on that
optimized caching module.
Is mod_url_cache a modified copy of the current ones?
Why are the current onesnotfulfilling your needs?
Thanks,
JJ
I am getting the following error when running CGIs on the current
version of NetWare (6.5 sp2):
(32)Broken pipe: ap_content_length_filter: apr_bucket_read() failed
I am working on tracking down the problem.
Changes done to mod_cgi.c, mod_include.(h c) back in August 22, 2003
are exposinga
The problem is in our own library. The recent changes cause itto
don't catch the end of the generated file and go on reading it until
the read fails.
In 2.1 previous tomod_cgi.c:r1.153
the return value of ap_pass_brigade() was not checked in
include_cmd(), but now it is done. That change
To replace the addrspace field that was added in the cgi_exec_info_t
struct in mod_cgi.hI will like to propose extendingthe use of the
detached (apr_int32_t) field in cgi_exec_info_t and apr_procattr_t structs.
Currently that field is set to 0 by default and1 if the process to be
created will
Loading a process in its own address space require that all of the
modules that it has direct dependencieson,have alsoto be loaded
in that same address space. This is an expensive process, especially
when it comes to CGIs that are loaded to serve their content and
then unloaded.
A marshaling
-Jacques Clar wrote: new field in apr_procattr_t and cgi_exec_info_t structuresThis, however, would require one, if a module would ever be responsiblefor allocating the apr_procattr_t or cgi_exec_info_t themselves, since ifthey're and old build, they'd allocate too little space
Jean-Jacques Clar 5/18/2004 10:57:40 AM
Just replaced tabs with spaces and reworked indentation within brackets.
If no objections will commit later.
Thanks,
JJ
sorry I had to zip the patch, size was causing a failure from apache mail server:
ezmlm-reject: fatal: Sorry, I don't accept
Duh?
What is that?
I was just playing with Cervisia and gvcs setting up my connection to cvs and updating my files.
Did I screw up something?
JJ [EMAIL PROTECTED] 4/27/2004 11:28:25 AM
On Tue, 27 Apr 2004 [EMAIL PROTECTED] wrote: clar 2004/04/27 11:22:27 Log: no message Status: Vendor Tag:
I added comments within the following lines (jjc) that are not in the attached patch.
I am just explaining what I did.
My only question is why 0.5 is added to the mean value when displaying the results (lines 988-90-92-94)?
Please review the attached patch and let me know if my corrections are
Sometime negative values are displayed in the min column of the Connection Times statistics.
I surewill find out why, but the following patch make sure that cannot happen and does not hurt.
Please comments.
If no comments, I will commit it later today.
Thanks,
@@ -1286,9 +1286,9 @@ c-done =
Just a quick surveyon how robust ab should be.
Ithink that ab should not seg fault on any user parameters,
Icould spent some time making it a little bit more robust.
Is it of any interest, or the general thinking is; if the user enters an out of range or a bogus value and seg fault, this
The added call to usage() on line 2165 is missing the closing parenthesis.
JJ [EMAIL PROTECTED] 3/17/2004 11:22:35 AM
madhum 2004/03/17 10:22:35 Modified: support ab.c Log: Limit the concurrency to MAX_CONCURRENCY. Otherwise, ab may dump core (calloc fails) when a arbitrarily huge value is
+1 NetWare
Jean-Jacques [EMAIL PROTECTED] 3/13/2004 5:32:53 AM
Hi,There are 2.0.49-rc2 tarballs available at: http://httpd.apache.org/dev/dist/The differences with respect to the rc1 tarball are:- BeOS specific MPM fixes- Netware specific rand.c fixes- Documentation update- Berkeley DB
I am Inside the conditional filter in mod_cache,
output filter type = AP_FTYPE_CONTENT_SET-2 (18),
and the r-status field is currently a 304. I would like
to change it to a 200, but it looks like the status field
was already stuffed in the rec-headers_out table,
or is it somewhere else?
Is
Joe,
You will probably need more info, but if you have time give it a try and
if you run into problems we could contact greg ames from the ASF.
Thanks,
JJ
[EMAIL PROTECTED] 1/12/2004 9:45:50 AM
Jean-Jacques Clar wrote: Attached are 2.0.48 numbers on RH AS 2.1 and 3.0. Apache is build
Who has the right to vote for backport changes,
is it only members or that right also apply to committers?
Thank you,
Jean-Jacques
This patch make it possible toreturn 304s when using the quick handler,
ifa fresh cache entity is found.
The r-mtime field is usually set in default_handler() (core.c) by calling
ap_update_mtime(), which does not happen when the quick handler
find a fresh cache entry.
What did you use for ThreadsPerChild? The default?
yes we used the default values for the worker threading model.
I'm very curious to know why prefork is beating worker. Could you get a profile of the CPU usage for both cases? I've had good luck with oprofile. I had a little trouble compiling
HyperThreading enabled or not when limited to 1/2 CPUs, if these are HTCPUs? That can make a difference (either way) when benchmarking IIRC.
CPUs are not HT.
The current request falls in one of the MUST RFC condition,
line 168 mod_cache.c:
---
else { if (ap_cache_liststr(NULL, cc_in, "no-store", NULL) || ap_cache_liststr(NULL, pragma, "no-cache", NULL) || (auth != NULL)) { /* delete the previously cached file */ cache_remove_url(r,
1. What was the CPU utilization during the tests
I think CPU utilization is above 90% during the test,
at least on my 4 CPUs box, which is not the one used to
gather the submitted results. I could find out but will be
delighted to have more details on what you are exactly
looking for based on
These are comments copied from ap_cache_check_freshness()
line 163 cache_util.c:
--
* - RFC2616 14.9.4 End to end reload, Cache-Control: no-cache. no-cache in * either the request or the cached response means that we must * revalidate the request unconditionally, overriding
Bill,
The patch you committed is only for21285.
My bet for 21287 (no mutex lock protection in decrement_refcount).
from the bug description:
"There are no mutex lock protection in decrement_refcount if it isdefined USE_ATOMICS.I think you simply forgot the mutex in function
Attached are 2.0.48 numbers on RH AS 2.1 and 3.0.
Apache is build with worker MPM and default options on both versions.
C:
Apache is servicing more requests per secon 2.1 on 1 and 2 CPUs, 3.0 is picking up the slack between 2 and 4 CPUs.
Prefork still serving static requests faster than worker
There is a memory leak with your patch when running my test.
Using only the following part, it does not leak and works fine;
+ if (obj-cleanup) {+ /* If obj-cleanup is set, the object has been prematurly+ * ejected from the cache by the garbage collector. Add the+ * object back to the cache. If
Changes in decrement_refcount() and remove_url().
Patch extracted and modified from attachment in Bug 21285.
The submitted patch was combining fix for 21285 and 21287.
It is for decrement_refcount().
Alsoextended mutex protection in remove_url().
Thanks,
Jean-Jacques
Just replaced tabswith spaces to follow guidelines.
[EMAIL PROTECTED] 12/12/2003 1:23:20 PM
Changes in decrement_refcount() and remove_url().
Patch extracted and modified from attachment in Bug 21285.
The submitted patch was combining fix for 21285 and 21287.
It is for decrement_refcount().
Just replaced tabswith spaces to follow guidelines. [EMAIL PROTECTED] 12/11/2003 5:00:31 PM
Bugzilla Defect #21285
This is a rework of the already posted patch.
It address the following situation;
1- request comes in for streaming response
2- before that request could be completed, the entry
Bugzilla Defect #21285
This is a rework of the already posted patch.
It address the following situation;
1- request comes in for streaming response
2- before that request could be completed, the entry is ejected from the cache
3- when completing the write body step, the incomplete entry is
The fields:
apr_size_t cache_sizeapr_size_t object_cntfrom the struct mem_cache_confare useless since the introduction of
the cache_pqueue stuff.
Just removing them.
Thanks,
Jean-Jacques
mod_mem_cache.patch
Description: Binary data
The only thing I don't understand is why this went from:- queue_clock - mobj-total_refsto:queue_clock - mobj-total_refsWhen the other case negated the whole statement. Is this intentional ordid you forget to account for the - in this case being distributed acrossthe ()'s?
If you look at the
The heap in cache_pqueue is a max head, it always keep the object
with the highest priority in its root position. The object with the highest
priority is the onethat is ejected when a newelement has to be
inserted when the cache is full (max nb of entities or size reached).
The return value of
ather than negating the returns). I'd also add a comment infoo_algo where the negated value is stored to explain why it is negated.Cliff Woolley wrote: On Wed, 19 Nov 2003, Jean-Jacques Clar wrote: The heap in cache_pqueue is a max head, it always keep the object withthe highest priority in its roo
Priority value stored in the mem_cache_objectis = 0.
The stored valueincreases with thepriority of the cached entry.
A smaller value means a smaller priority.
All the logic in cache_pqueue for Bubbling Up and Percolating Down
is wrong. It does not ensure that the smallest element is always at
Looking at two pairs of fields used to keep the current cache size and object count:
from struct:
mem_cache_conf (mod_mem_cache.c):
apr_size_t cache_size
apr_size_t object_cnt
cache_cache_t (cache_cache.c):
apr_size_t current_size
cache_pqueue_t (cache_pqueue.c):
apr_size_t size (-1,
If they are no longer being used, remove them. I expect we were using them at one time, then Ian (?) added the nifty cache_pqueue stuff which made the fields superfluous.Thanks, will do.
MAXMEMFREEfixes were checked in the 2.1 tree last week for most of the MPMs
(except Win32). A back port has also been proposed.
I will like to propose insertion ofthe MAXMEMFREE directive in the configuration file.
I am suggesting that 100 (KBytes) should be used as a default size.
Used
This is a double send from last week (look at the date).
It was probably lost on a queue somewhere for a week.
Anyway, that proposal did not fly.
I will try to get memory/performance data on my Linux box by the end of the week.
As a possible solution for NetWare, we will put the MAXMEMFREE
MAXMEMFREEfixes were checked in the 2.1 tree last week for most of the MPMs
(except Win32). A back port has also been proposed.
I will like to propose insertion ofthe MAXMEMFREE directive in the configuration file.
I am suggesting that 100 (KBytes) should be used as a default size, but could
Having atime structure in the mpm keeping updated sec and uSEC
field could help performance for the web server.
It could replace most calls to apr_time_now() and apr_time_sec()
at run-time:
NetWare has its main thread resuming every 100 uSEC (1 sec)
to do the idle server maintenance.
I have two Questions/Suggestions to improve the current performance
of Apache 2.
1:
Apache and Apr are keeping the time stamp in uSEC, it implies multiple
calls to apr_time_sec(), which is just doing a 64 bits division to
transform
uSEC in sec.
I think this is a waste of CPU cycles for
The function ap_meets_conditions() is doing unnecessary calls to
apr_time_now() and apr_time_sec().
Sincethe variables (mtime)has itstarget valuein
second, it should
be possible to use r-request_time instead of calling
apr_time_now().
I did not see cases where the delay between a call to
Since the target value is in second, it should be possible to use
r-request_time instead of calling apr_time_now().
I ran a load test on my box and did not ever see a difference between the
sec value in r-request_time and the one returned by apr_time_now().
end of the patchpart..
Just bypassing a call to apr_filepath_merge by using the
canonical_filename.
Jean-Jacques
request.c.patch
Description: Binary data
the modules are still in the experimental area?
Thank you,
Jean-Jacques Clar
on the same allocator (and under the same limit) using
the new bucket_alloc_create_ex. Sound
good?Makes sense to me.Brian
Works for me.
Jean-Jacques Clar
I have been trying to understand why MaxMemFree is not working as
I am expecting it to in our MPM (NetWare).
I am getting to the conclusion that either the memory allocation in our
MPM
is missing a piece or there is a missing function in the apr_bucket
set.
I think this is relevant to other
Which values are youusing for MaxRequestsPerChild.
If mine is different than 0, then nice, I don't see a leak because
each of my Threads are kill once in a while.
But if it is = 0, then it leaks when refreshing entries in the cache,
could it be the fact in NetWare we have one process and
for the same 1KB file through 2.0.41-pre2
with the mod_mem_cache settings below, and I can't see
any sign of memory growth in the httpd yet.
Thanks,
Brian
On Tue, 2002-09-10 at 14:40, Jean-Jacques Clar wrote:
I guess my leak is from a different source.
Refreshing files in mod_mem_cache drive up
file through 2.0.41-pre2
with the mod_mem_cache settings below, and I can't see
any sign of memory growth in the httpd yet.
Thanks,
Brian
On Tue, 2002-09-10 at 14:40, Jean-Jacques Clar wrote:
I guess my leak is from a different source.
Refreshing files in mod_mem_cache drive up
properly on Netware, then all
kinds
of bad things might happen...
Bill
-Original Message-
From: Jean-Jacques Clar [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 12, 2002 10:38 AM
To: [EMAIL PROTECTED]
Subject: Re: memory leak in apr_buck_alloc...
I am using WebBench to run my
to determine the amount of leaked memory?
Jean-Jacques Clar wrote:
Using the atomic test, we do past all of them.
Is it enough or there are other actions that should be taken?
I ran another test requesting the same single small file (GIF, 223
bytes) from 5 clients
instead of 1. rate is at 275 req/sec
Why will I wouldn't see my allocated memory decrease if it has been
freed?
JJ
[EMAIL PROTECTED] 09/12/02 04:19PM
On Thu, Sep 12, 2002 at 04:16:35PM -0600, Jean-Jacques Clar wrote:
Memory in Apache never shrinks.
Is this a true statement?
When reaching a cruising state (stable load, same
Anyone with some kind of architecture diagrams related to the
memory management in APR?
Thanks,
Jean-Jacques Clar
I think I have been seeing the same thing for a long time when
refreshing entries
in mod_mem_cache. If you find something in the alloc path I
will be
more than happy to try it in my lab.
"Jean-Jacques Clar" [EMAIL PROTECTED] on 05/21/2002 06:23:58
PMTo: W G
Stoddard/R
I guess my leak is from a different source.
Refreshing files in mod_mem_cache drive up the allocated memory in
aprlib.
It is consistent and has been there since I started using mod_cache
(3-4 months ago).
Justin,
Were you using caching modules when you were able to reproduce/fix
Brad's leak?
htpasswd.patch
Description: Binary data
I
am building 2.0.39, but it was the same thing with
2.0.35. [EMAIL PROTECTED] 06/20/02 10:26AM
On Thu, Jun 20, 2002 at 09:09:45AM -0600, Jean-Jacques Clar
wrote: There are errors when building 2.0.x on Linux RH7.3 with the
workerWhich version of 2.0 exactly?-aaron
2002, Jean-Jacques Clar
wrote: I am building 2.0.39, but it was the same thing with
2.0.35. [EMAIL PROTECTED] 06/20/02 10:26AM
On Thu, Jun 20, 2002 at 09:09:45AM -0600, Jean-Jacques Clar
wrote: There are errors when building 2.0.x on Linux RH7.3 with the
worker Which version of 2.0 exactly
I have been using mem cachefor over two
months
onNetWare 6 servers with a great deal of
acceleration.
However, because ofissues to be resolved, it is used
only in a
test environment and not on production
servers.
--Jean-Jacques
On using the pq code:
I currently have it crashing in cache_pq_pop after only few
refresh ofa cache entry,
I am trying to fix it.
cache_pq_remove() - cache_pq_pop()-page
fault
Jean-Jacques
[EMAIL PROTECTED] 06/13/02 09:45AM
Arnauld Dravet wrote: Yup i know it's a work-in-progress
... I'm
[EMAIL PROTECTED] 06/06/02 03:33PM On Thu, Jun
06, 2002 at 03:25:01PM -0600, Jean-Jacques Clar wrote: Here are results
done on a 4 x 450Mhz server with 1 Gb of RAM, 1 SysKonnect Gb
LAN. The tests were done using WebBench with 26 clients and 1
controller running the default static test
86 matches
Mail list logo
