Re: [OT] Re: mod_perl shared memory with MM
Perrin Harkins wrote: > > > Can I ask why you are not useing IPC::Sharedlight (as its pure C and > > apparently much faster than IPC::Shareable - I've never benchmarked it > > as I've also used IPC::Sharedlight). > > Full circle back to the original topic... > IPC::MM is implemented in C and offers an actual hash interface backed by a > BTree in shared memory. IPC::ShareLite only works for individual scalars. > Not tried that one ! I'ce used the obvious Sharedlight plus Storable to serialise hashes. > It wouldn't surprise me if a file system approach was faster than either of > these on Linux, because of the agressive caching. I would be an interesting benchmark ... Althought it may only be a performance win on a lightly loaded machine,the assymption being that the stat'ing is fast on a lowly loaded system with fast understressed disks. I could be completly wrong here tho ;-). Has anyone used the file system approach on a RAM disk ? Greg > > - Perrin
Re: [OT] Re: mod_perl shared memory with MM
> Can I ask why you are not useing IPC::Sharedlight (as its pure C and > apparently much faster than IPC::Shareable - I've never benchmarked it > as I've also used IPC::Sharedlight). Full circle back to the original topic... IPC::MM is implemented in C and offers an actual hash interface backed by a BTree in shared memory. IPC::ShareLite only works for individual scalars. It wouldn't surprise me if a file system approach was faster than either of these on Linux, because of the agressive caching. - Perrin
Re: [OT] Re: mod_perl shared memory with MM
DeWitt Clinton wrote:
>
> On Sun, Mar 11, 2001 at 03:33:12PM +0100, Christian Jaeger wrote:
>
> > I've looked at Cache::FileCache now and think it's (currently) not
> > possible to use for IPC::FsSharevars:
> >
> > I really miss locking capabilities. Imagine a script that reads a
> > value at the beginning of a request and writes it back at the end of
> > the request. If it's not locked during this time, another instance
> > can read the same value and then write another change back which is
> > then overwritten by the first instance.
>
> I'm very intrigued by your thinking on locking. I had never
> considered the transaction based approach to caching you are referring
> to. I'll take this up privately with you, because we've strayed far
> off the mod_perl topic, although I find it fascinating.
>
> > - why don't you use 'real' constants for $SUCCESS and the like? (use
> > constant)
>
> Two reasons, mostly historical, and not necessarily good ones.
>
> One, I benchmarked some code once that required high performance, and
> the use of constants was just slightly slower.
>
> Two, I like the syntax $hash{$CONSTANT}. If I remember correctly,
> $hash{CONSTANT} didn't work. This may have changed in newer versions
> of Perl.
>
> Obviously those are *very* small issues, and so it is mostly by habit
> that I don't use constant. I would consider changing, but it would
> mean asking everyone using the code to change too, because they
> currently import and use the constants as Exported scalars.
>
> Do you know of a very important reason to break compatibility and
> force the switch? I'm not opposed to switching if I have to, but I'd
> like to minimize the impact on the users.
>
> > - you probably should either append the userid of the process to
> > /tmp/FileCache or make this folder globally writeable (and set the
> > sticky flag). Otherwise other users get a permission error.
>
> As of version 0.03, the cache directories, but not the cache entries,
> are globally writable by default. Users can override this by changing
> the 'directory_umask' option, or keep data private altogether by
> changing the 'cache_root'. What version did you test with? There may
> be a bug in there.
>
> > - why don't you use Storable.pm? It should be much faster than Data::Dumper
>
> The TODO contains "Replace Data::Dumper with Storable (maybe)". :)
>
> The old File::Cache module used Storable, btw.
>
> It will be trivial to port the new Cache::FileCache to use Storable.
> I simply wanted to wait until I had the benchmarking code so I could
> be sure that Storeable was faster. Actually, I'm not 100% sure that I
> expect Storeable to be faster than Data::Dumper. If Data::Dumper
> turns out to be about equally fast, then I'll stay with it, because it
> is available on all Perl installations, I believe.
>
> Do you know if Storeable is definitely faster? If you have benchmarks
> then I am more than happy to switch now. Or, do you know of a reason,
> feature wise, that I should switch? Again, it is trivial to do so.
I've found it to be arround 5 - 10 % faster - on simple stuff on some
benchmarking I did arround a year ago.
Can I ask why you are not useing IPC::Sharedlight (as its pure C and
apparently much faster than IPC::Shareable - I've never benchmarked it
as I've also used IPC::Sharedlight).
Greg
>
> > >I have some preliminary benchmark code -- only good for relative
> > >benchmarking, but it is a start. I'd be happy to post the results
> > >here if people are interested.
> >
> > Could you send me the code?, then I'll look into benchmarking my
> > module too.
>
> I checked it in as Cache::CacheBenchmark. It isn't good code, nor
> does it necessarily work just yet. I simply checked it in while I was
> in the middle of working on it. I'm turning it into a real
> benchmarking class for the cache, and hopefully that will help you a
> little bit.
>
> Cheers,
>
> -DeWitt
Re: [OT] Re: mod_perl shared memory with MM
> I'm very intrigued by your thinking on locking. I had never
> considered the transaction based approach to caching you are referring
> to. I'll take this up privately with you, because we've strayed far
> off the mod_perl topic, although I find it fascinating.
One more suggestion before you take this off the list: it's nice to have
both. There are uses for explicit locking (I remember Randal saying he
wished File::Cache had some locking support), but most people will be happy
with atomic updates, and that's usually faster. Gunther's eXtropia stuff
supports various locking options, and you can read some of the reasoning
behind it in the docs at
http://new.extropia.com/development/webware2/webware2.html. (See chapters
13 and 18.)
> > - why don't you use 'real' constants for $SUCCESS and the like? (use
> > constant)
>
> Two reasons, mostly historical, and not necessarily good ones.
>
> One, I benchmarked some code once that required high performance, and
> the use of constants was just slightly slower.
Ick.
> Two, I like the syntax $hash{$CONSTANT}. If I remember correctly,
> $hash{CONSTANT} didn't work. This may have changed in newer versions
> of Perl.
No, the use of constants as hash keys or in interpolated strings still
doesn't work. I tried the constants module in my last project, and I found
it to be more trouble than it was worth. It's annoying to have to write
things like $hash{&CONSTANT} or "string @{[&CONSTANT]}".
> Do you know if Storeable is definitely faster?
It is, and it's now part of the standard distribution.
http://www.astray.com/pipermail/foo/2000-August/000169.html
- Perrin
Re: [OT] JMS-like event framework for Perl
Hi, Gunther!
>Although SOAP::Lite interfaces with SMTP and POP3, I don't know if
>SOAP::Lite really qualifies to be in the same class as products like
>JMS, Neon, or Microsoft MSMQ.
I think not or at least not yet :).
>However, I think the description of what the author is interested in
>is a bit vague. The words seem to indicate that he would be
>satisfied by setting up a SOAP service on an HTTP server that
>listens to calls made to it (the events) and then runs.
>This is a simpler model and does not really describe
>publish/subscribe as being a feature he wants.
Agree. It can be built as simple SOAP call against your server.
Though I'm finding many similarities between publish/subscribe and
async messaging they share common problem in this aspect: client has
to tell server where to run callback or response in async model. You
don't need it if you have static environment (when you're responsible
for client and server and client has fixed endpoint), but that's not
interesting case for us.
What can be done in my understanding (and I'm working on it), we can
pass either URL for service description that is able to handle
callback, or even fragment that describes it.
--
Notification service.
Author creates the resource (something that can be pointed by URL and
may be interesting for others) and registers it:
manageResource(URL, {name, description})
System responds with status of the request: registered or not.
Registration may fail if URL is already registered or system decided
you may not register such URL (its not on your IP address) or for
any other reason.
User would like to know about any changes for specified resource. She
sends request and asks register her for this resource and send
notification if something changes:
manageNotification(URL, {notification} [, granularity])
Notification specifies how user would like to be notified (HTTP,
SMTP, etc), in which format and granularity specifies for which event
(resource updated, dropped, etc.). Notification registration may
succeed or fail for reasons similar to resource registration.
Author modifies the resource and would like to notify everybody who
is registered about update:
manageUpdate(URL, message)
It will register update with specified message for particular
resource and notify everybody who is subscribed for this resource.
User can be notified in two modes: active and passive. Active mode
supposes you have something that is able to accept notification, like
http server (for HTTP or SOAP requests) or mail account (for SMTP
requests), and passive notification will be stored on server and you
can come later and read notifications youve got (source, message and
time). You can combine these approaches: subscribe in active mode for
something youre urgent to know about and in passive mode for
everything else.
--
Notification is simpler because server knows exactly what parameters
he wants to pass. For example, I'd like to manage Meerkats checks
against my web site (for some sites one hour can be too often and for
others it can be to rare) and to achive that, I need to register URL
of my site with notification server (as author), Meerkat needs to
register (as User of this resource) and then I just need to send
updates to notification server and it will take care about the rest
and will notify eveybody who is subscribed for this service. Almost
nothing has to be changed on Meerkat's side, and other users can
register for the same service, like search engines, catalogs or
anybody else (like you and me).
Asynchronous messages are little bit different (actually it's not
neccessarily should be SMTP against POP3, it could be HTTP only, but
process on server side could take significant amount of time): A
calls B, then B calls A. Two problems: A should let B know how to
call it and pass something that will let Alater identify that this is
response for particular request, like messageID.
This model can definitely be built on top of SOAP (I hope to have
demonstration soon), but it'll require design changes in toolkit on
client side. I'd like to stay with 'collection' interface, when:
A collectionserver
--request-->
add MID --request-->
accept request with MID and
collection's endpoints
start process and
make fake response
<-response--
<-response-- with MID
time passed.
client runs request for particular MID
--request-->
nothing for you yet
<-response--
finish process and
make real response
<-response-- with MID
--request-->
yes, we have something for your MID
<-respo
[OT] Re: mod_perl shared memory with MM
On Sun, Mar 11, 2001 at 03:33:12PM +0100, Christian Jaeger wrote:
> I've looked at Cache::FileCache now and think it's (currently) not
> possible to use for IPC::FsSharevars:
>
> I really miss locking capabilities. Imagine a script that reads a
> value at the beginning of a request and writes it back at the end of
> the request. If it's not locked during this time, another instance
> can read the same value and then write another change back which is
> then overwritten by the first instance.
I'm very intrigued by your thinking on locking. I had never
considered the transaction based approach to caching you are referring
to. I'll take this up privately with you, because we've strayed far
off the mod_perl topic, although I find it fascinating.
> - why don't you use 'real' constants for $SUCCESS and the like? (use
> constant)
Two reasons, mostly historical, and not necessarily good ones.
One, I benchmarked some code once that required high performance, and
the use of constants was just slightly slower.
Two, I like the syntax $hash{$CONSTANT}. If I remember correctly,
$hash{CONSTANT} didn't work. This may have changed in newer versions
of Perl.
Obviously those are *very* small issues, and so it is mostly by habit
that I don't use constant. I would consider changing, but it would
mean asking everyone using the code to change too, because they
currently import and use the constants as Exported scalars.
Do you know of a very important reason to break compatibility and
force the switch? I'm not opposed to switching if I have to, but I'd
like to minimize the impact on the users.
> - you probably should either append the userid of the process to
> /tmp/FileCache or make this folder globally writeable (and set the
> sticky flag). Otherwise other users get a permission error.
As of version 0.03, the cache directories, but not the cache entries,
are globally writable by default. Users can override this by changing
the 'directory_umask' option, or keep data private altogether by
changing the 'cache_root'. What version did you test with? There may
be a bug in there.
> - why don't you use Storable.pm? It should be much faster than Data::Dumper
The TODO contains "Replace Data::Dumper with Storable (maybe)". :)
The old File::Cache module used Storable, btw.
It will be trivial to port the new Cache::FileCache to use Storable.
I simply wanted to wait until I had the benchmarking code so I could
be sure that Storeable was faster. Actually, I'm not 100% sure that I
expect Storeable to be faster than Data::Dumper. If Data::Dumper
turns out to be about equally fast, then I'll stay with it, because it
is available on all Perl installations, I believe.
Do you know if Storeable is definitely faster? If you have benchmarks
then I am more than happy to switch now. Or, do you know of a reason,
feature wise, that I should switch? Again, it is trivial to do so.
> >I have some preliminary benchmark code -- only good for relative
> >benchmarking, but it is a start. I'd be happy to post the results
> >here if people are interested.
>
> Could you send me the code?, then I'll look into benchmarking my
> module too.
I checked it in as Cache::CacheBenchmark. It isn't good code, nor
does it necessarily work just yet. I simply checked it in while I was
in the middle of working on it. I'm turning it into a real
benchmarking class for the cache, and hopefully that will help you a
little bit.
Cheers,
-DeWitt
Re: mod_perl shared memory with MM
At 22:23 Uhr -0500 10.3.2001, DeWitt Clinton wrote: >On Sat, Mar 10, 2001 at 04:35:02PM -0800, Perrin Harkins wrote: > > Christian Jaeger wrote: >> > Yes, it uses a separate file for each variable. This way also locking >> > is solved, each variable has it's own file lock. >> >> You should take a look at DeWitt Clinton's Cache::FileCache module, >> announced on this list. It might make sense to merge your work into >> that module, which is the next generation of the popular File::Cache >> module. > >Yes! I'm actively looking for additional developers for the Perl >Cache project. I'd love new implementations of the Cache interface. >Cache::BerkeleyDBCache would be wonderful. Check out: > > http://sourceforge.net/projects/perl-cache/ > >For what it is worth, I don't explicitly lock. I do atomic writes >instead, and have yet to hear anyone report a problem in the year the >code has been public. I've looked at Cache::FileCache now and think it's (currently) not possible to use for IPC::FsSharevars: I really miss locking capabilities. Imagine a script that reads a value at the beginning of a request and writes it back at the end of the request. If it's not locked during this time, another instance can read the same value and then write another change back which is then overwritten by the first instance. IPC::FsSharevars even goes one step further: instead of locking everything for a particular session, it only locks individual variables. So you can say "I use the variables $foo and %bar from session 12345 and will write %bar back", in which case %bar of session 12345 is locked until it is written back, while $foo and @baz are still unlocked and may be read (and written) by other instances. :-) Such behaviour is useful if you have framesets where a browser may request several frames of the same session in parallel (you can see an example on http://testwww.ethz.ch, click on 'Suche' then on the submit button, the two appearing frames are executed in parallel and both access different session variables), or for handling session independant (global) data. One thing to be careful about in such situations is dead locking. IPC::FsSharevars prevents dead locks by getting all needed locks at the same time (this is done by first requesting a general session lock and then trying to lock all needed variable container files - if it fails, the session lock is freed again and the process waits for a unix signal indicating a change in the locking states). Getting all locks at the same time is more efficient than getting locks always in the same order. BTW some questions/suggestions for DeWitt: - why don't you use 'real' constants for $SUCCESS and the like? (use constant) - you probably should either append the userid of the process to /tmp/FileCache or make this folder globally writeable (and set the sticky flag). Otherwise other users get a permission error. - why don't you use Storable.pm? It should be much faster than Data::Dumper >I have some preliminary benchmark code -- only good for relative >benchmarking, but it is a start. I'd be happy to post the results >here if people are interested. Could you send me the code?, then I'll look into benchmarking my module too.
Re: signal delivered to a different process?
On Sat, 10 Mar 2001, Richard Chen wrote:
> On Sat, Mar 10, 2001 at 12:41:25PM +0800, Stas Bekman wrote:
> > On Fri, 9 Mar 2001, Richard Chen wrote:
> >
> > > This is pretty weird situation. I have installed a signal
> > > handler in startup.pl which showed that the signal is
> > > delivered to a different process!
> > >
> > > Here are the demo files:
> > >
> > > $ cat conf/startup.pl
> > > #!/usr/local/bin/perl
> > > use lib '/usr/local/apache/modules';
> > > $SIG{USR2}=sub {
> > > print STDERR "Received USR2 signal:\n";
> > > print STDERR "In SIG, pid=$$, ppid=".getppid()."\n";
> > > };
> > >
> > > # cat /usr/local/apache/modules/Apache/ChildInit.pm
> > > package Apache::ChildInit;
> > > sub handler {
> > > print STDERR "In ChildInit, pid=$$, ppid=".getppid()."\n";
> > > return;
> > > }
> > > 1;
> > >
> > > In httpd.conf I have these 2 lines added:
> > > PerlRequire conf/startup.pl
> > > PerlChildInitHandler Apache::ChildInit
> > >
> > > After launching the modperl process we have:
> > >
> > > $ ps -ef|grep http|grep root
> > > root 5964 1 0 16:17 ?00:00:00 /usr/local/apache/bin/httpd
> > >
> > > $ tail -1 error_log
> > > In ChildInit, pid=5965, ppid=5964
> > >
> > > So the modperl parent process has pid 5964.
> > > After sending USR2 signal to the modperl parent process,
> > > these lines appear in the error_log file:
> > > Received USR2 signal:
> > > In SIG, pid=5963, ppid=1
> > >
> > > So the signal is delivered to the process 5963 instead of the correct 5964.
> > >
> > > Has anyone seen this before?
> > > Is this a bug in modperl or if the procedure for installing the handler
> > > is incorrect?
> >
> > I think that it has nothing to with mod_perl, it's the OS that handles
> > pizza^H^H^H^H^Hsignal deliveries.
> >
>
> > BTW, what kind of process 5963 was? How come that if it wasn't the httpd
> > process you have installed the signal handler in, it knew to print the
> > stuff to the error_log which it has never opened? looks like you are not
> > getting the parent process pid correctly.
> >
> > You also didn't tell, how did you send the signal.
> >
>
> I think the problem is with the procedure of installing the handler.
> I did not know what kind of process 5963 was because it did not
> show up in the ps command even within the sig handler. But now
> I caught it by modifying the startup.pl routine to show the pid
> when the sig handler is defined:
>
> $ cat startup.pl
> #!/usr/local/bin/perl
> use lib '/usr/local/apache/modules';
> print STDERR "In startup, pid=$$\n";
> $SIG{USR2}=sub {
> print STDERR "Received USR2 signal\n";
> print STDERR "In SIG, pid=$$, ppid=".getppid()."\n";
> };
>
> Now here are the steps to show the problem
>
> Start up the modperl server:
>
> # ../bin/httpd -d /usr/local/apache
> In startup, pid=11525
>
> This means that at the time when the sig handler is defined, pid=11525.
> However, the ps command shows a different pid for modperl:
>
> # ps -ef|grep http|grep root
> root 11526 1 1 08:28 ?00:00:00 ../bin/httpd -d /usr/local/apach
>
> # tail -1 ../logs/error_log
> In ChildInit, pid=11527, ppid=11526
>
> The above clearly shows that the httpd server has pid 11526,
> not 11525. This means that process 11525 started by the startup
> procedure immediately exits after forking the real httpd process 11526.
>
> Now we send signal to the httpd server:
>
> # kill -USR2 11526
>
> And here are the lines recorded in error_log:
>
> # tail -2 ../logs/error_log
> Received USR2 signal
> In SIG, pid=11525, ppid=1
>
> So clearly the signal handler still remembers that at
> the time when it was defined, the pid was 11525, not 11526
> for httpd.
>
> Apparently modperl did some preparatory work before forking out
> the real httpd. This is something new to me.
Funny, but it's new to me too :) I see the same thing. I thought it has
something to do with immediate restart, but no, the extra fork happens
after the immediate restart. The real pid is located in logs/httpd.pid
or similar.
> I wonder if there is another way to install the sig handler so
> that it is defined inside the final httpd root process.
I'm not sure you can do anything at all in the parent. Once it spawns the
first child, the only thing it does is handling accept() calls and
maintaining the scoreboard so it will know what to do about its kids.
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://apachetoday.com http://logilune.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
