Re: [PHP-DEV] How deep is copy on write?
Yep. PHP does clock up memory very quickly for big arrays, objects with lots of
members and/or lots of small objects with large overheads. There are a LOT of
zvals and zobjects and things around the place, and their overhead isn't all that
small.
Of course, if you go to the trouble to construct arrays using references, you can
avoid some of that, because a copy-on-write will just copy the reference. It does
mean you're passing references, though.
$bar['baz'] = 1;
$poink['narf'] = 1;
$a['foo']['bar'] =& $bar;
$a['foo']['poink'] =& $poink;
Then if you test($a), $bar and $poink will be changed, since they are 'passed by
reference'--no copying needs to be done. It's almost as if $b were passed by
reference, but setting $b['blip'] wouldn't show up in $a, because $a itself would
be copied in that case, including the references, which would continue to refer to
$bar and $poink. So a much quicker copy, but obviously not the same level of
isolation that you might expect or desire. Unless you did some jiggerypokery like
$b_bar=$b['bar']; $b['bar']=$b_bar; which would break the reference and make a
copy of just that part of the array. But this is a pretty nasty caller-callee
co-operative kind of thing. Just a thought to throw into the mix, though.
Disclaimer: I'm somewhat out of my depth here. But I'm sure someone will jump on
me if I'm wrong.
Ben.
On 19/01/11 6:09 PM, Larry Garfield wrote:
That's what I was afraid of. So it does copy the entire array. Crap. :-)
Am I correct that each level in the array represents its own ZVal, with the
additional memory overhead a ZVal has (however many bytes that is)?
That is, the array below would have $a, foo, bar, baz, bob, narf, poink,
poink/narf = 8 ZVals? (That seems logical to me because each its its own
variable that just happens to be an array, but I want to be sure.)
--Larry Garfield
On Wednesday, January 19, 2011 1:01:44 am Ben Schmidt wrote:
It does the whole of $b. It has to, because when you change 'baz', a
reference in 'bar' needs to change to point to the newly copied 'baz', so
'bar' is written...and likewise 'foo' is written.
Ben.
On 19/01/11 5:45 PM, Larry Garfield wrote:
Hi folks. I have a question about the PHP runtime that I hope is
appropriate for this list. (If not, please thwap me gently; I bruise
easily.)
I know PHP does copy-on-write. However, how "deeply" does it copy when
dealing with nested arrays?
This is probably easiest to explain with an example...
$a['foo']['bar']['baz'] = 1;
$a['foo']['bar']['bob'] = 1;
$a['foo']['bar']['narf'] = 1;
$a['foo']['poink']['narf'] = 1;
function test($b) {
// Assume each of the following lines in isolation...
// Does this copy just the one variable baz, or the full array?
$b['foo']['bar']['baz'] = 2;
// Does this copy $b, or just $b['foo']['poink']?
$b['foo']['poink']['stuff'] = 3;
return $b;
}
// I know this is wasteful; I'm trying to figure out just how wasteful.
$a = test($a);
test() in this case should take $b by reference, but I'm trying to
determine how much of a difference it is. (In practice my use case has
a vastly larger array, so any inefficiencies are multiplied.)
--Larry Garfield
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] How deep is copy on write?
That's what I was afraid of. So it does copy the entire array. Crap. :-)
Am I correct that each level in the array represents its own ZVal, with the
additional memory overhead a ZVal has (however many bytes that is)?
That is, the array below would have $a, foo, bar, baz, bob, narf, poink,
poink/narf = 8 ZVals? (That seems logical to me because each its its own
variable that just happens to be an array, but I want to be sure.)
--Larry Garfield
On Wednesday, January 19, 2011 1:01:44 am Ben Schmidt wrote:
> It does the whole of $b. It has to, because when you change 'baz', a
> reference in 'bar' needs to change to point to the newly copied 'baz', so
> 'bar' is written...and likewise 'foo' is written.
>
> Ben.
>
> On 19/01/11 5:45 PM, Larry Garfield wrote:
> > Hi folks. I have a question about the PHP runtime that I hope is
> > appropriate for this list. (If not, please thwap me gently; I bruise
> > easily.)
> >
> > I know PHP does copy-on-write. However, how "deeply" does it copy when
> > dealing with nested arrays?
> >
> > This is probably easiest to explain with an example...
> >
> > $a['foo']['bar']['baz'] = 1;
> > $a['foo']['bar']['bob'] = 1;
> > $a['foo']['bar']['narf'] = 1;
> > $a['foo']['poink']['narf'] = 1;
> >
> > function test($b) {
> >
> >// Assume each of the following lines in isolation...
> >
> >// Does this copy just the one variable baz, or the full array?
> >$b['foo']['bar']['baz'] = 2;
> >
> >// Does this copy $b, or just $b['foo']['poink']?
> >$b['foo']['poink']['stuff'] = 3;
> >
> >return $b;
> >
> > }
> >
> > // I know this is wasteful; I'm trying to figure out just how wasteful.
> > $a = test($a);
> >
> > test() in this case should take $b by reference, but I'm trying to
> > determine how much of a difference it is. (In practice my use case has
> > a vastly larger array, so any inefficiencies are multiplied.)
> >
> > --Larry Garfield
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] How deep is copy on write?
It does the whole of $b. It has to, because when you change 'baz', a reference in
'bar' needs to change to point to the newly copied 'baz', so 'bar' is
written...and likewise 'foo' is written.
Ben.
On 19/01/11 5:45 PM, Larry Garfield wrote:
Hi folks. I have a question about the PHP runtime that I hope is appropriate
for this list. (If not, please thwap me gently; I bruise easily.)
I know PHP does copy-on-write. However, how "deeply" does it copy when
dealing with nested arrays?
This is probably easiest to explain with an example...
$a['foo']['bar']['baz'] = 1;
$a['foo']['bar']['bob'] = 1;
$a['foo']['bar']['narf'] = 1;
$a['foo']['poink']['narf'] = 1;
function test($b) {
// Assume each of the following lines in isolation...
// Does this copy just the one variable baz, or the full array?
$b['foo']['bar']['baz'] = 2;
// Does this copy $b, or just $b['foo']['poink']?
$b['foo']['poink']['stuff'] = 3;
return $b;
}
// I know this is wasteful; I'm trying to figure out just how wasteful.
$a = test($a);
test() in this case should take $b by reference, but I'm trying to determine
how much of a difference it is. (In practice my use case has a vastly larger
array, so any inefficiencies are multiplied.)
--Larry Garfield
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] How deep is copy on write?
Hi folks. I have a question about the PHP runtime that I hope is appropriate
for this list. (If not, please thwap me gently; I bruise easily.)
I know PHP does copy-on-write. However, how "deeply" does it copy when
dealing with nested arrays?
This is probably easiest to explain with an example...
$a['foo']['bar']['baz'] = 1;
$a['foo']['bar']['bob'] = 1;
$a['foo']['bar']['narf'] = 1;
$a['foo']['poink']['narf'] = 1;
function test($b) {
// Assume each of the following lines in isolation...
// Does this copy just the one variable baz, or the full array?
$b['foo']['bar']['baz'] = 2;
// Does this copy $b, or just $b['foo']['poink']?
$b['foo']['poink']['stuff'] = 3;
return $b;
}
// I know this is wasteful; I'm trying to figure out just how wasteful.
$a = test($a);
test() in this case should take $b by reference, but I'm trying to determine
how much of a difference it is. (In practice my use case has a vastly larger
array, so any inefficiencies are multiplied.)
--Larry Garfield
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
On 19/01/11 3:51 PM, Stas Malyshev wrote: Hi! If anything, I would argue that PHP is a language unsuited to beginners (and other scripting languages), as it is so flexible it doesn't enforce good programming practice. Java is much more a 'beginner language' because it has much stricter Contrary to popular belief, people usually don't start with programming to be taught good practices and become enlightened in the ways of Art. They usually start because they need their computers to do something for them. And scripting languages are often the easiest way to make that happen. Java, on the other hand, forces you to deal with exceptions, patterns, interfaces, generics, covariants and contravariants, locking, etc. which you neither want nor need to know, only because somebody somewhere decided that it's right for you. Yeah, well, I was playing Devil's advocate and went a bit far (as you have too--arguing is fun, isn't it?). On a more serious note, I think what is much more helpful for beginners is a good teacher, or good materials to learn from. Almost any language can be a good one for beginners if taught well. Including PHP; indeed I have recommended and taught people PHP as beginners without much trouble. And Java. At any rate, the important thing is that beginners shouldn't hold a good language back, particularly if the innovations are not obligatory for them to use. Smiles, Ben. -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
Hi! like "map" express this well. The idea is that the executor can start up with no variables in scope, though hopefully shared code segments, For that you would probably need to put some severe restrictions on your code, such as: 1. No usage of default properties or statics in classes or functions. 2. No assigning of constants to any variable (comparison and operators may be ok, not sure how refcounts work out) 3. No defining new functions or classes or including new files This probably could still do something useful - such as run 3 sql queries in parallel and return the result - but I'm not sure how you could enforce such conditions... If you do not, you'll have some "interesting" race conditions leading to variables disappearing, leaking, being assigned wrong values, etc. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
Hi! If anything, I would argue that PHP is a language unsuited to beginners (and other scripting languages), as it is so flexible it doesn't enforce good programming practice. Java is much more a 'beginner language' because it has much stricter Contrary to popular belief, people usually don't start with programming to be taught good practices and become enlightened in the ways of Art. They usually start because they need their computers to do something for them. And scripting languages are often the easiest way to make that happen. Java, on the other hand, forces you to deal with exceptions, patterns, interfaces, generics, covariants and contravariants, locking, etc. which you neither want nor need to know, only because somebody somewhere decided that it's right for you. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
Hi! Yes, I expected the two functions - tsrm_new_interpreter() and init_executor() to do that, as it is the function called in php_request_startup() in main/main.c As far as I remember, you need to run the whole request startup for the the thread, otherwise there will be unitilialized pieces. TSRM magic will create needed per-thread structures and call ctors, but ctors usually just null out stuff, you'd still need to fill it in. Another possible application would be a parallel_include() type call, which would call a given PHP file for each member of an array (or a PDO result set), buffering the output from each, and inserting into the output stream in sequence once each fragment is done (hopefully interacting well with normal output buffering, if you didn't want the results sent yet). This would allow a large number of results to be rendered in parallel on multicore systems. That's what webservers do already, don't they? :) I hope it will be possible to share already compiled code between threads; this may mean disabling "eval" inside the thread or otherwise The main problems you will be facing are the following: 1. All ZE structures are per-thread. This means using one thread's structures in another will be non-trivial task, as all code assumes that current thread's structures are used. 2. Even if you manage to hack around it by always passing the tsrm_ls pointers, etc. - memory managers are per-thread too. Which means you will be using data in one thread that is controlled by MM residing in another thread. Without locking. 3. You may think this is not very bad, since you'll be using stuff that's quite static, like classes and functions - they don't get deallocated inside request, so who cares which MM uses them? However, while classes themselves don't, structures containing them - hashtables - can change, be rebuilt, etc. and if it happens in a wrong moment, you're in trouble. 4. Next problem with using classes/functions is that they can contain variables - zvals, as default properties, static variables, etc. Since ZF is refcounting, these zvals may be modified by anybody who uses these variables - even just for reading. Again, no locking. Which, again, means trouble. 5. Then come resources and module globals. Imagine some function touches in some way some resource - connection, file, etc. - that another thread is using at the same time, without locking? Modules generally assume resources belong to their respective threads, so you'll need to run module initializations for each thread separately. hobbling the compiler to avoid separate threads trying to modify the optree at once. If a shared optree cannot be achieved, then I guess it would have to go back to the APC, but it would be good to avoid overheads where possible to keep the thread startup cost low. Because of the things described above, it will be very challenging to avoid those startup costs. Even extremely restricted parallelism can help speed up some types of work, so limitations I am happy to accept. If you restrict it to using only copied data and never running any PHP code, it might work. Alternatively, you might launch independent engine instances that don't share structures and have them communicate, like Erlang does. Though, unlike Erlang, PHP engine would not help you much in this, I'm afraid. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
On 19/01/11 10:50, Stefan Marr wrote: > On 18 Jan 2011, at 22:16, Sam Vilain wrote: >> there doesn't seem to >> be an interpreter under the sun which has successfully pulled off >> threading with shared data. > Could you explain what you mean with that statement? > > Sorry, but that's my topic, and the most well know interpreters that 'pulled > off' threading with shared data are for Java. The interpreter I am working on > is for manycore systems (running on a 64-core Tilera chip) and executes > Smalltalk (https://github.com/smarr/RoarVM). You raise a very good point. My statement is too broad and should probably apply only to dynamic languages, executed on reference counted VMs. Look at some major ones - PHP, Python, Ruby, Perl, most JS engines - none of them actually thread properly. Well, Perl's "threading" does run full speed, but actually copies every variable on the heap for each new thread, massively bloating the process. So the question is why should this be so, if C++ and Java, even interpreted on a JVM, can do it? In general, Java's basic types typically correspond with types that can be dealt with atomically by processors, or are small enough to be passed by value. This already makes things a lot easier. I've had another reason for the differences explained to me. I'm not sure I understand it fully enough to be able to re-explain it, but I'll try anyway. As I grasped the concept, the key to making VMs fully threadable with shared state, is to first allow reference addresses to change, such as via generational garbage collection. This allows you to have much clearer "stack frames", perhaps even really stored on the thread-local/C stack, as opposed to most dynamic language interpreters which barely use the C stack at all. Then, when the long-lived objects are discovered at scope exit time they can be safely moved into the next memory pool, as well as letting access to "old" objects be locked (or copied, in the case of Software Transactional Memory). Access to objects in your own frame can therefore be fast, and the number of locks that have to be held reduced. Perhaps to support/refute this argument, in your JVM, how do you handle: - memory allocation: object references' timeline and garbage collection - call stack frames and/or return continuations - the C stack or the heap? - atomicity of functions (that's the "synchronized" keyword?) - timely object destruction I put it forward that the overall design of the interpreter, and therefore what is possible in terms of threading, is highly influenced by these factors. When threading in C or C++ for instance (and this includes HipHop-TBB), the call stack frame is on the C stack, so shared state is possible so long as you pass heap pointers around and synchronise appropriately. The "virtual" machine is of a different nature, and it can work. For JVMs, as far as I know references are temporary and again the nature of the execution environment is different. For VMs where there is basically nothing on the stack, and everything on the heap, it becomes a lot harder. To talk about a VM I know better, Perl has about 6 internal stacks all represented on the heap; a function call/return stack, a lexical scope stack to represent what is in scope, a variable stack (the "tmps" stack) for variables declared in those scopes and for timely destruction, a stack to implement local($var) called the "save" stack, a "mark" stack used for garbage collection, ok well only 5 but I think you get my point. From my reading of the PHP internals so far there are similar set there too, so comparisons are quite likely to be instructive. It's a bit hard figuring out everything that is going on internally (all these internal void* types don't help either), and whether or not there is some inherent property of reference counting, or whether it just makes a shared state model harder, is a question I'm not sure is easy to answer. In any case, full shared state is not required for a large set of useful parallelism APIs, and in fact contains a number of pitfalls which are difficult to explain, debug and fix. I'm far more interested in simple acceleration of tight loops - to make use of otherwise idle CPU cores (perhaps virtual as in hyperthreading) to increase throughput - and APIs like "map" express this well. The idea is that the executor can start up with no variables in scope, though hopefully shared code segments, call some function on the data it is passed in, and pass the answers back to the main thread and then set about cleaning itself up. Sam -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
Strongly second this. PHP is not a toy language restricted to beginners. If it has advanced features, beginners simply don't need to use them. If anything, I would argue that PHP is a language unsuited to beginners (and other scripting languages), as it is so flexible it doesn't enforce good programming practice. Java is much more a 'beginner language' because it has much stricter syntax, type checking, exception handling, etc., which force and even teach people to program well in some regards (or at least do something to raise their awareness that they're programming sloppily!). Mind you, it's pretty easy to write bad code in any language Ben. On 19/01/11 9:36 AM, Hannes Landeholm wrote: Hello, I don't think a language becomes a "beginners language" just because many new programmers use it. And it's still not a good argument for not including new features. As long as the new thread doesn't share any memory/variables with the spawning context, no "reasoning" is required at all. It's when you start sharing objects that things get complex. Just a simple threading implementation with a strictly defined way to IPC would be very helpful. It's not super useful in web application programming as handling web requests is already packaged into small units of work.. web requests. So in that sense a web application is already "multi threaded". However it's interesting for CGI scripts. The other week I wrote a PHP CGI proxy for example. Because PHP didn't have threading, I had to bother with select polling. Hannes On 18 January 2011 23:10, Stas Malyshev wrote: Hi! Sorry, but that's my topic, and the most well know interpreters that 'pulled off' threading with shared data are for Java. The interpreter Given to what complications Java programmers should go to make their threaded code work, I have a lot of doubt that 95% of PHP users would be able to write correct threaded programs. Reasoning about threaded programs is very hard, and IMHO putting it into the beginners language would be a mistake. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
Hello, I don't think a language becomes a "beginners language" just because many new programmers use it. And it's still not a good argument for not including new features. As long as the new thread doesn't share any memory/variables with the spawning context, no "reasoning" is required at all. It's when you start sharing objects that things get complex. Just a simple threading implementation with a strictly defined way to IPC would be very helpful. It's not super useful in web application programming as handling web requests is already packaged into small units of work.. web requests. So in that sense a web application is already "multi threaded". However it's interesting for CGI scripts. The other week I wrote a PHP CGI proxy for example. Because PHP didn't have threading, I had to bother with select polling. Hannes On 18 January 2011 23:10, Stas Malyshev wrote: > Hi! > > > Sorry, but that's my topic, and the most well know interpreters that >> 'pulled off' threading with shared data are for Java. The interpreter >> > > Given to what complications Java programmers should go to make their > threaded code work, I have a lot of doubt that 95% of PHP users would be > able to write correct threaded programs. Reasoning about threaded programs > is very hard, and IMHO putting it into the beginners language would be a > mistake. > > -- > Stanislav Malyshev, Software Architect > SugarCRM: http://www.sugarcrm.com/ > (408)454-6900 ext. 227 > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > >
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
Hi! Sorry, but that's my topic, and the most well know interpreters that 'pulled off' threading with shared data are for Java. The interpreter Given to what complications Java programmers should go to make their threaded code work, I have a lot of doubt that 95% of PHP users would be able to write correct threaded programs. Reasoning about threaded programs is very hard, and IMHO putting it into the beginners language would be a mistake. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
Hi Sam: I am following the discussion very interested, but just a question for clarification: On 18 Jan 2011, at 22:16, Sam Vilain wrote: > there doesn't seem to > be an interpreter under the sun which has successfully pulled off > threading with shared data. Could you explain what you mean with that statement? Sorry, but that's my topic, and the most well know interpreters that 'pulled off' threading with shared data are for Java. The interpreter I am working on is for manycore systems (running on a 64-core Tilera chip) and executes Smalltalk (https://github.com/smarr/RoarVM). Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] Persistent, file backed, mmap APC hack
Hi, I use a multithreaded MPM of apache called "peruser" where we're hosting a bunch of sites, each with their own virtual host. Each virtual host needs to be isolated so I'm using chroot and other stuff to keep them apart. APC was a must as getting good throughput was important. However, APC would make all sites share memory containing cached PHP files and user data. A major security breach. So I was looking at the source code of APC and saw that it was using mmap() to map shared memory. If mmap used a filed backed shared memory instead you could a) get a persistent cache so the child processes could be terminated without dropping it and b) inherit the file system permissions c) make chroot define where the cache would be, essentially giving every virtual host their own APC cache. You could also use shm_get(), however IPC was not enabled on my kernel. I then proceeded to write a hack/proof of concept to APC 1.3.7 which accomplished this. This is a breakdown of the differences between vanilla APC and my hack: - Vanilla APC will allocate all shared memory when Apache starts (before chroot etc) + Hacked APC will delay shared memory initialization upon the first request of the child thread (after chroot etc) + Hacked APC will reserve a memory region when Apache starts so there will be no difference in address location. + Hacked APC only supports exactly one memory segment. + Hacked APC must be built with --enable-mmap + Hacked APC is recommended to be built with --enable-apc-pthreadrwlocks + Hacked APC adds a line "Hacks: Hacked for shared MMAP support" to phpinfo() to indicate that it has been loaded. - Vanilla APC just uses mmap() as a way to allocate shared memory + Hacked APC utilizes the file backed up feature of mmap, you must set apc_mmap_file_mask to a place to store the APC cache/swap, and this file will have the exact size of apc.shm_size. - Vanilla APC allocates shared memory once, it never loads previously memory. + Hacked APC will always load existing memory from the cache. Every time you start Apache the start time is recorded. If the start time differs, the cache will be zeroed, otherwise the cache will be loaded and the already initialized cache structures in it be used. Note: This could lead to a race-like condition if two PHP instances use the same cache and where started at different times. - Vanilla APC uses apc_mmap_file_mask as a temporary file and requires XX as a placeholder for unique hash. + Hacked APC uses apc_mmap_file_mask as an exact file name and requires it to be specified. - Vanilla APC has no Environment separation. The cache is initialized once and shared between all apache childs. + Hacked APC will load the cache from file specified by apc_mmap_file_mask. In addition, it will create it if it doesn't exist with the permissions 0600 and with the uid/gid of the the child process. The path is also affected by chroot. This is a double layer of security. I'm using this to get different APC caches per virtual host and limit memory. However it should also, theoretically, be possible to use it to enable a shared cache for PHP instances running with fast cgi. This could save a lot of memory. However you would then have to make some changes. For example, the memory is currently pre-mapped when apache is started so all children will share the same addresses. One way to solve this would be to implement the cache so no direct memory addresses are used, only offsets. Also, the hack currently stores the PHP start time and reinitializes the cache file when apache is restarted, and if you're removing the reinitialization you would then have a persistent data storage. A really crazy idea would then be to use APC as a persistent database instead of mySQL. Just gotta make sure APC is ACID compliant first! ;) Some notes on performance: I load tested this with a wordpress site and I did not detect any reduction in performance. I got 2x the throughput with both vanilla and hacked APC. This could however be slower in some cases. For example, this uses disk access to read and store the cache. However it maps the file in shared mode, so no disk reading should actually take place if it's already open. And simply reading/writing the memory does not guarantee that the changes are written to disk until the memory is actually unmapped. I should also not have to mention that this is an experimental hack. If you use this for anything important you're insane. The patch: http://pastebin.com/4GS83nKs Regards, Hannes Landeholm -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
On 18/01/11 22:17, Stas Malyshev wrote: >> 1) any hints or clues from people familiar with the Zend subsystems - >> such as memory management, and the various stacks, to provide hints as >> to how to set them up "correctly" > > Zend Engine keeps all state (including memory manager state, etc.) > separate in each thread, which means once you've created a new thread > it has to run initializations for the data structures. It should > happen automatically when you build the engine in threaded mode > (--enable-maintainer-zts). Yes, I expected the two functions - tsrm_new_interpreter() and init_executor() to do that, as it is the function called in php_request_startup() in main/main.c It seems to do a lot of the work, and as far as I could tell there is no TSRM function to reap an individual thread etc. There is also zend_startup() - which seems to do a bit more. If anyone knowledgeable would care to give or point to an overview, that would be very useful. > You can not share any data between the engine threads - unless you > communicate it through some channel external to the engine - and even > in this case you should use a copy, never the original pointer. Sure, I'm expecting to have to pass in all data as deep copies as well as the return value from the function. This is useful for array_map-like functions. The parallel_for API, while it worked in the context of HipHop, is unlikely to work with Zend; there doesn't seem to be an interpreter under the sun which has successfully pulled off threading with shared data. Another possible application would be a parallel_include() type call, which would call a given PHP file for each member of an array (or a PDO result set), buffering the output from each, and inserting into the output stream in sequence once each fragment is done (hopefully interacting well with normal output buffering, if you didn't want the results sent yet). This would allow a large number of results to be rendered in parallel on multicore systems. > This also means you can not use PHP functions, classes, etc. from one > thread in another one. I hope it will be possible to share already compiled code between threads; this may mean disabling "eval" inside the thread or otherwise hobbling the compiler to avoid separate threads trying to modify the optree at once. If a shared optree cannot be achieved, then I guess it would have to go back to the APC, but it would be good to avoid overheads where possible to keep the thread startup cost low. Even extremely restricted parallelism can help speed up some types of work, so limitations I am happy to accept. > I'm not sure what you tried to do in your code, so hard to say what > exactly went wrong there. > Another caveat: while Zend Engine makes a lot of effort to keep the > state localized and thus be thread-safe, not all libraries PHP is > using do so, so running multithreaded PHP with these libraries may > cause various trouble. Yes, currently I am not looking at calling individual module startup functions to avoid this problem (and save time on thread startup). It seems that there is a facility for limiting the available functions visible to the created executor, too, which may make this easy to make "safe". Thanks for your feedback, Sam -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] Update to the windows.php.net downloads page
even better would be to report a bug at bugs.php.net and attach the patch to it :) Thanks! On Tue, Jan 18, 2011 at 6:36 PM, Pierre Joye wrote: > hi, > > Yes, I asked you to send me a patch, easier to review and apply. > > Cheers, > > On Tue, Jan 18, 2011 at 5:09 PM, Sascha Meyer wrote: >> Good afternoon, >> >> I added a link to the qa/releases/snapshots archives on the PHP for Windows >> downloads page, because older releases of the Windows PHP builds were not >> linked on the downloads site. I would also like to improve the functionality >> of windows.php.net and do some code restructuring, that's why I'd like to >> ask for a SVN account for future changes to code and design. >> >> The modified listing.php is attached to this mail (svn path: >> http://svn.php.net/repository/web/php-windows/trunk/docroot/listing.php) >> >> Regards, >> >> Sascha >> -- >> Freundliche Grüße / Kind regards, >> >> Sascha Meyer >> -- >> EE: http://www.experts-exchange.com/M_761556.html >> ZCE: http://www.zend.com/en/yellow-pages#show-ClientCandidateID=ZEND011290 >> >> Neu: GMX De-Mail - Einfach wie E-Mail, sicher wie ein Brief! >> Jetzt De-Mail-Adresse reservieren: http://portal.gmx.net/de/go/demail >> >> -- >> PHP Internals - PHP Runtime Development Mailing List >> To unsubscribe, visit: http://www.php.net/unsub.php >> > > > > -- > Pierre > > @pierrejoye | http://blog.thepimp.net | http://www.libgd.org > -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] Update to the windows.php.net downloads page
hi, Yes, I asked you to send me a patch, easier to review and apply. Cheers, On Tue, Jan 18, 2011 at 5:09 PM, Sascha Meyer wrote: > Good afternoon, > > I added a link to the qa/releases/snapshots archives on the PHP for Windows > downloads page, because older releases of the Windows PHP builds were not > linked on the downloads site. I would also like to improve the functionality > of windows.php.net and do some code restructuring, that's why I'd like to ask > for a SVN account for future changes to code and design. > > The modified listing.php is attached to this mail (svn path: > http://svn.php.net/repository/web/php-windows/trunk/docroot/listing.php) > > Regards, > > Sascha > -- > Freundliche Grüße / Kind regards, > > Sascha Meyer > -- > EE: http://www.experts-exchange.com/M_761556.html > ZCE: http://www.zend.com/en/yellow-pages#show-ClientCandidateID=ZEND011290 > > Neu: GMX De-Mail - Einfach wie E-Mail, sicher wie ein Brief! > Jetzt De-Mail-Adresse reservieren: http://portal.gmx.net/de/go/demail > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] Update to the windows.php.net downloads page
On Tue, 2011-01-18 at 17:09 +0100, Sascha Meyer wrote: > Good afternoon, > > I added a link to the qa/releases/snapshots archives on the PHP for Windows > downloads page [...] I would love if we could move all that back to php.net, downloads should use our mirrors for serving users best and giving them one location to look at and no confusion about not finding some stuff. CC'ing the webmaster list. > The modified listing.php is attached to this mail svn diff would be preferred to see the actual change (especially if somebody changes the page in between..) johannes -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] [PATCH] Update to the windows.php.net downloads page
Good afternoon, I added a link to the qa/releases/snapshots archives on the PHP for Windows downloads page, because older releases of the Windows PHP builds were not linked on the downloads site. I would also like to improve the functionality of windows.php.net and do some code restructuring, that's why I'd like to ask for a SVN account for future changes to code and design. The modified listing.php is attached to this mail (svn path: http://svn.php.net/repository/web/php-windows/trunk/docroot/listing.php) Regards, Sascha -- Freundliche Grüße / Kind regards, Sascha Meyer -- EE: http://www.experts-exchange.com/M_761556.html ZCE: http://www.zend.com/en/yellow-pages#show-ClientCandidateID=ZEND011290 Neu: GMX De-Mail - Einfach wie E-Mail, sicher wie ein Brief! Jetzt De-Mail-Adresse reservieren: http://portal.gmx.net/de/go/demail <> -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] SVN Account Request: sascham78
Providing updates to the windows.php.net websites: code cleanup, template updates, adding links to the download archives, maintenance work -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] fully enabling dl() for FPM SAPI
On Fri, 14 Jan 2011, Antony Dovgal wrote: > Are there any objections if I disable E_DEPRECATED notice in dl() for FPM > SAPI? > The notice is already disabled for CGI/FastCGI, CLI and Embed SAPIs. > I believe there's no reason for this notice in case of FPM, too. IMO, it should be disabled in fastcgi too. Ages ago we came to the conclusion it was dangerous/annoying/broken to have dl() work with processes that run multiple-requests. cheers, Derick -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] fully enabling dl() for FPM SAPI
On Tue, 2011-01-18 at 11:00 +0300, Antony Dovgal wrote: > On 01/17/2011 09:19 PM, Johannes Schlüter wrote: > > I think it can be quite dangerous if you have extensions living shorter > > than the PHP process. Not only might dlclose() cause some leaks but > > there are a few extensions playing with function pointers or opcode > > handlers which aren't properly reset so a following request might try to > > jump to invalid memory. > > dlclose()? I can assure you I'm not going to call dlclose() on each request > shutdown. > > Yes, that means once an extension is loaded it'll stay till the death of this > particular child process). > But it does work here for the last 5 or 6 years this way and this is indeed > what I want. > > > TBH I'm not even sure dlclose() is called at all since I wasn't able to track > this call down through all the handlers, destructors and so on in 5 min I > spent on this.. module_destructor() calls it and expected it to be hit on rshutdown, else has really weird behavior - for some processes an extension would be loaded, for others not which results in non-deterministic behavior for the outside. http://lxr.php.net/opengrok/xref/PHP_5_3/Zend/zend_API.c#2120 johannes -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Experiments with a threading library for Zend: spawning a new executor
Hi! 1) any hints or clues from people familiar with the Zend subsystems - such as memory management, and the various stacks, to provide hints as to how to set them up "correctly" Zend Engine keeps all state (including memory manager state, etc.) separate in each thread, which means once you've created a new thread it has to run initializations for the data structures. It should happen automatically when you build the engine in threaded mode (--enable-maintainer-zts). You can not share any data between the engine threads - unless you communicate it through some channel external to the engine - and even in this case you should use a copy, never the original pointer. This also means you can not use PHP functions, classes, etc. from one thread in another one. I'm not sure what you tried to do in your code, so hard to say what exactly went wrong there. Another caveat: while Zend Engine makes a lot of effort to keep the state localized and thus be thread-safe, not all libraries PHP is using do so, so running multithreaded PHP with these libraries may cause various trouble. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] [url fixes for] Experiments with a threading library for Zend: spawning a new executor
On 18/01/11 17:21, Sam Vilain wrote: > (full code is available at http://github.com/openparallel/php-src.git ) *ahem* that should be http://github.com/openparallel/php-src In fact to skip straight to the function, try http://github.com/openparallel/php-src/blob/9205db3/ext/tbb/tbb.c#L208 > http://openparallel.wordpress.com/2010/11/01/tbb-in-wordpress-–-white-paper/ While I'm here, I may as well provide a version of that link which doesn't get mangled on the way through to the http gateway... http://xrl.us/hiphoptbbwordpress Sam -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] fully enabling dl() for FPM SAPI
On 01/17/2011 09:19 PM, Johannes Schlüter wrote: > I think it can be quite dangerous if you have extensions living shorter > than the PHP process. Not only might dlclose() cause some leaks but > there are a few extensions playing with function pointers or opcode > handlers which aren't properly reset so a following request might try to > jump to invalid memory. dlclose()? I can assure you I'm not going to call dlclose() on each request shutdown. Yes, that means once an extension is loaded it'll stay till the death of this particular child process). But it does work here for the last 5 or 6 years this way and this is indeed what I want. TBH I'm not even sure dlclose() is called at all since I wasn't able to track this call down through all the handlers, destructors and so on in 5 min I spent on this.. > Additionally there's no restriction on this once safe_mode is gone, so > anybody could load any C extension - while that can be fixed by > advertising disable_function=dl That's right, disabling it is not a problem. -- Wbr, Antony Dovgal --- http://pinba.org - realtime statistics for PHP -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
