Re: How can I increase the shell's (or specific application's) memory limit?
Hi Daniel, Generally, I think it's bad programming practice to retrieve such big datasets if it is possible to do otherwise. I definitely agree that it is bad practice, and in that respect I'm inclined towards doing batch loading as you suggest too. However, there's some data agregation I'll have to take into account, and as it involves testing for the presence of specific tables of a merge table set, I'd have to rewrite part of that logic. All doable, of course, and no big issue either, but it would be a lot faster for me if I could simply increase the memory limit Still, I very much hear you, and I know that what you suggest _is_ the proper approach, so I may end up doing that too. ;) Also: there is another perhaps more elegant (read: robust) way, being a hybrid solution between the PHP script and using mysqldump. I can then use PHP for working out the batches, and retrieve the batches using a (set of) commandline mysqldump call(s). The generated batches can then directly be dumped into the proper merge tables. The only catch is that I directly left join data in into the merge tables, so I'd have to first do a blunt dump of the lhs of the data, then of the rhs(es) (both to temp tables) and then afterwards left join them into the eventual merge tables. This is the main reason why I hadn't chosen this solution, as at present I can combine all of these steps in one query...:/ If someone knows a clean way to increase the memory limit, I'd be happy to hear about it. If not, I'll do some rewrite... Cheers, Olafo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: How can I increase the shell's (or specific application's) memory limit?
On 4/3/06, Olaf Greve <[EMAIL PROTECTED]> wrote: > Hi, > > I've got a question which is probably pretty easy to answer: how can I > assign more memory to a PHP script running in a shell and/or in a browser. > > Some more background info: > I'm building a PHP script that has to retrieve pretty large sets of data > from a remote MySQL database, then process it, and store the results to > a local database. > > The issue: > The script (surprise, surprise) quickly runs out of memory. Now, I have > already tried to increase the memory limit in php.ini (followed by an > Apache restart, of course), but even when setting the limit to something > high like 384MB or so, the script still bails out with a memory limit > error when retrieving as little as some 50MB of data... > > Now, of course I could rewrite my PHP script such that it will retrieve > smaller batches of data, but being a programmer I'm lazy, and I'd rather > simply assign more memory to the script (actually, it's not only due to > laziness, but also due to the fact that the script has to agregate data > etc., and I'd rather have it do that in 1 run for a variety of reasons). > > It seems to me like setting the memory limit in php.ini above a value of > 64MB (or so) doesn't seem to have any effect anymore. My assumption then > is that the memory limit is somehow enforced elsewhere (the shell > perhaps, and/or Apache?). > > Can anyone tell me how to adjust this such that I can successfully > assign say 384MB of memory to PHP scripts ran both from browsers (i.e. > through Apache 2.2 and mod_php) as from the commandline? > > Tnx in advance, and cheers, > Olafo > > ___ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > Hi Olaf, Generally, I think it's bad programming practice to retrieve such big datasets if it is possible to do otherwise. Concider this example: I needed an app which would write a file which has a set size. The most obvious way to do this, is to make a random string which would be X bytes long. That, however, is not feasible, and very slow. The solution is to find a data size which is not too small and not too big, so disk I/O and CPU time are balanced well. Now, I suggest that you retrieve only a certain amount of SQL rows at a time, process them, and throw them into your local database. This will make your application a lot faster, by the way. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
How can I increase the shell's (or specific application's) memory limit?
Hi, I've got a question which is probably pretty easy to answer: how can I assign more memory to a PHP script running in a shell and/or in a browser. Some more background info: I'm building a PHP script that has to retrieve pretty large sets of data from a remote MySQL database, then process it, and store the results to a local database. The issue: The script (surprise, surprise) quickly runs out of memory. Now, I have already tried to increase the memory limit in php.ini (followed by an Apache restart, of course), but even when setting the limit to something high like 384MB or so, the script still bails out with a memory limit error when retrieving as little as some 50MB of data... Now, of course I could rewrite my PHP script such that it will retrieve smaller batches of data, but being a programmer I'm lazy, and I'd rather simply assign more memory to the script (actually, it's not only due to laziness, but also due to the fact that the script has to agregate data etc., and I'd rather have it do that in 1 run for a variety of reasons). It seems to me like setting the memory limit in php.ini above a value of 64MB (or so) doesn't seem to have any effect anymore. My assumption then is that the memory limit is somehow enforced elsewhere (the shell perhaps, and/or Apache?). Can anyone tell me how to adjust this such that I can successfully assign say 384MB of memory to PHP scripts ran both from browsers (i.e. through Apache 2.2 and mod_php) as from the commandline? Tnx in advance, and cheers, Olafo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
