php-general Digest 29 Oct 2011 10:53:25 -0000 Issue 7543
Topics (messages 315517 through 315534):
Re: Why does this script run out of memory?
315517 by: Daniel Brown
315518 by: Tommy Pham
315519 by: Paul Halliday
315522 by: Jim Long
315533 by: shiplu
Re: Object size
315520 by: Tommy Pham
mysql_fetch_array() vs mysql_fetch_assoc() WAS: Re: [PHP] Why does this script
run out of memory?
315521 by: Daniel Brown
315523 by: Paul Halliday
315525 by: Jim Long
315526 by: Daniel Brown
Friday Distraction
315524 by: Nathan Nobbe
315527 by: Daniel Brown
315528 by: Eric Butera
315529 by: Nathan Nobbe
315530 by: Eric Butera
315531 by: Nathan Nobbe
Re:
315532 by: Nirmalya Lahiri
Zend Amf and Drupal?
315534 by: Lars Nielsen
Administrivia:
To subscribe to the digest, e-mail:
php-general-digest-subscr...@lists.php.net
To unsubscribe from the digest, e-mail:
php-general-digest-unsubscr...@lists.php.net
To post to the list, e-mail:
php-gene...@lists.php.net
----------------------------------------------------------------------
--- Begin Message ---
On Fri, Oct 28, 2011 at 16:21, Jim Long <p...@umpquanet.com> wrote:
>
> I will try experimenting with Daniel's idea of unbuffered
> queries, but my understanding is that while an unbuffered result
> resource is in use, no other SQL transactions can be conducted.
> Maybe I can get around that by using one MySQL connection for the
> unbuffered query, and another separate MySQL connection for the
> incidental SQL queries that I need to perform as I process each
> record from the large dataset.
Just for the sake of a kick-start, you could throw together a
simple class and then call that quite easily.
--
</Daniel P. Brown>
Network Infrastructure Manager
http://www.php.net/
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 9:38 AM, Jim Long <p...@umpquanet.com> wrote:
> I'm running PHP 5.3.8 on FreeBSD 8.2 with MySQL 5.1.55.
>
>
Jim,
Installed from packages or standard port tree build? Did you do any tweak
for the ports build? Any special compiler parameters in your make.conf?
I've noticed that you used MySQL extensions. Have you tried MySQLi to see
if there's any difference?
Regards,
Tommy
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 1:38 PM, Jim Long <p...@umpquanet.com> wrote:
> I'm running PHP 5.3.8 on FreeBSD 8.2 with MySQL 5.1.55.
>
> The script below is designed to be able to WHILE it's way through
> a MySQL query result set, and process each row.
>
> However, it runs out of memory a little after a quarter million
> rows. The schema fields total to about 200 bytes per row, so
> the row size doesn't seem very large.
>
> Why is this running out of memory?
>
> Thank you!
>
> Jim
>
> <?php
>
> $test_db_host = "localhost";
> $test_db_user = "foo";
> $test_db_pwd = "bar";
> $test_db_name = "farkle";
>
> $db_host = $test_db_host;
> $db_user = $test_db_user;
> $db_name = $test_db_name;
> $db_pwd = $test_db_pwd;
>
> if (!($db_conn = mysql_connect( $db_host, $db_user, $db_pwd )))
> die( "Can't connect to MySQL server\n" );
>
> if (!mysql_select_db( $db_name, $db_conn ))
> die( "Can't connect to database $db_name\n" );
>
> $qry = "select * from test_table order by contract";
>
> if ($result = mysql_query( $qry, $db_conn )) {
>
> $n = 0;
> while ($row = mysql_fetch_assoc( $result )) {
> // process row here
> $n++;
> } // while
>
Whats the difference between fetch_assoc and fetch_row?
I use:
while ($row = mysql_fetch_row($theQuery)) {
doCartwheel;
}
on just under 300 million rows and nothing craps out. I have
memory_limit set to 4GB though. Although, IIRC I pushed it up for GD
not mysql issues.
Same OS and php ver, MySQL is 5.1.48
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 02:57:02PM -0700, Tommy Pham wrote:
> On Fri, Oct 28, 2011 at 9:38 AM, Jim Long <p...@umpquanet.com> wrote:
>
> > I'm running PHP 5.3.8 on FreeBSD 8.2 with MySQL 5.1.55.
> >
> >
> Jim,
>
> Installed from packages or standard port tree build? Did you do any tweak
> for the ports build? Any special compiler parameters in your make.conf?
> I've noticed that you used MySQL extensions. Have you tried MySQLi to see
> if there's any difference?
>
> Regards,
> Tommy
MySQL server and PHP and extensions built from ports, without any
local tweaks. Nothing very interesting in /etc/make.conf:
MASTER_SITE_FREEBSD=1
CPUTYPE?=p3
USA_RESIDENT=YES
NO_INET6=YES
WITHOUT_IPV6=YES
NO_I4B=true
NO_BLUETOOTH=true
NO_IPFILTER=true
NO_KERBEROS=true
NO_ATM=true # do not build ATM related programs and libraries
NOUUCP=true # do not build uucp related programs
NO_UUCP=true # do not build uucp related programs
NO_GAMES=true
NO_PROFILE=true
PERL_VERSION=5.10.1
Port options for php5-extensions are:
_OPTIONS_READ=php5-extensions-1.5
WITH_BCMATH=true
WITHOUT_BZ2=true
WITHOUT_CALENDAR=true
WITH_CTYPE=true
WITHOUT_CURL=true
WITHOUT_DBA=true
WITH_DOM=true
WITHOUT_EXIF=true
WITH_FILEINFO=true
WITH_FILTER=true
WITHOUT_FRIBIDI=true
WITH_FTP=true
WITHOUT_GD=true
WITHOUT_GETTEXT=true
WITHOUT_GMP=true
WITH_HASH=true
WITHOUT_ICONV=true
WITHOUT_IMAP=true
WITHOUT_INTERBASE=true
WITH_JSON=true
WITHOUT_LDAP=true
WITHOUT_MBSTRING=true
WITHOUT_MCRYPT=true
WITHOUT_MSSQL=true
WITH_MYSQL=true
WITHOUT_MYSQLI=true
WITHOUT_ODBC=true
WITHOUT_OPENSSL=true
WITHOUT_PCNTL=true
WITH_PDF=true
WITH_PDO=true
WITHOUT_PDO_SQLITE=true
WITH_PGSQL=true
WITH_POSIX=true
WITHOUT_PSPELL=true
WITHOUT_READLINE=true
WITHOUT_RECODE=true
WITH_SESSION=true
WITHOUT_SHMOP=true
WITH_SIMPLEXML=true
WITHOUT_SNMP=true
WITHOUT_SOAP=true
WITHOUT_SOCKETS=true
WITHOUT_SQLITE=true
WITHOUT_SQLITE3=true
WITHOUT_SYBASE_CT=true
WITHOUT_SYSVMSG=true
WITHOUT_SYSVSEM=true
WITHOUT_SYSVSHM=true
WITHOUT_TIDY=true
WITH_TOKENIZER=true
WITHOUT_WDDX=true
WITH_XML=true
WITH_XMLREADER=true
WITHOUT_XMLRPC=true
WITH_XMLWRITER=true
WITHOUT_XSL=true
WITHOUT_YAZ=true
WITHOUT_ZIP=true
WITHOUT_ZLIB=true
As Daniel suggested, using mysql_query_unbuffered works a treat,
at the expense of a small amount of additional programming
complexity. In my prior work with Postgres, I found that it
would handle small or large datasets with equal ease, so I was
surprised to find that MySQL blew up given a sufficient number of
repeated calls to mysql_fetch_row();
Thank you for mentioning MySQLi. Although it is alphabetically
adjacent in the documentation, it had never drawn my attention.
I'll build the PHP extension and take a look when time permits.
Jim
--- End Message ---
--- Begin Message ---
I had a spider written in PHP long ago. I had similar problems.
because there were millions of rows of urls and I was fetching them in
one single query. See inline, could this modification help you. Please
test.
On Fri, Oct 28, 2011 at 10:38 PM, Jim Long <p...@umpquanet.com> wrote:
>
> I'm running PHP 5.3.8 on FreeBSD 8.2 with MySQL 5.1.55.
>
> The script below is designed to be able to WHILE it's way through
> a MySQL query result set, and process each row.
>
> However, it runs out of memory a little after a quarter million
> rows. The schema fields total to about 200 bytes per row, so
> the row size doesn't seem very large.
>
> Why is this running out of memory?
>
> Thank you!
>
> Jim
>
> <?php
>
> $test_db_host = "localhost";
> $test_db_user = "foo";
> $test_db_pwd = "bar";
> $test_db_name = "farkle";
>
> $db_host = $test_db_host;
> $db_user = $test_db_user;
> $db_name = $test_db_name;
> $db_pwd = $test_db_pwd;
>
> if (!($db_conn = mysql_connect( $db_host, $db_user, $db_pwd )))
> die( "Can't connect to MySQL server\n" );
>
> if (!mysql_select_db( $db_name, $db_conn ))
> die( "Can't connect to database $db_name\n" );
>
$limit=10;
$offset=0;
while(1){
> $qry = "select * from test_table order by contract $offset, $limit";
>
> if ($result = mysql_query( $qry, $db_conn )) {
>
> $n = 0;
> while ($row = mysql_fetch_assoc( $result )) {
> // process row here
> $n++;
> } // while
>
> mysql_free_result($result);
> echo "$n\n";
>
> } else {
>
> die( mysql_error() . "\n" );
// break the loop
break;
>
> }
$offset+=$limit;
}
> ?>
>
>
Its the same thing but you are fetching data in chunks.
Now this portion "order by contract" on quarter million rows is not a
good practice. It will slow down your query time and make the script
severely slow.
I had about 100 millions of rows in my table in the url and I was
sorting on last-visit column. Later I removed the order by and it was
much faster.
Try it and let us know.
Thanks
--
Shiplu Mokadd.im
Follow me, http://twitter.com/shiplu
Innovation distinguishes between follower and leader
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 9:43 AM, QI.VOLMAR QI <qi.vol...@gmail.com> wrote:
> Is there a function or method that calculate the object size in PHP?.
> Or maybe that calculates a xml buffer of a file? I need to check a
> size of a file, and if it's size is bigger than 500kb. I will need to
> make another file to the subsequent content. Any help will serve.
>
> Tks,
> Volmar
>
>
>
http://php.net/ref.filesystem
http://php.net/function.filesize
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 18:13, Paul Halliday <paul.halli...@gmail.com> wrote:
>
> Whats the difference between fetch_assoc and fetch_row?
>
> I use:
> while ($row = mysql_fetch_row($theQuery)) {
> doCartwheel;
> }
>
> on just under 300 million rows and nothing craps out. I have
> memory_limit set to 4GB though. Although, IIRC I pushed it up for GD
> not mysql issues.
>
> Same OS and php ver, MySQL is 5.1.48
Please don't hijack other's threads to ask a question. I've
started this as a new thread to address this question.
mysql_fetch_array() grabs all of the data and places it in a
simple numerically-keyed array.
By contrast, mysql_fetch_assoc() grabs it and populates an
associative array. This means that the column names (or aliases, et
cetera) become the keys for the array. With mysql_fetch_assoc(), you
can still call an array key by number, but it's not vice-versa with
mysql_fetch_array().
The difference in overhead, if you meant that (in which case, my
apologies for reading it as a question of functional difference), is
variable: it's based mainly on the difference between the bytes
representing the integers used as keys in mysql_fetch_array() versus
the size in bytes of the strings used as keys in mysql_fetch_assoc().
--
</Daniel P. Brown>
Network Infrastructure Manager
http://www.php.net/
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 7:19 PM, Daniel Brown <danbr...@php.net> wrote:
> On Fri, Oct 28, 2011 at 18:13, Paul Halliday <paul.halli...@gmail.com> wrote:
>>
>> Whats the difference between fetch_assoc and fetch_row?
>>
>> I use:
>> while ($row = mysql_fetch_row($theQuery)) {
>> doCartwheel;
>> }
>>
>> on just under 300 million rows and nothing craps out. I have
>> memory_limit set to 4GB though. Although, IIRC I pushed it up for GD
>> not mysql issues.
>>
>> Same OS and php ver, MySQL is 5.1.48
>
> Please don't hijack other's threads to ask a question. I've
> started this as a new thread to address this question.
>
> mysql_fetch_array() grabs all of the data and places it in a
> simple numerically-keyed array.
>
> By contrast, mysql_fetch_assoc() grabs it and populates an
> associative array. This means that the column names (or aliases, et
> cetera) become the keys for the array. With mysql_fetch_assoc(), you
> can still call an array key by number, but it's not vice-versa with
> mysql_fetch_array().
>
> The difference in overhead, if you meant that (in which case, my
> apologies for reading it as a question of functional difference), is
> variable: it's based mainly on the difference between the bytes
> representing the integers used as keys in mysql_fetch_array() versus
> the size in bytes of the strings used as keys in mysql_fetch_assoc().
>
> --
> </Daniel P. Brown>
> Network Infrastructure Manager
> http://www.php.net/
>
Sorry.
I was just throwing it out there with the hope that there might be a
tidbit that would help the OP.
I have a simliar setup and I can query far more than a 1/4 million
rows. What I offered is what I am doing differently.
--
Paul Halliday
http://www.squertproject.org/
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 06:19:56PM -0400, Daniel Brown wrote:
> On Fri, Oct 28, 2011 at 18:13, Paul Halliday <paul.halli...@gmail.com> wrote:
> >
> > Whats the difference between fetch_assoc and fetch_row?
> >
> > I use:
> > while ($row = mysql_fetch_row($theQuery)) {
> > ? ?doCartwheel;
> > }
> >
> > on just under 300 million rows and nothing craps out. I have
> > memory_limit set to 4GB though. Although, IIRC I pushed it up for GD
> > not mysql issues.
> >
> > Same OS and php ver, MySQL is 5.1.48
>
> Please don't hijack other's threads to ask a question. I've
> started this as a new thread to address this question.
>
> mysql_fetch_array() grabs all of the data and places it in a
> simple numerically-keyed array.
>
> By contrast, mysql_fetch_assoc() grabs it and populates an
> associative array. This means that the column names (or aliases, et
> cetera) become the keys for the array. With mysql_fetch_assoc(), you
> can still call an array key by number, but it's not vice-versa with
> mysql_fetch_array().
I'm not seeing any numeric keys in my mysql_fetch_assoc() arrays.
However, mysql_fetch_row (by default) does both: the array will be
indexed numerically from 0 to N-1 corresponding to the table's N
columns, and the array will also have string key indices which
correspond to the query's column names. So by default,
mysql_fetch_row uses twice the amount of data, because each field
appears in the array twice.
var_dump( $row ) will show in graphic detail.
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 18:48, Jim Long <p...@umpquanet.com> wrote:
>
> I'm not seeing any numeric keys in my mysql_fetch_assoc() arrays.
You're absolutely correct, that's my mistake: substitute
mysql_fetch_row() for mysql_fetch_assoc(). Duh.
Time to call it a week....
--
</Daniel P. Brown>
Network Infrastructure Manager
http://www.php.net/
--- End Message ---
--- Begin Message ---
Hi gang,
Thinking database i/o would be the slowest part of a request in your new zf
/ amf app?
Leave it to Zend_Amf to burn more cycles marshaling the protocol!
http://s289.photobucket.com/albums/ll238/quickshiftin/?action=view¤t=ScreenShot2011-10-24at74724PM.png
Happy Halloween!
-nathan
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 18:36, Nathan Nobbe <quickshif...@gmail.com> wrote:
> Hi gang,
>
> Thinking database i/o would be the slowest part of a request in your new zf
> / amf app?
>
> Leave it to Zend_Amf to burn more cycles marshaling the protocol!
>
> http://s289.photobucket.com/albums/ll238/quickshiftin/?action=view¤t=ScreenShot2011-10-24at74724PM.png
Ends up looking startlingly like the original Sim City.
--
</Daniel P. Brown>
Network Infrastructure Manager
http://www.php.net/
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 7:07 PM, Daniel Brown <danbr...@php.net> wrote:
> On Fri, Oct 28, 2011 at 18:36, Nathan Nobbe <quickshif...@gmail.com> wrote:
>> Hi gang,
>>
>> Thinking database i/o would be the slowest part of a request in your new zf
>> / amf app?
>>
>> Leave it to Zend_Amf to burn more cycles marshaling the protocol!
>>
>> http://s289.photobucket.com/albums/ll238/quickshiftin/?action=view¤t=ScreenShot2011-10-24at74724PM.png
>
> Ends up looking startlingly like the original Sim City.
>
> --
> </Daniel P. Brown>
> Network Infrastructure Manager
> http://www.php.net/
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>
kcachegrind is great.
SimCity is great! Just the other day I was thinking about that
soundtrack it played over the PC speaker, hilarious.
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 5:23 PM, Eric Butera <eric.but...@gmail.com> wrote:
> On Fri, Oct 28, 2011 at 7:07 PM, Daniel Brown <danbr...@php.net> wrote:
> > On Fri, Oct 28, 2011 at 18:36, Nathan Nobbe <quickshif...@gmail.com>
> wrote:
> >> Hi gang,
> >>
> >> Thinking database i/o would be the slowest part of a request in your new
> zf
> >> / amf app?
> >>
> >> Leave it to Zend_Amf to burn more cycles marshaling the protocol!
> >>
> >>
> http://s289.photobucket.com/albums/ll238/quickshiftin/?action=view¤t=ScreenShot2011-10-24at74724PM.png
> >
> > Ends up looking startlingly like the original Sim City.
> >
> > --
> > </Daniel P. Brown>
> > Network Infrastructure Manager
> > http://www.php.net/
> >
> > --
> > PHP General Mailing List (http://www.php.net/)
> > To unsubscribe, visit: http://www.php.net/unsub.php
> >
> >
>
> kcachegrind is great.
>
> SimCity is great! Just the other day I was thinking about that
> soundtrack it played over the PC speaker, hilarious.
>
Agreed, the only thing that isn't great in this context .. Zend_Amf, haha!
-nathan
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 7:24 PM, Nathan Nobbe <quickshif...@gmail.com> wrote:
>
>
> On Fri, Oct 28, 2011 at 5:23 PM, Eric Butera <eric.but...@gmail.com> wrote:
>>
>> On Fri, Oct 28, 2011 at 7:07 PM, Daniel Brown <danbr...@php.net> wrote:
>> > On Fri, Oct 28, 2011 at 18:36, Nathan Nobbe <quickshif...@gmail.com>
>> > wrote:
>> >> Hi gang,
>> >>
>> >> Thinking database i/o would be the slowest part of a request in your
>> >> new zf
>> >> / amf app?
>> >>
>> >> Leave it to Zend_Amf to burn more cycles marshaling the protocol!
>> >>
>> >>
>> >> http://s289.photobucket.com/albums/ll238/quickshiftin/?action=view¤t=ScreenShot2011-10-24at74724PM.png
>> >
>> > Ends up looking startlingly like the original Sim City.
>> >
>> > --
>> > </Daniel P. Brown>
>> > Network Infrastructure Manager
>> > http://www.php.net/
>> >
>> > --
>> > PHP General Mailing List (http://www.php.net/)
>> > To unsubscribe, visit: http://www.php.net/unsub.php
>> >
>> >
>>
>> kcachegrind is great.
>>
>> SimCity is great! Just the other day I was thinking about that
>> soundtrack it played over the PC speaker, hilarious.
>
> Agreed, the only thing that isn't great in this context .. Zend_Amf, haha!
> -nathan
>
I saw your post on that the other day - looks like there's a native
php extension you might look into. The author was quite proud of the
hasty response it was capable of, going as far as to say his library
was faster than flash can decode it.
--- End Message ---
--- Begin Message ---
On Fri, Oct 28, 2011 at 5:26 PM, Eric Butera <eric.but...@gmail.com> wrote:
> On Fri, Oct 28, 2011 at 7:24 PM, Nathan Nobbe <quickshif...@gmail.com>
> wrote:
> >
> >
> > On Fri, Oct 28, 2011 at 5:23 PM, Eric Butera <eric.but...@gmail.com>
> wrote:
> >>
> >> On Fri, Oct 28, 2011 at 7:07 PM, Daniel Brown <danbr...@php.net> wrote:
> >> > On Fri, Oct 28, 2011 at 18:36, Nathan Nobbe <quickshif...@gmail.com>
> >> > wrote:
> >> >> Hi gang,
> >> >>
> >> >> Thinking database i/o would be the slowest part of a request in your
> >> >> new zf
> >> >> / amf app?
> >> >>
> >> >> Leave it to Zend_Amf to burn more cycles marshaling the protocol!
> >> >>
> >> >>
> >> >>
> http://s289.photobucket.com/albums/ll238/quickshiftin/?action=view¤t=ScreenShot2011-10-24at74724PM.png
> >> >
> >> > Ends up looking startlingly like the original Sim City.
> >> >
> >> > --
> >> > </Daniel P. Brown>
> >> > Network Infrastructure Manager
> >> > http://www.php.net/
> >> >
> >> > --
> >> > PHP General Mailing List (http://www.php.net/)
> >> > To unsubscribe, visit: http://www.php.net/unsub.php
> >> >
> >> >
> >>
> >> kcachegrind is great.
> >>
> >> SimCity is great! Just the other day I was thinking about that
> >> soundtrack it played over the PC speaker, hilarious.
> >
> > Agreed, the only thing that isn't great in this context .. Zend_Amf,
> haha!
> > -nathan
> >
>
> I saw your post on that the other day - looks like there's a native
> php extension you might look into. The author was quite proud of the
> hasty response it was capable of, going as far as to say his library
> was faster than flash can decode it.
>
Yeah, I got it built under php 5.3, it rips.
Funny thing is amfphp has a userpace serializer of it's own and it of
course takes less time than db i/o.
Dropping in the extension takes the time spent serializing down to
practically nothing.
Suffice it to say we'll be migrating to the extension on subsequent revision
of the project I alluded to.
Also, the more I use zf, the less I like, lol. Look at this line from the
documentation - Fools!
"Zend_Server_Interface provides an interface that mimics PHP 5's
SoapServer class;
all server classes should implement this interface in order to provide a
standard server API."
-nathan
--- End Message ---
--- Begin Message ---
.Do you want to feel orgasms several times?
http://macrokid.ma.funpic.de/com.friend.page.php?jroyahoo_id=09ep0
--- End Message ---
--- Begin Message ---
Hey List,
I am making a webservice in Drupal with AMF Zend to provide some methods
to a Flash app. I have now managed to use standard Drupal services
(node,user and files) and I have made my own method to create users.
But! ... When I try to pass on arguments from Flash to Drupal it fails.
It says that it expects zero arguments but I have provided 4 arguments.
I have posted some info here : http://drupal.org/node/1323678
Can you give me some guidelines? or pointers on what to do?
Kind Regards
Lars Nielsen
www.lfweb.dk / www.gearworks.dk
--- End Message ---
