php-general Digest 3 Sep 2013 01:31:03 -0000 Issue 8353
Topics (messages 322021 through 322023):
Re: PHP-5.5.2 +opcache segfaults with Piwik
322021 by: Jan Ehrhardt
322022 by: Lester Caine
refernces, arrays, and why does it take up so much memory?
322023 by: Daevid Vincent
Administrivia:
To subscribe to the digest, e-mail:
php-general-digest-subscr...@lists.php.net
To unsubscribe from the digest, e-mail:
php-general-digest-unsubscr...@lists.php.net
To post to the list, e-mail:
php-gene...@lists.php.net
----------------------------------------------------------------------
--- Begin Message ---
Jan Ehrhardt in php.general (Mon, 02 Sep 2013 10:57:14 +0200):
>Could you try to add a function_exists check to
>libs/upgradephp/upgrade.php?
>
>This at the function declaration of _json_encode:
>if (!function_exists('_json_encode')) { function _json_encode($var, ...
>
>And a extra } at the end.
This patch, together with upgrading to the latest OPcache from github
solved my segfaults and fatal errors.
Jan
--- End Message ---
--- Begin Message ---
Jan Ehrhardt wrote:
Could you try to add a function_exists check to
>libs/upgradephp/upgrade.php?
>
>This at the function declaration of _json_encode:
>if (!function_exists('_json_encode')) { function _json_encode($var, ...
>
>And a extra } at the end.
This patch, together with upgrading to the latest OPcache from github
solved my segfaults and fatal errors.
Jan - could you post the solution on http://dev.piwik.org/trac/ticket/4093 -
better that you get the credit than one of us pinch it ;)
--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk
--- End Message ---
--- Begin Message ---
I'm confused on how a reference works I think.
I have a DB result set in an array I'm looping over. All I simply want to do
is make the array key the "id" of the result set row.
This is the basic gist of it:
private function _normalize_result_set()
{
foreach($this->tmp_results as $k => $v)
{
$id = $v['id'];
$new_tmp_results[$id] =& $v; //2013-08-29 [dv] using a
reference here cuts the memory usage in half!
unset($this->tmp_results[$k]);
/*
if ($i++ % 1000 == 0)
{
gc_enable(); // Enable Garbage Collector
var_dump(gc_enabled()); // true
var_dump(gc_collect_cycles()); // # of elements
cleaned up
gc_disable(); // Disable Garbage Collector
}
*/
}
$this->tmp_results = $new_tmp_results;
//var_dump($this->tmp_results); exit;
unset($new_tmp_results);
}
Without using the =& reference, my data works great:
$new_tmp_results[$id] = $v;
array (size=79552)
6904 =>
array (size=4)
'id' => string '6904' (length=4)
'studio_id' => string '5' (length=1)
'genres' => string '34|' (length=3)
6905 =>
array (size=4)
'id' => string '6905' (length=4)
'studio_id' => string '5' (length=1)
'genres' => string '6|37|' (length=5)
However it takes a stupid amount of memory for some unknown reason.
MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES
MEMORY PEAK USAGE: 216,530,944 BYTES
When using the reference the memory drastically goes down to what I'd EXPECT
it to be (and actually the problem I'm trying to solve).
MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES
MEMORY PEAK USAGE: 82,051,072 BYTES
However my array is all kinds of wrong:
array (size=79552)
6904 => &
array (size=4)
'id' => string '86260' (length=5)
'studio_id' => string '210' (length=3)
'genres' => string '8|9|10|29|58|' (length=13)
6905 => &
array (size=4)
'id' => string '86260' (length=5)
'studio_id' => string '210' (length=3)
'genres' => string '8|9|10|29|58|' (length=13)
Notice that they're all the same values, although the keys seem right. I
don't understand why that happens because
foreach($this->tmp_results as $k => $v)
Should be changing $v each iteration I'd think.
Honestly, I am baffled as to why those unsets() make no difference. All I
can think is that the garbage collector doesn't run. But then I had also
tried to force gc() and that still made no difference. *sigh*
I had some other cockamamie idea where I'd use the same tmp_results array in
a tricky way to avoid a second array. The concept being I'd add 1 million
to the ['id'] (which we want as the new array key), then unset the existing
sequential key, then when all done, loop through and shift all the keys by 1
million thereby they'd be the right index ID. So add one and unset one
immediately after. Clever right? 'cept it too made no difference on memory.
Same thing is happening as above where the gc() isn't running or something
is holding all that memory until the end. *sigh*
Then I tried a different way using array_combine() and noticed something
very disturbing.
http://www.php.net/manual/en/function.array-combine.php
private function _normalize_result_set()
{
if (!$this->tmp_results || count($this->tmp_results) < 1)
return;
$D_start_mem_usage = memory_get_usage();
foreach($this->tmp_results as $k => $v)
{
$id = $v['id'];
$tmp_keys[] = $id;
if ($v['genres'])
{
$g = explode('|', $v['genres']);
$this->tmp_results[$k]['g'] = $g; //this causes a
massive spike in memory usage
}
}
//var_dump($tmp_keys, $this->tmp_results); exit;
echo "\nMEMORY USED BEFORE array_combine:
".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK:
(".number_format(memory_get_peak_usage(true)).")<br>\n";
$this->tmp_results = array_combine($tmp_keys,
$this->tmp_results);
echo "\nMEMORY USED FOR array_combine:
".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK:
(".number_format(memory_get_peak_usage(true)).")<br>\n";
var_dump($tmp_keys, $this->tmp_results); exit;
}
Just the simple act of adding that 'g' variable element to the array causes
a massive change in memory usage. WHAT THE F!?
MEMORY USED BEFORE array_combine: 105,315,264 PEAK: (224,395,264)
MEMORY USED FOR array_combine: 106,573,040 PEAK: (224,395,264)
And taking out the
$this->tmp_results[$k]['g'] = $g;
Results in
MEMORY USED BEFORE array_combine: 8,050,456 PEAK: (78,118,912)
MEMORY USED FOR array_combine: 8,050,376 PEAK: (86,507,520)
Just as a wild guess, I also added 'g' to my SQL so that PHP would already
have a placeholder variable there in tmp_results, but that made no
difference. And still used up nearly double the memory as above.
SELECT DISTINCT `id`, sag.`genres`, 'g' FROM.
--- End Message ---
