Re: [PHP] memory allocation error

[Carol Peck]

On 11/13/2012 6:29 AM, Matijn Woudt wrote:




On Tue, Nov 13, 2012 at 12:23 AM, Carol Peck > wrote:



On 11/12/2012 11:51 AM, Matijn Woudt wrote:


On Mon, Nov 12, 2012 at 3:17 PM, Carol Peck mailto:carolap...@gmail.com>> wrote:

Sebastian,
Yes, I do , but this particular error never gets into my
custom handler.
I have also set it so that fatal errors fall through, and
that doesn't seem to make any difference (again, probably
because it never gets there).

Carol


Can you post the code of the error handler? I guess the bug is
there. Bugs like these are mostly because of recursion errors or
something. (Don't know if it's possible, but an error inside the
error handler?)
Fatal errors will btw always fall through, you can't catch fatal
errors. You should, for testing, turn off your custom error
handler and see what it does.

- Matijn

Ps. Please bottom-post on this mailing list.

Here is the error class (I hope I"m posting the code the right
way).  I will turn this off for a while.
It's main purpose is to format the message before logging.  I do
know that things are getting logged properly.

Thanks again.




Your hosting is using SuPHP which most likely causes these problems. 
You have two options:

1) Get a better host, or even better, get a dedicated server.
2) Increase your memory limit, even though your script doesn't really 
reaches that limit, it seems that SuPHP somehow messes that up. Read 
more at [1].


- Matijn

[1] http://forum.inmotionhosting.com/viewtopic.php?t=3147

Matijn,
Thanks so much, I had found that post but thought it was related to the 
SilverStripe CMS they were referencing.  I've been coming to the same 
conclusion about the host.

Cheers,
Carol

Re: [PHP] memory allocation error

[Matijn Woudt]

On Tue, Nov 13, 2012 at 12:23 AM, Carol Peck  wrote:

>
> On 11/12/2012 11:51 AM, Matijn Woudt wrote:
>
>
>  On Mon, Nov 12, 2012 at 3:17 PM, Carol Peck  wrote:
>
>> Sebastian,
>> Yes, I do , but this particular error never gets into my custom handler.
>> I have also set it so that fatal errors fall through, and that doesn't
>> seem to make any difference (again, probably because it never gets there).
>>
>> Carol
>>
>>
>  Can you post the code of the error handler? I guess the bug is there.
> Bugs like these are mostly because of recursion errors or something. (Don't
> know if it's possible, but an error inside the error handler?)
> Fatal errors will btw always fall through, you can't catch fatal errors.
> You should, for testing, turn off your custom error handler and see what it
> does.
>
>  - Matijn
>
>  Ps. Please bottom-post on this mailing list.
>
> Here is the error class (I hope I"m posting the code the right way).  I
> will turn this off for a while.
> It's main purpose is to format the message before logging.  I do know that
> things are getting logged properly.
>
> Thanks again.
>



Your hosting is using SuPHP which most likely causes these problems. You
have two options:
1) Get a better host, or even better, get a dedicated server.
2) Increase your memory limit, even though your script doesn't really
reaches that limit, it seems that SuPHP somehow messes that up. Read more
at [1].

- Matijn

[1] http://forum.inmotionhosting.com/viewtopic.php?t=3147

Re: [PHP] memory allocation error

[Carol Peck]

On 11/12/2012 11:51 AM, Matijn Woudt wrote:


On Mon, Nov 12, 2012 at 3:17 PM, Carol Peck > wrote:


Sebastian,
Yes, I do , but this particular error never gets into my custom
handler.
I have also set it so that fatal errors fall through, and that
doesn't seem to make any difference (again, probably because it
never gets there).

Carol


Can you post the code of the error handler? I guess the bug is there. 
Bugs like these are mostly because of recursion errors or something. 
(Don't know if it's possible, but an error inside the error handler?)
Fatal errors will btw always fall through, you can't catch fatal 
errors. You should, for testing, turn off your custom error handler 
and see what it does.


- Matijn

Ps. Please bottom-post on this mailing list.
Here is the error class (I hope I"m posting the code the right way).  I 
will turn this off for a while.
It's main purpose is to format the message before logging.  I do know 
that things are getting logged properly.


Thanks again.

message  = $message;
$this->filename = $filename;
$this->line  = $linenum;
$this->vars = $vars;
  }
  public static function handle($errno, $errmsg, $filename, $line, $vars)
  {
$self = new self($errmsg, $filename, $line, $vars);
switch ($errno) {
  case E_USER_ERROR:
return $self->handleError();
  case E_USER_WARNING:
  case E_WARNING:
return $self->handleWarning();
  case E_USER_NOTICE:
  case E_NOTICE:
return $self->handleNotice();
  default:
// Returning false tells PHP to revert control to the
// default error handler
return false;
}
  }

  public function handleError()
  {
// Get backtrace
ob_start();
debug_print_backtrace();
$backtrace = ob_get_flush();
$body =<filename}
Line:  {$this->line}
Backtrace:
{$backtrace}
EOT;
// Use PHP's error_log() function to send an email
//error_log($body, 1, 'sysad...@example.com', "Fatal error 
occurred\n");
error_log($body, 1, 'cp...@wyom.net', "Subject: Fatal error 
occurred\n");

 return error_log($body, 3, APP_PATH . $this->_errorLog);
exit(1); // non 0 exit status indicates script failure
  }

  /**
   * Handle warnings
   *
   * Warnings indicate environmental errors; email sysadmin and
   * continue
   *
   * @return boolean
   */
public function handleWarning()
{
  $body =<filename}
Line:  {$this->line}
EOT;
$body = date('[Y-m-d H:i:s] ') . $body . "\n";

 return error_log($body,3, APP_PATH . $this->_errorLog);
  }

  /**
   * Handle notices
   *
   * @return boolean
   */
public function handleNotice()
{
  $body =<filename}
Line:  {$this->line}
EOT;
$body = date('[Y-m-d H:i:s] ') . $body . "\n";
return error_log($body, 3, APP_PATH . $this->_noticeLog);
  }
}

/**
 * Set ErrorHandler::handle() as default error handler
 */
set_error_handler(array('ErrorHandler', 'handle'));

Re: [PHP] memory allocation error

[Matijn Woudt]

On Mon, Nov 12, 2012 at 3:17 PM, Carol Peck  wrote:

> Sebastian,
> Yes, I do , but this particular error never gets into my custom handler.
> I have also set it so that fatal errors fall through, and that doesn't
> seem to make any difference (again, probably because it never gets there).
>
> Carol
>
>
Can you post the code of the error handler? I guess the bug is there. Bugs
like these are mostly because of recursion errors or something. (Don't know
if it's possible, but an error inside the error handler?)
Fatal errors will btw always fall through, you can't catch fatal errors.
You should, for testing, turn off your custom error handler and see what it
does.

- Matijn

Ps. Please bottom-post on this mailing list.

Re: [PHP] memory allocation error

[Jim Lucas]

On 11/12/2012 8:54 AM, Carol Peck wrote:

Jim,
I just found that the die didn't fix it after all - just ran into it again.
So still looking for ideas!
thanks,

Carol



Then it must be something in either your code or the way PHP is doing 
some garbage collection with the libs you are using.



--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory allocation error

[Jim Lucas]

On 11/12/2012 7:50 AM, Carol Peck wrote:

Jim,
  Thanks for your idea - using die prevents it from coming up.  As I
mentioned, it is rather random so sometimes hard to verify.
My auto_prepend and auto_append have no value in  php.ini.  I'm
wondering why you suggested that?


If something was injecting code using the auto_append param, then you 
would not know about it, but it would still cause you issues.  And by 
issuing a die or exit at the end of the code would show if it was your 
code or something running after all your script has completed.




Best,
Carol

On 11/12/2012 8:09 AM, Jim Lucas wrote:

On 11/11/2012 08:45 AM, Carol Peck wrote:

Hi all,
I've been chasing around a memory allocation error for some time and
can't figure it out. It is somewhat random - I can run the script 3
times and then it will happen, or sometimes the first time.

It happens at the very end of a script, actually after the script has
finished running. I will see the following after the closing 
tag:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to
allocate 494142432 bytes) in Unknown on line 0

Just previous to this my mem usage is 3772104


The script itself does some database work (deletes, inserts, selects)
but nothing very heavy and writes out an XML file. I use adodb 5.18, the
server is PHP 5.3.18 (this was updated recently) and it is a VPS on
inmotionhosting.com.

As you can see, the memory limit is 256M so it's really high and I never
see that I'm using more than 4M. The error doesn't fall through my error
class - I'm assuming that's because it is happening after the script is
complete.

I am completely out of ideas on how to trap it or figure it out.
Any ideas would be appreciated!






Try adding exit or die at the end of what you know is the end of your
scripts. See if the problem continues.  Maybe at the very end of your
customer error handler.

If the problem stops showing up, you might want to look at your PHP
config to see if anything is setup in the "auto_append_file" section.







--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: [PHP] memory allocation error

[James]

> Original Message 
>From: James 
>To: "Carol Peck" , "Jim Lucas" 
>Cc: php-general@lists.php.net
>Sent: Mon, Nov 12, 2012, 12:53 PM
>Subject: Re: [PHP] memory allocation error
>
>This could be an issue with the library you're using, adodb, I'd check to see 
>if it has any debugging options to enable. I'm not familiar with it at all but 
>that may be helpful. I'd also check out adodbs bug tracker, if one exists. 
>Another suggestion would be use a profiler, such as xdebug, I believe thats 
>the name.
>
>Carol Peck  wrote:
>
>>Jim,
>>I just found that the die didn't fix it after all - just ran into it
>>again.
>>So still looking for ideas!
>>thanks,
>>
>>Carol
>>
>>On 11/12/2012 8:09 AM, Jim Lucas wrote:
>>> On 11/11/2012 08:45 AM, Carol Peck wrote:
>>>> Hi all,
>>>> I've been chasing around a memory allocation error for some time and
>>>> can't figure it out. It is somewhat random - I can run the script 3
>>>> times and then it will happen, or sometimes the first time.
>>>>
>>>> It happens at the very end of a script, actually after the script
>>has
>>>> finished running. I will see the following after the closing 
>>
>>>> tag:
>>>>
>>>> Fatal error: Allowed memory size of 268435456 bytes exhausted (tried
>>to
>>>> allocate 494142432 bytes) in Unknown on line 0
>>>>
>>>> Just previous to this my mem usage is 3772104
>>>>
>>>>
>>>> The script itself does some database work (deletes, inserts,
>>selects)
>>>> but nothing very heavy and writes out an XML file. I use adodb 5.18,
>>the
>>>> server is PHP 5.3.18 (this was updated recently) and it is a VPS on
>>>> inmotionhosting.com.
>>>>
>>>> As you can see, the memory limit is 256M so it's really high and I
>>never
>>>> see that I'm using more than 4M. The error doesn't fall through my
>>error
>>>> class - I'm assuming that's because it is happening after the script
>>is
>>>> complete.
>>>>
>>>> I am completely out of ideas on how to trap it or figure it out.
>>>> Any ideas would be appreciated!
>>>>
>>>>
>>>>
>>>>
>>>
>>> Try adding exit or die at the end of what you know is the end of your
>>
>>> scripts. See if the problem continues.  Maybe at the very end of your
>>
>>> customer error handler.
>>>
>>> If the problem stops showing up, you might want to look at your PHP
>>> config to see if anything is setup in the "auto_append_file" section.
>>>
>>
>>
>>--
>>PHP General Mailing List (http://www.php.net/)
>>To unsubscribe, visit: http://www.php.net/unsub.php
>
>--
>Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Sorry for top posting.


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory allocation error

[James]

This could be an issue with the library you're using, adodb, I'd check to see 
if it has any debugging options to enable. I'm not familiar with it at all but 
that may be helpful. I'd also check out adodbs bug tracker, if one exists. 
Another suggestion would be use a profiler, such as xdebug, I believe thats the 
name.

Carol Peck  wrote:

>Jim,
>I just found that the die didn't fix it after all - just ran into it
>again.
>So still looking for ideas!
>thanks,
>
>Carol
>
>On 11/12/2012 8:09 AM, Jim Lucas wrote:
>> On 11/11/2012 08:45 AM, Carol Peck wrote:
>>> Hi all,
>>> I've been chasing around a memory allocation error for some time and
>>> can't figure it out. It is somewhat random - I can run the script 3
>>> times and then it will happen, or sometimes the first time.
>>>
>>> It happens at the very end of a script, actually after the script
>has
>>> finished running. I will see the following after the closing 
>
>>> tag:
>>>
>>> Fatal error: Allowed memory size of 268435456 bytes exhausted (tried
>to
>>> allocate 494142432 bytes) in Unknown on line 0
>>>
>>> Just previous to this my mem usage is 3772104
>>>
>>>
>>> The script itself does some database work (deletes, inserts,
>selects)
>>> but nothing very heavy and writes out an XML file. I use adodb 5.18,
>the
>>> server is PHP 5.3.18 (this was updated recently) and it is a VPS on
>>> inmotionhosting.com.
>>>
>>> As you can see, the memory limit is 256M so it's really high and I
>never
>>> see that I'm using more than 4M. The error doesn't fall through my
>error
>>> class - I'm assuming that's because it is happening after the script
>is
>>> complete.
>>>
>>> I am completely out of ideas on how to trap it or figure it out.
>>> Any ideas would be appreciated!
>>>
>>>
>>>
>>>
>>
>> Try adding exit or die at the end of what you know is the end of your
>
>> scripts. See if the problem continues.  Maybe at the very end of your
>
>> customer error handler.
>>
>> If the problem stops showing up, you might want to look at your PHP 
>> config to see if anything is setup in the "auto_append_file" section.
>>
>
>
>-- 
>PHP General Mailing List (http://www.php.net/)
>To unsubscribe, visit: http://www.php.net/unsub.php

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Re: [PHP] memory allocation error

[Carol Peck]

Jim,
I just found that the die didn't fix it after all - just ran into it again.
So still looking for ideas!
thanks,

Carol

On 11/12/2012 8:09 AM, Jim Lucas wrote:

On 11/11/2012 08:45 AM, Carol Peck wrote:

Hi all,
I've been chasing around a memory allocation error for some time and
can't figure it out. It is somewhat random - I can run the script 3
times and then it will happen, or sometimes the first time.

It happens at the very end of a script, actually after the script has
finished running. I will see the following after the closing  
tag:


Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to
allocate 494142432 bytes) in Unknown on line 0

Just previous to this my mem usage is 3772104


The script itself does some database work (deletes, inserts, selects)
but nothing very heavy and writes out an XML file. I use adodb 5.18, the
server is PHP 5.3.18 (this was updated recently) and it is a VPS on
inmotionhosting.com.

As you can see, the memory limit is 256M so it's really high and I never
see that I'm using more than 4M. The error doesn't fall through my error
class - I'm assuming that's because it is happening after the script is
complete.

I am completely out of ideas on how to trap it or figure it out.
Any ideas would be appreciated!






Try adding exit or die at the end of what you know is the end of your 
scripts. See if the problem continues.  Maybe at the very end of your 
customer error handler.


If the problem stops showing up, you might want to look at your PHP 
config to see if anything is setup in the "auto_append_file" section.





--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory allocation error

[Carol Peck]

Jim,
 Thanks for your idea - using die prevents it from coming up.  As I 
mentioned, it is rather random so sometimes hard to verify.
My auto_prepend and auto_append have no value in  php.ini.  I'm 
wondering why you suggested that?


Best,
Carol

On 11/12/2012 8:09 AM, Jim Lucas wrote:

On 11/11/2012 08:45 AM, Carol Peck wrote:

Hi all,
I've been chasing around a memory allocation error for some time and
can't figure it out. It is somewhat random - I can run the script 3
times and then it will happen, or sometimes the first time.

It happens at the very end of a script, actually after the script has
finished running. I will see the following after the closing  
tag:


Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to
allocate 494142432 bytes) in Unknown on line 0

Just previous to this my mem usage is 3772104


The script itself does some database work (deletes, inserts, selects)
but nothing very heavy and writes out an XML file. I use adodb 5.18, the
server is PHP 5.3.18 (this was updated recently) and it is a VPS on
inmotionhosting.com.

As you can see, the memory limit is 256M so it's really high and I never
see that I'm using more than 4M. The error doesn't fall through my error
class - I'm assuming that's because it is happening after the script is
complete.

I am completely out of ideas on how to trap it or figure it out.
Any ideas would be appreciated!






Try adding exit or die at the end of what you know is the end of your 
scripts. See if the problem continues.  Maybe at the very end of your 
customer error handler.


If the problem stops showing up, you might want to look at your PHP 
config to see if anything is setup in the "auto_append_file" section.





--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory allocation error

[Jim Lucas]

On 11/11/2012 08:45 AM, Carol Peck wrote:

Hi all,
I've been chasing around a memory allocation error for some time and
can't figure it out. It is somewhat random - I can run the script 3
times and then it will happen, or sometimes the first time.

It happens at the very end of a script, actually after the script has
finished running. I will see the following after the closing  tag:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to
allocate 494142432 bytes) in Unknown on line 0

Just previous to this my mem usage is 3772104


The script itself does some database work (deletes, inserts, selects)
but nothing very heavy and writes out an XML file. I use adodb 5.18, the
server is PHP 5.3.18 (this was updated recently) and it is a VPS on
inmotionhosting.com.

As you can see, the memory limit is 256M so it's really high and I never
see that I'm using more than 4M. The error doesn't fall through my error
class - I'm assuming that's because it is happening after the script is
complete.

I am completely out of ideas on how to trap it or figure it out.
Any ideas would be appreciated!






Try adding exit or die at the end of what you know is the end of your 
scripts. See if the problem continues.  Maybe at the very end of your 
customer error handler.


If the problem stops showing up, you might want to look at your PHP 
config to see if anything is setup in the "auto_append_file" section.


--
Jim Lucas

http://www.cmsws.com/
http://www.cmsws.com/examples/

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory allocation error

[Carol Peck]

Sebastian,
Yes, I do , but this particular error never gets into my custom handler.
I have also set it so that fatal errors fall through, and that doesn't 
seem to make any difference (again, probably because it never gets there).


Carol

On 11/11/2012 11:16 AM, Sebastian Krebs wrote:

Hi,

Do you use a custom error handler?

Regards,
Sebastian


2012/11/11 Carol Peck


Hi all,
I've been chasing around a memory allocation error for some time and can't
figure it out.  It is somewhat random - I can run the script 3 times and
then it will happen, or sometimes the first time.

It happens at the very end of a script, actually after the script has
finished running.  I will see the following after the closing  tag:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to
allocate 494142432 bytes) in Unknown on line 0

Just previous to this my mem usage is 3772104


The script itself does some database work (deletes, inserts, selects) but
nothing very heavy and writes out an XML file.  I use adodb 5.18, the
server is PHP 5.3.18 (this was updated recently) and it is a VPS on
inmotionhosting.com.

As you can see, the memory limit is 256M so it's really high and I never
see that I'm using more than 4M.  The error doesn't fall through my error
class - I'm assuming that's because it is happening after the script is
complete.

I am completely out of ideas on how to trap it or figure it out.
Any ideas would be appreciated!




--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php








--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] memory allocation error

[Carol Peck]

Hi all,
I've been chasing around a memory allocation error for some time and 
can't figure it out.  It is somewhat random - I can run the script 3 
times and then it will happen, or sometimes the first time.


It happens at the very end of a script, actually after the script has 
finished running.  I will see the following after the closing  tag:


Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to 
allocate 494142432 bytes) in Unknown on line 0


Just previous to this my mem usage is 3772104


The script itself does some database work (deletes, inserts, selects) 
but nothing very heavy and writes out an XML file.  I use adodb 5.18, 
the server is PHP 5.3.18 (this was updated recently) and it is a VPS on 
inmotionhosting.com.


As you can see, the memory limit is 256M so it's really high and I never 
see that I'm using more than 4M.  The error doesn't fall through my 
error class - I'm assuming that's because it is happening after the 
script is complete.


I am completely out of ideas on how to trap it or figure it out.
Any ideas would be appreciated!




--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory allocation

[Marcelo Bianchi]

On 09/25/2012 02:40 PM, shiplu wrote:

Also if the final data on presentation layer is very small you can
calculate it in mysql using stored procedure.


okay, also something to try.
thank you !
regards,
marcelo

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory allocation

[shiplu]

I would like to see how you are reading data from database?


There are many ways to reduce memory usage. Here is a very common way
reduce memory.

instead of

$res = mysql_query("select * from table1 limit 1000");
while($row = mysql_fetch_assoc($res)){
 /// process row.
}

Do this,

for($x = 0; $x< 1000; $x+=100;}
$res = mysql_query(sprintf("select required_col1,  required_col2,
...  from table1 limit %d, %d",$x, 100));
while($row = mysql_fetch_assoc($res)){
/// process row.
}
}



-- 
Shiplu.Mokadd.im
ImgSign.com | A dynamic signature machine
Innovation distinguishes between follower and leader

Re: [PHP] memory allocation

[Marcelo Bianchi]

Uhm, Okay,

I get it ... I am loading all at once, will try this out.

Here is what I do:

-- begin code
$network = "IA";
$station = "BJI";
$channel = "BHZ";
$year = 2011;

$top = NULL;

$result = mysql_query('select '.$year.' as year, net, station, loc, 
channel, start_day, start_time, end_day, end_time from data_'.$year.' 
where net = "'.$network.'" and station = "'.$station.'" and scal_date 
BETWEEN "'.$year.'-01-01" AND "'.$year.'-12-31" and channel = 
"'.$channel.'" ') or die(mysql_error());


echo "Loading $year ... ";
while ($r = mysql_fetch_array($result, MYSQL_ASSOC)) {
if ($top === NULL)
$top = new Segments($r, 120);
$top->add($r);
unset ($r);
}
echo "Done\n";

echo "Nothing free: ".memory_get_usage(True)." \n";
mysql_free_result($result);
mysql_close($db_link);
unset($result);
echo "SQL free: ".memory_get_usage(True)." \n";
unset($top);
echo "Segments free: ".memory_get_usage(True)." \n";
-- end code

The Segments object is a class with some functionality but, after 
processing the results of each row it stores two integers (or floats due 
to the large numbers, number of seconds from epoch * 1.0 + 
microseconds).


I will try to implement this chunk mode of fetching the data from sql to 
see if the php consumption goes down.


regards,

Marcelo

On 09/25/2012 02:23 PM, shiplu wrote:

I would like to see how you are reading data from database?


There are many ways to reduce memory usage. Here is a very common way
reduce memory.

instead of

$res = mysql_query("select * from table1 limit 1000");
while($row = mysql_fetch_assoc($res)){
  /// process row.
}

Do this,

for($x = 0; $x< 1000; $x+=100;}
 $res = mysql_query(sprintf("select required_col1,  required_col2,
...  from table1 limit %d, %d",$x, 100));
 while($row = mysql_fetch_assoc($res)){
 /// process row.
 }
}



--
Shiplu.Mokadd.im 
ImgSign.com | A dynamic signature machine
Innovation distinguishes between follower and leader




--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory allocation

[Marcelo Bianchi]

Hi Matijn,

That is good test, to install php 32bit side to side with the 64bit to 
test. Will see how hard is to accomplish that.


regards,

Marcelo

On 09/25/2012 01:58 PM, Matijn Woudt wrote:

On Tue, Sep 25, 2012 at 12:49 PM, Marcelo Bianchi
 wrote:

Dear list,

I developed a script that have to create considerable array of integer
numbers in memory. Currently my script has to accommodate in memory two
arrays of around 120.000 numbers (for one year of data, I would like to
reach a 2 years per query) that are loaded from a mysql server.

I developed the script on a ubuntu 32bit machine and when I bring this
script to our server, a 64bit opensuse the normal 128M of memory limit from
php was exhausted.

On my ubuntu system the script was consuming around 30Mb of memory / year,
on the opensuse to run the same script it consumes more than 90Mb / year of
memory.

Is there a way to reduce this memory consumption on a 64bits machine ?

Also, I made a test with four different virtual box machines (two opensuse
and two ubuntu machines with the same PHP version, opensuse 11.4 and ubuntu
11.04 all running php-5.3.5) and it was quite choking the result for me. I
am still wandering what is the difference between those and how to explain
the results.

A print screen of my machine running the four virtual boxes can be seen at:

https://sites.google.com/site/foo4funreborn/phpcomp

I would greatly thanks any help,
with my best regards,

Marcelo Bianchi


Hi,

If you suspect it's because of 64bits, you can always install 32bit
PHP next to the 64bit version. At some point Ubuntu should support
this out of the box with apt-get, but it seems that it's broken,
atleast for now. Other than that, there's probably little you can do,
as PHP will use 64bit integers and 64bit pointers internally, and
there's pretty much nothing you can do about that.

What I would suggest is just change the memory limit of PHP, you can
do that with ini_set inside the PHP script. I assume you have plenty
of ram in the system? If not, you should probably just go back to
32bit OS.

- Matijn




--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory allocation

[Matijn Woudt]

On Tue, Sep 25, 2012 at 12:49 PM, Marcelo Bianchi
 wrote:
> Dear list,
>
> I developed a script that have to create considerable array of integer
> numbers in memory. Currently my script has to accommodate in memory two
> arrays of around 120.000 numbers (for one year of data, I would like to
> reach a 2 years per query) that are loaded from a mysql server.
>
> I developed the script on a ubuntu 32bit machine and when I bring this
> script to our server, a 64bit opensuse the normal 128M of memory limit from
> php was exhausted.
>
> On my ubuntu system the script was consuming around 30Mb of memory / year,
> on the opensuse to run the same script it consumes more than 90Mb / year of
> memory.
>
> Is there a way to reduce this memory consumption on a 64bits machine ?
>
> Also, I made a test with four different virtual box machines (two opensuse
> and two ubuntu machines with the same PHP version, opensuse 11.4 and ubuntu
> 11.04 all running php-5.3.5) and it was quite choking the result for me. I
> am still wandering what is the difference between those and how to explain
> the results.
>
> A print screen of my machine running the four virtual boxes can be seen at:
>
> https://sites.google.com/site/foo4funreborn/phpcomp
>
> I would greatly thanks any help,
> with my best regards,
>
> Marcelo Bianchi

Hi,

If you suspect it's because of 64bits, you can always install 32bit
PHP next to the 64bit version. At some point Ubuntu should support
this out of the box with apt-get, but it seems that it's broken,
atleast for now. Other than that, there's probably little you can do,
as PHP will use 64bit integers and 64bit pointers internally, and
there's pretty much nothing you can do about that.

What I would suggest is just change the memory limit of PHP, you can
do that with ini_set inside the PHP script. I assume you have plenty
of ram in the system? If not, you should probably just go back to
32bit OS.

- Matijn

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] memory allocation

[Marcelo Bianchi]

Dear list,

I developed a script that have to create considerable array of integer 
numbers in memory. Currently my script has to accommodate in memory two 
arrays of around 120.000 numbers (for one year of data, I would like to 
reach a 2 years per query) that are loaded from a mysql server.


I developed the script on a ubuntu 32bit machine and when I bring this 
script to our server, a 64bit opensuse the normal 128M of memory limit 
from php was exhausted.


On my ubuntu system the script was consuming around 30Mb of memory / 
year, on the opensuse to run the same script it consumes more than 90Mb 
/ year of memory.


Is there a way to reduce this memory consumption on a 64bits machine ?

Also, I made a test with four different virtual box machines (two 
opensuse and two ubuntu machines with the same PHP version, opensuse 
11.4 and ubuntu 11.04 all running php-5.3.5) and it was quite choking 
the result for me. I am still wandering what is the difference between 
those and how to explain the results.


A print screen of my machine running the four virtual boxes can be seen at:

https://sites.google.com/site/foo4funreborn/phpcomp

I would greatly thanks any help,
with my best regards,

Marcelo Bianchi

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

RE: [PHP] debugging PHP memory corruption

[Andrew Morum]

>Hi Andy,
>Have you tried running it with valgrind? (See [1] for tips).
>Otherwise, you might have more luck at php-internals mailing list,
>since it is related to PHP development if you intend to fix either PHP
>or the plugin.
>
>Matijn
>[1] https://bugs.php.net/bugs-getting-valgrind-log.php

Thanks Matijn,

We have tried running Valgrind against it, but there is nothing obvious
(to us at least) in the output.

I'll try the Internals mailing list.

Andy


The information contained in this email is intended for the personal and 
confidential use
of the addressee only. It may also be privileged information. If you are not 
the intended
recipient then you are hereby notified that you have received this document in 
error and
that any review, distribution or copying of this document is strictly 
prohibited. If you have
received  this communication in error, please notify Brendata immediately on:

+44 (0)1268 466100, or email 'techni...@brendata.co.uk'

Brendata (UK) Ltd
Nevendon Hall, Nevendon Road, Basildon, Essex. SS13 1BX  UK
Registered Office as above. Registered in England No. 2764339

See our current vacancies at www.brendata.co.uk

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] debugging PHP memory corruption

[Matijn Woudt]

On Wed, Jan 4, 2012 at 3:34 PM, Andrew Morum  wrote:
> We've got a problem with PHP 5.3.8 and a third party (open source)
> library (WSo2 SOAP).
>
>
>
> At some point during the request to the PHP script, some structures seem
> to be getting corrupted causing PHP to crash.
>
>
>
> Depending on the code in the PHP script, it's either crashing during in
> a print_r/var_export or crashing when the PHP shuts down (the exact
> location of the crash moves around when we alter the code).
>
>
>
> We can reprod this on Win32 and Linux now. But how can we go about
> debugging where the corruption occurred? The address of the corrupted
> data is not known until the segfault/access violation occurs so we can't
> set a watch on it (as far as we can tell).
>
>
>
> Here's what we see in the debugger on Win32:
>
>
>
>
> Any tips welcomed.
>
>
>
> Andy.

Hi Andy,

Have you tried running it with valgrind? (See [1] for tips).
Otherwise, you might have more luck at php-internals mailing list,
since it is related to PHP development if you intend to fix either PHP
or the plugin.

Matijn

[1] https://bugs.php.net/bugs-getting-valgrind-log.php

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] debugging PHP memory corruption

[Andrew Morum]

We've got a problem with PHP 5.3.8 and a third party (open source)
library (WSo2 SOAP).



At some point during the request to the PHP script, some structures seem
to be getting corrupted causing PHP to crash.



Depending on the code in the PHP script, it's either crashing during in
a print_r/var_export or crashing when the PHP shuts down (the exact
location of the crash moves around when we alter the code).



We can reprod this on Win32 and Linux now. But how can we go about
debugging where the corruption occurred? The address of the corrupted
data is not known until the segfault/access violation occurs so we can't
set a watch on it (as far as we can tell).



Here's what we see in the debugger on Win32:



Unhandled exception at 0x1008eb45 (php5ts.dll) in php-cgi.exe:

Access violation reading location 0x4cac5400



php5ts.dll!_zval_ptr_dtor(_zval_struct * * zval_ptr=0x4cab5400)  + 0x5
bytes

php5ts.dll!zend_hash_destroy(_hashtable * ht=0x034cc0f0)  + 0x27 bytes


php5ts.dll!zend_objects_free_object_storage(_zend_object *
object=0x03489fa0, void * * * tsrm_ls=0x003f42b0)  + 0x2b bytes

php5ts.dll!zend_objects_store_free_object_storage(_zend_objects_store *
objects=0x003f685c, void * * * tsrm_ls=0x003f42b0)  + 0x9c bytes   

php5ts.dll!shutdown_executor(void * * * tsrm_ls=0x003f42b0)  + 0x2fe
bytes

php5ts.dll!zend_deactivate(void * * * tsrm_ls=0x003f42b0)  + 0x91 bytes

php5ts.dll!php_request_shutdown(void * dummy=0x)  + 0x31f bytes


php-cgi.exe!main(int argc=1, char * * argv=0x003f31d0)  + 0xe89 bytes

php-cgi.exe!__tmainCRTStartup()  + 0x10f bytes



Any tips welcomed.



Andy.



The information contained in this email is intended for the personal and 
confidential use
of the addressee only. It may also be privileged information. If you are not 
the intended
recipient then you are hereby notified that you have received this document in 
error and
that any review, distribution or copying of this document is strictly 
prohibited. If you have
received  this communication in error, please notify Brendata immediately on:

+44 (0)1268 466100, or email 'techni...@brendata.co.uk'

Brendata (UK) Ltd
Nevendon Hall, Nevendon Road, Basildon, Essex. SS13 1BX  UK
Registered Office as above. Registered in England No. 2764339

See our current vacancies at www.brendata.co.uk

[PHP] memory overflow :/

[Tontonq Tontonq]

hi  i can't see anything wrong that will cause memory problem

but parsing 1gb memory limit doesn't come enough for just parsing a 50 kb
file

but when i try to parse another file that is 24 kb 24 mb becomes enough
memory

here is the script

http://pastebin.com/H9mG7ucU

if you go to
rss.php?id=175069119656&titlebaslik=1

no problem

when you try to parse
rss.php?id=102741716484127&titlebaslik=1

Allowed memory size of 25165824 bytes exhausted (tried to allocate 90564532
bytes)

Just tried to increase with

ini_set('memory_limit','2048M');
still same error
Allowed memory size of -2147483648 bytes exhausted (tried to allocate
2137883596 bytes) in rss.php  on line 24

any idea ?

Re: [PHP] Memory limit Problem

[Richard Quadling]

On 27 June 2011 14:16, Graham Drabble  wrote:
> On 23 Jun 2011 usene...@drabble.me.uk (Graham Drabble) wrote in
> news:xns9f0d980579291grahamdrabbleline...@id-77355.user.dfncis.de:
>
>> On 23 Jun 2011 ja...@nixsecurity.org wrote in
>> news:23a261c.efc7a3a4b4a0468ab221437b2018e...@webmail.lomag.net:
>>
>>>
>>> Check the value of the upload_max_filesize and post_max_size
>>> directives in the php.ini.
>>
>> From phpinfo()
>>
>> upload_max_filesize 800M 800M
>> post_max_size 1000M 1000M
>>
>> (fwiw the file I'm uploading is ~85MB)
>
> Any ideas from anyone?
>
> FWIW, the setup works on another server but doesn't work on this
> server. The upload_max_filesize, post_max_size and memory_limit are the
> same on both servers.
>
> Both are PHP Version 5.2.6-1+lenny9

I'd check for other differences in the phpinfo() output.

Anything that is different should be examined and verified as being appropriate.

-- 
Richard Quadling
Twitter : EE : Zend : PHPDoc
@RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY : bit.ly/lFnVea

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit Problem

[Lester Caine]

Graham Drabble wrote:

 From phpinfo()
>
>  upload_max_filesize 800M 800M
>  post_max_size 1000M 1000M
>
>  (fwiw the file I'm uploading is ~85MB)

Any ideas from anyone?

FWIW, the setup works on another server but doesn't work on this
server. The upload_max_filesize, post_max_size and memory_limit are the
same on both servers.

Both are PHP Version 5.2.6-1+lenny9


Memory the same size on both machines? The error message you gave was certainly 
one of being 'out of resources' rather than hitting the file size limits ...


--
Lester Caine - G8HFL
-
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk//
Firebird - http://www.firebirdsql.org/index.php

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit Problem

[Graham Drabble]

On 23 Jun 2011 usene...@drabble.me.uk (Graham Drabble) wrote in 
news:xns9f0d980579291grahamdrabbleline...@id-77355.user.dfncis.de:

> On 23 Jun 2011 ja...@nixsecurity.org wrote in
> news:23a261c.efc7a3a4b4a0468ab221437b2018e...@webmail.lomag.net: 
> 
>> 
>> Check the value of the upload_max_filesize and post_max_size
>> directives in the php.ini. 
> 
> From phpinfo()
> 
> upload_max_filesize 800M 800M 
> post_max_size 1000M 1000M 
> 
> (fwiw the file I'm uploading is ~85MB)

Any ideas from anyone?

FWIW, the setup works on another server but doesn't work on this 
server. The upload_max_filesize, post_max_size and memory_limit are the 
same on both servers.

Both are PHP Version 5.2.6-1+lenny9 

-- 
Graham Drabble
http://www.drabble.me.uk/

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit Problem

[Graham Drabble]

On 23 Jun 2011 ja...@nixsecurity.org wrote in
news:23a261c.efc7a3a4b4a0468ab221437b2018e...@webmail.lomag.net: 

> 
> Check the value of the upload_max_filesize and post_max_size
> directives in the php.ini. 

>From phpinfo()

upload_max_filesize 800M 800M 
post_max_size 1000M 1000M 

(fwiw the file I'm uploading is ~85MB)

-- 
Graham Drabble
http://www.drabble.me.uk/

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit Problem

[james]

Check the value of the upload_max_filesize and post_max_size directives in the 
php.ini.


> Original Message 
>From: Graham Drabble 
>To: php-general@lists.php.net
>Sent: Thu, Jun 23, 2011, 9:46 AM
>Subject: [PHP] Memory limit Problem
>
>Hi,
>
>I've got a problem with setting the memory limit in php.ini.
>
>I have
>memory_limit = 1024M  ;
>set in php.ini
>
>Checking with phpinfo says I have
>
>memory_limit 1024M 1024M
>
>However when uploading a large file I get
>
>Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
>allocate 81439166 bytes)
>
>(ie it seems I only have 128M available).
>
>There's plenty of RAM on the server.
>
>Any ideas?
>--
>Graham Drabble
>http://www.drabble.me.uk/
>
>--
>PHP General Mailing List (http://www.php.net/)
>To unsubscribe, visit: http://www.php.net/unsub.php


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit Problem

[Peet Grobler]

On 6/23/2011 3:29 PM, Graham Drabble wrote:
> (ie it seems I only have 128M available).
> 
> There's plenty of RAM on the server.
> 
> Any ideas?

Check for the upload_max_filesize parameter.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory limit Problem

[Graham Drabble]

Hi,

I've got a problem with setting the memory limit in php.ini.

I have
memory_limit = 1024M  ;
set in php.ini

Checking with phpinfo says I have

memory_limit 1024M 1024M 

However when uploading a large file I get

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to 
allocate 81439166 bytes) 

(ie it seems I only have 128M available).

There's plenty of RAM on the server.

Any ideas?
-- 
Graham Drabble
http://www.drabble.me.uk/

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory usage/release & GC

[Tommy Pham]

On Fri, Dec 31, 2010 at 2:02 AM, Peter Lind  wrote:
>
> On Dec 31, 2010 6:20 AM, "Tommy Pham"  wrote:
>>
>> Hi folks,
>>
>> With the recent thread about password & security, I wrote a small quick
>> script to generate a random or all possible passwords based on certain
>> parameters for a brute force use.  On a very long running execution for a
>> complex password in length with full use of the keys (94 characters),
>> including upper case, the script seems to consumes more memory (shown in
>> Windows task manager) as time progress.  Below are snippets from the
>> script
>> file that does the workload:
>>
>> while (!$this->isMax())
>> {
>>        for ($b = 0; $b <= $this->pwdLength; $b++)
>>        {
>>                if ($this->counter[$b] < $this->max)
>>                {
>>                        $this->pwd[$b] =
>> $this->charList[$this->counter[$b]];
>>                        $this->counter[$b]++;
>>                        break;
>>                }
>>                else
>>                {
>>                        $this->counter[$b] = 1;
>>                        $this->pwd[$b] = $this->charList[0];
>>                }
>>        }
>> }
>>
>> private function isMax()
>> {
>>        for ($a = $this->pwdLength-1; $a>=0; $a--)
>>        {
>>                if ($this->counter[$a] < $this->max) return false;
>>        }
>>        return true;
>> }
>>
>> Could someone please tell me why the above code consumes additional memory
>> as time progress for the execution of the while loop?  Researching PHP GC
>> on
>> google didn't shed light on problem.  Generating all possible combinations
>> for 20 length with 94 possibilities each, the script easily consumes more
>> than 1GB RAM in few minutes.  BTW, gc_enabled() reports on.
>>
>> Thanks,
>> Tommy
>>
>>
>
> Are you storing or throwing away the passwords? Also, lots of code is
> missing from that post, no idea if you've got a memory leak in the rest of
> the code
>
> Regards
> Peter

Hi Peter,

Thanks for the reply.  After I sent the e-mail last night, I did a
little debugging.  As it turns out, xdebug was the culprit.  I gotta
remember to disable xdebug while coding...  Anyway, here's the full
code.  Anyone is welcome to use it.

$timeStart = microtime(true);

//set_time_limit(0);
ini_set('max_execution_time', 0);
ini_set('memory_limit', -1);
class PwdGen
{
const ALPHA_LOWER = 'abcdefghijklmnopqrstuvwxyz';
const ALPHA_UPPER = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ';
const NUMERIC = '0123456789';
const SPECIALS = '!...@#$%^&*()`~-_=+[]\{}|;:\'",./<>?';
private $chars;
private $charList;
private $max;
private $pwdLength;
private $pwdList; // array
private $pwd;
private $counter; // array
private $insertDb = false;

public function __construct($pwdLength = 8, $useUpper = true,
$useNumeric = true, $useSpecials = true, $insertDb = true, $specials =
'')
{
$this->insertDb = $insertDb;
$this->pwdLength = intval($pwdLength);
$this->chars = self::ALPHA_LOWER;
if ($useUpper) $this->chars .= self::ALPHA_UPPER;
if ($useNumeric) $this->chars .= self::NUMERIC;
if ($useSpecials) $this->chars .= empty($specials) ? 
self::SPECIALS
: $specials;
}
private function init()
{
$this->charList = str_split($this->chars, 1);
$this->max = sizeof($this->charList);
$this->pwd = str_split(str_repeat($this->charList[0], 
$this->pwdLength), 1);

$this->pwdList = array(); // array
$this->counter = array();
for ($a = 0; $a < $this->pwdLength; $a++)
$this->counter[$a] = 1;
$this->counter[$this->pwdLength-1] = 0;
}
public function setPasswordLength($pwdLength)
{
$this->pwdLength = intval($pwdLength);
}
public function setUseChars($useUpper = true, $useNumeric = true,
$useSpecials = true, $specials = '')
{
$this->chars = self::ALPHA_LOWER;
if ($useUpper) $this->chars .= self::ALPHA_UPPER;
if ($useNumeric) $this->chars .= self::NUMERIC;
if ($useSpecials) $this->chars .= empty($specials) ? 
self::SPECIALS
: $specials;
}
public function getRandom()
{
$this->init();
for ($a = 0; $a < $this->pwdLength; $a++)
$this->pwd[$a] = $this->charList[rand(0, $this->max - 
1)];
$pwd = implode('', $this->pwd);
$output = $this->max.': '.$this->chars.' ('.pow($this->max,
$this->pwdLength).') '.PHP_EOL.$pwd.PHP_EOL;
return $output;
}
public function getAll($continue = true)
{
$this->init();
$sql = 'INSERT INTO `password`(`guid`, `length`, `password`,
`counte

Re: [PHP] memory usage/release & GC

[Peter Lind]

On Dec 31, 2010 6:20 AM, "Tommy Pham"  wrote:
>
> Hi folks,
>
> With the recent thread about password & security, I wrote a small quick
> script to generate a random or all possible passwords based on certain
> parameters for a brute force use.  On a very long running execution for a
> complex password in length with full use of the keys (94 characters),
> including upper case, the script seems to consumes more memory (shown in
> Windows task manager) as time progress.  Below are snippets from the
script
> file that does the workload:
>
> while (!$this->isMax())
> {
>for ($b = 0; $b <= $this->pwdLength; $b++)
>{
>if ($this->counter[$b] < $this->max)
>{
>$this->pwd[$b] =
> $this->charList[$this->counter[$b]];
>$this->counter[$b]++;
>break;
>}
>else
>{
>$this->counter[$b] = 1;
>$this->pwd[$b] = $this->charList[0];
>}
>}
> }
>
> private function isMax()
> {
>for ($a = $this->pwdLength-1; $a>=0; $a--)
>{
>if ($this->counter[$a] < $this->max) return false;
>}
>return true;
> }
>
> Could someone please tell me why the above code consumes additional memory
> as time progress for the execution of the while loop?  Researching PHP GC
on
> google didn't shed light on problem.  Generating all possible combinations
> for 20 length with 94 possibilities each, the script easily consumes more
> than 1GB RAM in few minutes.  BTW, gc_enabled() reports on.
>
> Thanks,
> Tommy
>
>

Are you storing or throwing away the passwords? Also, lots of code is
missing from that post, no idea if you've got a memory leak in the rest of
the code

Regards
Peter

[PHP] memory usage/release & GC

[Tommy Pham]

Hi folks,

With the recent thread about password & security, I wrote a small quick
script to generate a random or all possible passwords based on certain
parameters for a brute force use.  On a very long running execution for a
complex password in length with full use of the keys (94 characters),
including upper case, the script seems to consumes more memory (shown in
Windows task manager) as time progress.  Below are snippets from the script
file that does the workload:

while (!$this->isMax())
{
for ($b = 0; $b <= $this->pwdLength; $b++)
{
if ($this->counter[$b] < $this->max)
{
$this->pwd[$b] =
$this->charList[$this->counter[$b]];
$this->counter[$b]++;
break;
}
else
{
$this->counter[$b] = 1;
$this->pwd[$b] = $this->charList[0];
}
}
}

private function isMax()
{
for ($a = $this->pwdLength-1; $a>=0; $a--)
{
if ($this->counter[$a] < $this->max) return false;
}
return true;
}

Could someone please tell me why the above code consumes additional memory
as time progress for the execution of the while loop?  Researching PHP GC on
google didn't shed light on problem.  Generating all possible combinations
for 20 length with 94 possibilities each, the script easily consumes more
than 1GB RAM in few minutes.  BTW, gc_enabled() reports on.

Thanks,
Tommy


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory investigation

[Larry Garfield]

Hi folks.  I have a complicated PHP app that is eating up more memory than I 
think it should.  I have a couple of theories as to where it could be going, 
but I need some way to verify it.

There are a number of large data structures (mostly arrays) that get built up 
throughout the course of the request.  What I'd like to be able to do is at 
certain points check to see how much memory those data structures are using.  
Because they're not all built at once, the usual "check memory before, build, 
check after" routine won't work.  Plus, that gets screwed up by PHP's use of 
copy-on-write at times.

I know that it would result in essentially over-reporting, but I would ideally 
like to be able to ignore the copy-on-write issue and say "if this variable 
were the only thing in memory (and its dependents, of course, for a nested 
array), what would its memory usage be?  I just have no idea how to do that.

Anyone have a suggestion for how to accomplish that?

--Larry Garfield

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Re: how do I use php://memory?

[Daniel Brown]

On Sat, Jan 30, 2010 at 12:18, Shawn McKenzie  wrote:
>
> Done.  Thanks Dan.  http://bugs.php.net/bug.php?id=50886

Thank you, sir.  I thanked you on Facebook when I saw the report
come in, but wanted to thank you properly here as well.

-- 

daniel.br...@parasane.net || danbr...@php.net
http://www.parasane.net/ || http://www.pilotpig.net/
Looking for hosting or dedicated servers?  Ask me how we can fit your budget!

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Re: how do I use php://memory?

[Shawn McKenzie]

Daniel P. Brown wrote:
> (Typing from the DROID, so forgive the top-posting.)
> 
> Shawn, would you take a few moments to submit this as a bug at
> http://bugs.php.net/? I know you well enough that, if you say the docs suck,
> they probably do.
> 
> On Jan 29, 2010 10:47 PM, "Shawn McKenzie"  wrote:
> 
> Eric Lee wrote:
>> On Sat, Jan 30, 2010 at 9:00 AM, Shawn McKenzie > wrote:
>>
>>> ...
> So maybe it only works with an open file/stream resource? Hard to tell
> with no docs.
> 
> 
> --
> 
> Thanks!
> -Shawn
> http://www.spidean.com
> 

Done.  Thanks Dan.  http://bugs.php.net/bug.php?id=50886

-- 
Thanks!
-Shawn
http://www.spidean.com

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] how do I use php://memory?

[Mari Masuda]

On Jan 29, 2010, at 10:57 PM, Mari Masuda wrote:

> 
> On Jan 29, 2010, at 4:38 PM, Nathan Nobbe wrote:
> 
>> On Fri, Jan 29, 2010 at 5:35 PM, Mari Masuda  
>> wrote:
>> Hello,
>> 
>> I have a function that uses tidy to attempt to clean up a bunch of crappy 
>> HTML that I inherited.  In order to use tidy, I write the crappy HTML to a 
>> temporary file on disk, run tidy, and extract and return the clean(er) HTML. 
>>  The program itself works fine but with all of the disk access, it runs 
>> quite slowly.
>> 
>> why read from disk in the first place?
>> 
>> http://us3.php.net/manual/en/tidy.parsestring.php
>> 
>> -nathan 
> 
> Thank you, this looks like exactly what I need.  Unfortunately I cannot get 
> it to work on my machine.

[snip]

So I figured it out... I was using the wrong command to compile libtidy.  (I am 
not a *nix geek so I had no idea I was messing it up.)  To get it working, I 
followed the instructions I found here: 
http://www.php.net/manual/en/ref.tidy.php#64281

My setup is slightly different from the person who wrote the directions in that 
I am running OS X 10.6.2 and PHP 5.2.12.  However, the only difference between 
the instructions I followed and what I actually had to do is that the line to 
comment out for me was line 525 instead of 508.  (The actual line is: typedef 
unsigned long ulong;)

Thank you everyone for your help!
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] how do I use php://memory?

[Mari Masuda]

On Jan 29, 2010, at 4:38 PM, Nathan Nobbe wrote:

> On Fri, Jan 29, 2010 at 5:35 PM, Mari Masuda  wrote:
> Hello,
> 
> I have a function that uses tidy to attempt to clean up a bunch of crappy 
> HTML that I inherited.  In order to use tidy, I write the crappy HTML to a 
> temporary file on disk, run tidy, and extract and return the clean(er) HTML.  
> The program itself works fine but with all of the disk access, it runs quite 
> slowly.
> 
> why read from disk in the first place?
> 
> http://us3.php.net/manual/en/tidy.parsestring.php
> 
> -nathan 

Thank you, this looks like exactly what I need.  Unfortunately I cannot get it 
to work on my machine.  I recompiled PHP with --with-tidy=/usr/local and this 
is the version and modules in use:

[Fri Jan 29 22:50:41] ~: php -vPHP 5.2.12 (cli) (built: Jan 29 2010 22:35:24) 
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies
[Fri Jan 29 22:52:30] ~: php -m
[PHP Modules]
ctype
date
dom
filter
gd
hash
iconv
json
libxml
mbstring
mysql
mysqli
pcre
PDO
pdo_mysql
pdo_sqlite
posix
Reflection
session
SimpleXML
SPL
SQLite
standard
tidy
tokenizer
xml
xmlreader
xmlwriter
zlib

[Zend Modules]

[Fri Jan 29 22:52:34] ~: 


When I run this test code
=
blahhello";
$config = array('indent' => true,
'wrap' => '0');

// Tidy
$tidy = new tidy();
var_dump($tidy);
$tidy->parseString($html, $config, 'utf8');
var_dump($tidy);
$tidy->cleanRepair();
var_dump($tidy);
echo tidy_get_output($tidy);
var_dump($tidy);
?>
=

I get this output:
=
object(tidy)#1 (2) {
  ["errorBuffer"]=>
  NULL
  ["value"]=>
  NULL
}
object(tidy)#1 (2) {
  ["errorBuffer"]=>
  NULL
  ["value"]=>
  NULL
}
object(tidy)#1 (2) {
  ["errorBuffer"]=>
  NULL
  ["value"]=>
  NULL
}
object(tidy)#1 (2) {
  ["errorBuffer"]=>
  NULL
  ["value"]=>
  NULL
}


I have no clue what I'm doing wrong...

Mari
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Re: how do I use php://memory?

[Daniel P. Brown]

(Typing from the DROID, so forgive the top-posting.)

Shawn, would you take a few moments to submit this as a bug at
http://bugs.php.net/? I know you well enough that, if you say the docs suck,
they probably do.

On Jan 29, 2010 10:47 PM, "Shawn McKenzie"  wrote:

Eric Lee wrote:
> On Sat, Jan 30, 2010 at 9:00 AM, Shawn McKenzie wrote:
>
>>...
So maybe it only works with an open file/stream resource? Hard to tell
with no docs.


--

Thanks!
-Shawn
http://www.spidean.com

-- 
PHP General Mailing List (http://www.php.net/)
To unsubsc...

Re: [PHP] Re: how do I use php://memory?

[Shawn McKenzie]

Eric Lee wrote:
> On Sat, Jan 30, 2010 at 9:00 AM, Shawn McKenzie wrote:
> 
>> Mari Masuda wrote:
>>
>>> Has anyone ever successfully used php://memory before?  If so, what
>>> can I do to use it in my code?  Thank you.
>> No, but I was intrigued to try it, so I tested this:
>>
>> $text = 'Some text.';
>> file_put_contents('php://memory', $text);
>> echo file_get_contents('php://memory');
>>
>> And it returned nothing.  The docs suck on this and it apparently
>> doesn't work.  I see others use it with fopen(), but there is no mention
>> of which file functions it works with and which it doesn't.
>>
>>
> 
> Shawn
> 
> I did a sample test from the manual with fopen like this,
> 
> 
>  
> $fp = fopen('php://memory', 'r+');
> 
> if ($fp)
> {
> fputs($fp, "line 1\n");
> }
> 
> rewind($fp);
> echo stream_get_contents($fp);
> 
> ?>
> 
> console output
> 
> F:\wc\trunk>php -f m.php
> line 1
> 
> 
> 
> Regards,
> Eric,

So maybe it only works with an open file/stream resource? Hard to tell
with no docs.


-- 
Thanks!
-Shawn
http://www.spidean.com

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Re: how do I use php://memory?

[Eric Lee]

On Sat, Jan 30, 2010 at 9:00 AM, Shawn McKenzie wrote:

> Mari Masuda wrote:
>
> > Has anyone ever successfully used php://memory before?  If so, what
> > can I do to use it in my code?  Thank you.
>
> No, but I was intrigued to try it, so I tested this:
>
> $text = 'Some text.';
> file_put_contents('php://memory', $text);
> echo file_get_contents('php://memory');
>
> And it returned nothing.  The docs suck on this and it apparently
> doesn't work.  I see others use it with fopen(), but there is no mention
> of which file functions it works with and which it doesn't.
>
>

Shawn

I did a sample test from the manual with fopen like this,




console output

F:\wc\trunk>php -f m.php
line 1



Regards,
Eric,

--
> Thanks!
> -Shawn
> http://www.spidean.com
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

[PHP] Re: how do I use php://memory?

[Shawn McKenzie]

Mari Masuda wrote:

> Has anyone ever successfully used php://memory before?  If so, what
> can I do to use it in my code?  Thank you.

No, but I was intrigued to try it, so I tested this:

$text = 'Some text.';
file_put_contents('php://memory', $text);
echo file_get_contents('php://memory');

And it returned nothing.  The docs suck on this and it apparently
doesn't work.  I see others use it with fopen(), but there is no mention
of which file functions it works with and which it doesn't.

-- 
Thanks!
-Shawn
http://www.spidean.com

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] how do I use php://memory?

[Jochem Maas]

Op 1/30/10 1:35 AM, Mari Masuda schreef:
> Hello,
> 
> I have a function that uses tidy to attempt to clean up a bunch of crappy 
> HTML that I inherited.  In order to use tidy, I write the crappy HTML to a 
> temporary file on disk, run tidy, and extract and return the clean(er) HTML.  
> The program itself works fine but with all of the disk access, it runs quite 
> slowly.  I saw on this page (http://www.php.net/manual/en/wrappers.php.php) 
> that I could write to memory by using php://memory.  Unfortunately, I could 
> not quite get it to work.  The problem is that in the below function, the 
> code within the [[[if (file_exists($dirty_file_path))]]] does not get run if 
> I change [[[$dirty_file_path]]] to "php://memory".  Has anyone ever 
> successfully used php://memory before?  If so, what can I do to use it in my 
> code?  Thank you.

what does it matter that it runs slowly, run it once and be done with it?
alternatively use the php tidy extension and avoid the file system and shelling 
out altogether.

actually I'd imagine shelling out from a webserver process is the bottle neck 
and not saving/reading
from the file system.

lastly I don't suppose you've heard of /dev/shm ?

and, er, no, I don't have experience with php://memory but you might try 
searching for
other people's code:


http://www.google.com/codesearch?q=php%3A%2F%2Fmemory&hl=en&btnG=Search+Code

> //==
> function cleanUpHtml($dirty_html, $enclose_text=true) {
> 
>   $parent_dir = "/filesWrittenFromPHP/";
>   $now = time();
>   $random = rand();
>   
>   //save dirty html to a file so tidy can process it
>   $dirty_file_path = $parent_dir . "dirty" . $now . "-" . $random . 
> ".txt";
>   $dirty_handle = fopen($dirty_file_path, "w");
>   fwrite($dirty_handle, $dirty_html);
>   fclose($dirty_handle);
> 
>   $cleaned_html = "";
>   $start = 0;
>   $end = 0;
> 
>   if (file_exists($dirty_file_path)) {
>   exec("/usr/local/bin/tidy -miq -wrap 0 -asxhtml --doctype 
> strict --preserve-entities yes --css-prefix \"tidy\" --tidy-mark no 
> --char-encoding utf8 --drop-proprietary-attributes yes  --fix-uri yes " . 
> ($enclose_text ? "--enclose-text yes " : "") . $dirty_file_path . " 2> 
> /dev/null");
> 
>   $tidied_html = file_get_contents($dirty_file_path);
>   
>   $start = strpos($tidied_html, "") + 6;
>   $end = strpos($tidied_html, "") - 1;
>   
>   $cleaned_html = trim(substr($tidied_html, $start, ($end - 
> $start)));
>   }
>   
>   unlink($dirty_file_path);
> 
> 
>   return $cleaned_html;
> }
> //==
> 
> 


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] how do I use php://memory?

[Nathan Nobbe]

On Fri, Jan 29, 2010 at 5:35 PM, Mari Masuda wrote:

> Hello,
>
> I have a function that uses tidy to attempt to clean up a bunch of crappy
> HTML that I inherited.  In order to use tidy, I write the crappy HTML to a
> temporary file on disk, run tidy, and extract and return the clean(er) HTML.
>  The program itself works fine but with all of the disk access, it runs
> quite slowly.


why read from disk in the first place?

http://us3.php.net/manual/en/tidy.parsestring.php

-nathan

[PHP] how do I use php://memory?

[Mari Masuda]

Hello,

I have a function that uses tidy to attempt to clean up a bunch of crappy HTML 
that I inherited.  In order to use tidy, I write the crappy HTML to a temporary 
file on disk, run tidy, and extract and return the clean(er) HTML.  The program 
itself works fine but with all of the disk access, it runs quite slowly.  I saw 
on this page (http://www.php.net/manual/en/wrappers.php.php) that I could write 
to memory by using php://memory.  Unfortunately, I could not quite get it to 
work.  The problem is that in the below function, the code within the [[[if 
(file_exists($dirty_file_path))]]] does not get run if I change 
[[[$dirty_file_path]]] to "php://memory".  Has anyone ever successfully used 
php://memory before?  If so, what can I do to use it in my code?  Thank you.


//==
function cleanUpHtml($dirty_html, $enclose_text=true) {

$parent_dir = "/filesWrittenFromPHP/";
$now = time();
$random = rand();

//save dirty html to a file so tidy can process it
$dirty_file_path = $parent_dir . "dirty" . $now . "-" . $random . 
".txt";
$dirty_handle = fopen($dirty_file_path, "w");
fwrite($dirty_handle, $dirty_html);
fclose($dirty_handle);

$cleaned_html = "";
$start = 0;
$end = 0;

if (file_exists($dirty_file_path)) {
exec("/usr/local/bin/tidy -miq -wrap 0 -asxhtml --doctype 
strict --preserve-entities yes --css-prefix \"tidy\" --tidy-mark no 
--char-encoding utf8 --drop-proprietary-attributes yes  --fix-uri yes " . 
($enclose_text ? "--enclose-text yes " : "") . $dirty_file_path . " 2> 
/dev/null");

$tidied_html = file_get_contents($dirty_file_path);

$start = strpos($tidied_html, "") + 6;
$end = strpos($tidied_html, "") - 1;

$cleaned_html = trim(substr($tidied_html, $start, ($end - 
$start)));
}

unlink($dirty_file_path);


return $cleaned_html;
}
//==


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory efficient hash table extension? like lchash ...

[J Ravi Menon]

> values were stored, the APC storage began to slow down *dramatically*.  I
> wasn't certain if APC was using only RAM or was possibly also writing to
> disk.  Performance tanked so quickly that I set it aside as an option and
> moved on.
IIRC, i think it is built over shm and there is no disk backing store.


> memcached gives no guarantee about data persistence.  I need to have a hash
> table that will contain all the values I set.  They don't need to survive a
> server shutdown (don't need to be written to disk), but I can not afford for
> the server to throw away values that don't fit into memory.  If there is a
> way to configure memcached guarantee storage, that might work.

True but the lru policy only kicks in lazily. So if you ensure that
you never hit near the max allowed limit (-m option), and you store
your key-val pairs with no expiry, it will be present till the next
restart. So essentially you would have to estimate the value for the
-m option to big enough to accommodate all possible key-val pairs (the
evictions counter in memcached stats should remain 0). BTW, I have
seen this implementation behavior in 1.2.x series but not sure it is
necessarily guaranteed in future versions.

Ravi



On Mon, Jan 25, 2010 at 3:49 PM, D. Dante Lorenso  wrote:
> J Ravi Menon wrote:
>>
>> PHP does expose sys V shared-memory apis (shm_* functions):
>> http://us2.php.net/manual/en/book.sem.php
>
>
> I will look into this.  I really need a key/value map, though and would
> rather not have to write my own on top of SHM.
>
>
>> If you already have apc installed, you could also try:
>> http://us2.php.net/manual/en/book.apc.php
>> APC also allows you to store user specific data too (it will be in a
>> shared memory).
>
>
> I've looked into the apc_store and apc_fetch routines:
> http://php.net/manual/en/function.apc-store.php
> http://www.php.net/manual/en/function.apc-fetch.php
> ... but quickly ran out of memory for APC and though I figured out how to
> configure it to use more (adjust shared memory allotment), there were other
> problems.  I ran into issues with logs complaining about "cache slamming"
> and other known bugs with APC version 3.1.3p1.  Also, after several million
> values were stored, the APC storage began to slow down *dramatically*.  I
> wasn't certain if APC was using only RAM or was possibly also writing to
> disk.  Performance tanked so quickly that I set it aside as an option and
> moved on.
>
>
>> Haven't tried these myself, so I would do some quick tests to ensure
>> if they meet your performance requirements. In theory, it should be
>> faster than berkeley-db like solutions (which is also another option
>> but it seems something similar like MongoDB was not good enough?).
>
>
> I will run more tests against MongoDB.  Initially I tried to use it to store
> everything.  If I only store my indexes, it might fare better. Certainly,
> though, running queries and updates against a remote server will always be
> slower than doing the lookups locally in ram.
>
>
>> I  am curious to know if someone here has run these tests. Note that
>> with memcached installed locally (on the same box running php), it can
>> be surprisingly efficient - using pconnect(),  caching the handler in
>> a static var for a given request cycle etc...
>
> memcached gives no guarantee about data persistence.  I need to have a hash
> table that will contain all the values I set.  They don't need to survive a
> server shutdown (don't need to be written to disk), but I can not afford for
> the server to throw away values that don't fit into memory.  If there is a
> way to configure memcached guarantee storage, that might work.
>
> -- Dante
>
>
>> On Sun, Jan 24, 2010 at 9:39 AM, D. Dante Lorenso 
>> wrote:
>>>
>>> shiplu wrote:

 On Sun, Jan 24, 2010 at 3:11 AM, D. Dante Lorenso 
 wrote:
>
> All,
>
> I'm loading millions of records into a backend PHP cli script that I
> need to build a hash index from to optimize key lookups for data that
> I'm importing into a MySQL database.  The problem is that storing this
> data in a PHP array is not very memory efficient and my millions of
> records are consuming about 4-6 GB of ram.
>
 What are you storing? An array of row objects??
 In that case storing only the row id is will reduce the memory.
>>>
>>> I am querying a MySQL database which contains 40 million records and
>>> mapping
>>> string columns to numeric ids.  You might consider it normalizing the
>>> data.
>>>
>>> Then, I am importing a new 40 million records and comparing the new
>>> values
>>> to the old values.  Where the value matches, I update records, but where
>>> they do not match, I insert new records, and finally I go back and delete
>>> old records.  So, the net result is that I have a database with 40
>>> million
>>> records that I need to "sync" on a daily basis.
>>>
 If you are loading full row objects, it will take a lot of memory.
 But if you just 

Re: [PHP] memory efficient hash table extension? like lchash ...

[D. Dante Lorenso]

J Ravi Menon wrote:

PHP does expose sys V shared-memory apis (shm_* functions):
http://us2.php.net/manual/en/book.sem.php



I will look into this.  I really need a key/value map, though and would 
rather not have to write my own on top of SHM.




If you already have apc installed, you could also try:
http://us2.php.net/manual/en/book.apc.php
APC also allows you to store user specific data too (it will be in a
shared memory).



I've looked into the apc_store and apc_fetch routines:
http://php.net/manual/en/function.apc-store.php
http://www.php.net/manual/en/function.apc-fetch.php
... but quickly ran out of memory for APC and though I figured out how 
to configure it to use more (adjust shared memory allotment), there were 
other problems.  I ran into issues with logs complaining about "cache 
slamming" and other known bugs with APC version 3.1.3p1.  Also, after 
several million values were stored, the APC storage began to slow down 
*dramatically*.  I wasn't certain if APC was using only RAM or was 
possibly also writing to disk.  Performance tanked so quickly that I set 
it aside as an option and moved on.




Haven't tried these myself, so I would do some quick tests to ensure
if they meet your performance requirements. In theory, it should be
faster than berkeley-db like solutions (which is also another option
but it seems something similar like MongoDB was not good enough?).



I will run more tests against MongoDB.  Initially I tried to use it to 
store everything.  If I only store my indexes, it might fare better. 
Certainly, though, running queries and updates against a remote server 
will always be slower than doing the lookups locally in ram.




I  am curious to know if someone here has run these tests. Note that
with memcached installed locally (on the same box running php), it can
be surprisingly efficient - using pconnect(),  caching the handler in
a static var for a given request cycle etc...


memcached gives no guarantee about data persistence.  I need to have a 
hash table that will contain all the values I set.  They don't need to 
survive a server shutdown (don't need to be written to disk), but I can 
not afford for the server to throw away values that don't fit into 
memory.  If there is a way to configure memcached guarantee storage, 
that might work.


-- Dante



On Sun, Jan 24, 2010 at 9:39 AM, D. Dante Lorenso  wrote:

shiplu wrote:

On Sun, Jan 24, 2010 at 3:11 AM, D. Dante Lorenso 
wrote:

All,

I'm loading millions of records into a backend PHP cli script that I
need to build a hash index from to optimize key lookups for data that
I'm importing into a MySQL database.  The problem is that storing this
data in a PHP array is not very memory efficient and my millions of
records are consuming about 4-6 GB of ram.


What are you storing? An array of row objects??
In that case storing only the row id is will reduce the memory.

I am querying a MySQL database which contains 40 million records and mapping
string columns to numeric ids.  You might consider it normalizing the data.

Then, I am importing a new 40 million records and comparing the new values
to the old values.  Where the value matches, I update records, but where
they do not match, I insert new records, and finally I go back and delete
old records.  So, the net result is that I have a database with 40 million
records that I need to "sync" on a daily basis.


If you are loading full row objects, it will take a lot of memory.
But if you just load the row id values, it will significantly decrease
the memory amount.

For what I am trying to do, I just need to map a string value (32 bytes) to
a bigint value (8 bytes) in a fast-lookup hash.


Besides, You can load row ids in a chunk by chunk basis. if you have
10 millions of rows to process. load 1 rows as a chunk. process
them then load the next chunk.  This will significantly reduce memory
usage.

When importing the fresh 40 million records, I need to compare each record
with 4 different indexes that will map the record to existing other records,
or into a "group_id" that the record also belongs to.  My current solution
uses a trigger in MySQL that will do the lookups inside MySQL, but this is
extremely slow.  Pre-loading the mysql indexes into PHP ram and processing
that was is thousands of times faster.

I just need an efficient way to hold my hash tables in PHP ram.  PHP arrays
are very fast, but like my original post says, they consume way too much
ram.


A good algorithm can solve your problem anytime. ;-)

It takes about 5-10 minutes to build my hash indexes in PHP ram currently
which makes up for the 10,000 x speedup on key lookups that I get later on.
 I just want to not use the whole 6 GB of ram to do this.   I need an
efficient hashing API that supports something like:

   $value = (int) fasthash_get((string) $key);
   $exists = (bool) fasthash_exists((string) $key);
   fasthash_set((string) $key, (int) $value);

Or ... it feels like a "memcached"

Re: [PHP] memory efficient hash table extension? like lchash ...

[J Ravi Menon]

PHP does expose sys V shared-memory apis (shm_* functions):

http://us2.php.net/manual/en/book.sem.php

If you already have apc installed, you could also try:

http://us2.php.net/manual/en/book.apc.php

APC also allows you to store user specific data too (it will be in a
shared memory).

Haven't tried these myself, so I would do some quick tests to ensure
if they meet your performance requirements. In theory, it should be
faster than berkeley-db like solutions (which is also another option
but it seems something similar like MongoDB was not good enough?).

I  am curious to know if someone here has run these tests. Note that
with memcached installed locally (on the same box running php), it can
be surprisingly efficient - using pconnect(),  caching the handler in
a static var for a given request cycle etc...

Ravi






On Sun, Jan 24, 2010 at 9:39 AM, D. Dante Lorenso  wrote:
> shiplu wrote:
>>
>> On Sun, Jan 24, 2010 at 3:11 AM, D. Dante Lorenso 
>> wrote:
>>>
>>> All,
>>>
>>> I'm loading millions of records into a backend PHP cli script that I
>>> need to build a hash index from to optimize key lookups for data that
>>> I'm importing into a MySQL database.  The problem is that storing this
>>> data in a PHP array is not very memory efficient and my millions of
>>> records are consuming about 4-6 GB of ram.
>>>
>>
>> What are you storing? An array of row objects??
>> In that case storing only the row id is will reduce the memory.
>
> I am querying a MySQL database which contains 40 million records and mapping
> string columns to numeric ids.  You might consider it normalizing the data.
>
> Then, I am importing a new 40 million records and comparing the new values
> to the old values.  Where the value matches, I update records, but where
> they do not match, I insert new records, and finally I go back and delete
> old records.  So, the net result is that I have a database with 40 million
> records that I need to "sync" on a daily basis.
>
>> If you are loading full row objects, it will take a lot of memory.
>> But if you just load the row id values, it will significantly decrease
>> the memory amount.
>
> For what I am trying to do, I just need to map a string value (32 bytes) to
> a bigint value (8 bytes) in a fast-lookup hash.
>
>> Besides, You can load row ids in a chunk by chunk basis. if you have
>> 10 millions of rows to process. load 1 rows as a chunk. process
>> them then load the next chunk.  This will significantly reduce memory
>> usage.
>
> When importing the fresh 40 million records, I need to compare each record
> with 4 different indexes that will map the record to existing other records,
> or into a "group_id" that the record also belongs to.  My current solution
> uses a trigger in MySQL that will do the lookups inside MySQL, but this is
> extremely slow.  Pre-loading the mysql indexes into PHP ram and processing
> that was is thousands of times faster.
>
> I just need an efficient way to hold my hash tables in PHP ram.  PHP arrays
> are very fast, but like my original post says, they consume way too much
> ram.
>
>> A good algorithm can solve your problem anytime. ;-)
>
> It takes about 5-10 minutes to build my hash indexes in PHP ram currently
> which makes up for the 10,000 x speedup on key lookups that I get later on.
>  I just want to not use the whole 6 GB of ram to do this.   I need an
> efficient hashing API that supports something like:
>
>        $value = (int) fasthash_get((string) $key);
>        $exists = (bool) fasthash_exists((string) $key);
>        fasthash_set((string) $key, (int) $value);
>
> Or ... it feels like a "memcached" api but where the data is stored locally
> instead of accessed via a network.  So this is how my search led me to what
> appears to be a dead "lchash" extension.
>
> -- Dante
>
> --
> D. Dante Lorenso
> da...@lorenso.com
> 972-333-4139
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory efficient hash table extension? like lchash ...

[D. Dante Lorenso]

shiplu wrote:

On Sun, Jan 24, 2010 at 3:11 AM, D. Dante Lorenso  wrote:

All,

I'm loading millions of records into a backend PHP cli script that I
need to build a hash index from to optimize key lookups for data that
I'm importing into a MySQL database.  The problem is that storing this
data in a PHP array is not very memory efficient and my millions of
records are consuming about 4-6 GB of ram.



What are you storing? An array of row objects??
In that case storing only the row id is will reduce the memory.


I am querying a MySQL database which contains 40 million records and 
mapping string columns to numeric ids.  You might consider it 
normalizing the data.


Then, I am importing a new 40 million records and comparing the new 
values to the old values.  Where the value matches, I update records, 
but where they do not match, I insert new records, and finally I go back 
and delete old records.  So, the net result is that I have a database 
with 40 million records that I need to "sync" on a daily basis.



If you are loading full row objects, it will take a lot of memory.
But if you just load the row id values, it will significantly decrease
the memory amount.


For what I am trying to do, I just need to map a string value (32 bytes) 
to a bigint value (8 bytes) in a fast-lookup hash.



Besides, You can load row ids in a chunk by chunk basis. if you have
10 millions of rows to process. load 1 rows as a chunk. process
them then load the next chunk.  This will significantly reduce memory
usage.


When importing the fresh 40 million records, I need to compare each 
record with 4 different indexes that will map the record to existing 
other records, or into a "group_id" that the record also belongs to.  My 
current solution uses a trigger in MySQL that will do the lookups inside 
MySQL, but this is extremely slow.  Pre-loading the mysql indexes into 
PHP ram and processing that was is thousands of times faster.


I just need an efficient way to hold my hash tables in PHP ram.  PHP 
arrays are very fast, but like my original post says, they consume way 
too much ram.



A good algorithm can solve your problem anytime. ;-)


It takes about 5-10 minutes to build my hash indexes in PHP ram 
currently which makes up for the 10,000 x speedup on key lookups that I 
get later on.  I just want to not use the whole 6 GB of ram to do this. 
   I need an efficient hashing API that supports something like:


$value = (int) fasthash_get((string) $key);
$exists = (bool) fasthash_exists((string) $key);
fasthash_set((string) $key, (int) $value);

Or ... it feels like a "memcached" api but where the data is stored 
locally instead of accessed via a network.  So this is how my search led 
me to what appears to be a dead "lchash" extension.


-- Dante

--
D. Dante Lorenso
da...@lorenso.com
972-333-4139

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] memory efficient hash table extension? like lchash ...

[shiplu]

On Sun, Jan 24, 2010 at 3:11 AM, D. Dante Lorenso  wrote:
> All,
>
> I'm loading millions of records into a backend PHP cli script that I
> need to build a hash index from to optimize key lookups for data that
> I'm importing into a MySQL database.  The problem is that storing this
> data in a PHP array is not very memory efficient and my millions of
> records are consuming about 4-6 GB of ram.
>

What are you storing? An array of row objects??
In that case storing only the row id is will reduce the memory.

If you are loading full row objects, it will take a lot of memory.
But if you just load the row id values, it will significantly decrease
the memory amount.

Besides, You can load row ids in a chunk by chunk basis. if you have
10 millions of rows to process. load 1 rows as a chunk. process
them then load the next chunk.  This will significantly reduce memory
usage.

A good algorithm can solve your problem anytime. ;-)

-- 
Shiplu Mokaddim
My talks, http://talk.cmyweb.net
Follow me, http://twitter.com/shiplu
SUST Programmers, http://groups.google.com/group/p2psust
Innovation distinguishes bet ... ... (ask Steve Jobs the rest)

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] memory efficient hash table extension? like lchash ...

[D. Dante Lorenso]

All,

I'm loading millions of records into a backend PHP cli script that I
need to build a hash index from to optimize key lookups for data that
I'm importing into a MySQL database.  The problem is that storing this
data in a PHP array is not very memory efficient and my millions of
records are consuming about 4-6 GB of ram.

I have tried using some external key/value storage solutions like
MemcacheDB, MongoDB, and straight MySQL, but none of these are fast
enough for what I'm trying to do.

Then I found the "lchash" extension for PHP and it looks like exactly
what I want.  It's a c-lib hash which is accessed from PHP.  Using it
would be slightly slower than using straight PHP arrays, but would be
much more memory efficient since not all data needs to be stored as PHP
zvals, etc.

Problem is that the lchash extension can't ben installed in my PHP 5.3 
build because "pecl install lchash" fails with a message about invalid 
checksum on the README file.  Apparently this extension has been 
neglected and abandoned and hasn't been updated since 2005.


Is there something like lchash that *is* being maintained?  What would 
you all suggest?


-- Dante


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] PHP Memory Management

[Waynn Lue]

>
> Waynn Lue wrote:
>
>> I've been running the script below:
>>
>> >  $appIds = getLotsOfAppIds();
>>  foreach ($appIds as $appId) {
>>echo "$appId\n";
>>//echo memory_get_usage() . "\n";
>>try {
>>  $getBundles = getBundles($appId);
>>  $numBundles = count($registeredBundles);
>>  echo $numBundles . "\n";
>>  continue;
>>}
>>  }
>> ?>
>>
>> And I get PHP Fatal Error: Allowed Memory Size Exhausted after it runs for
>> a
>> bit.  Looking at the memory usage, it's because $getBundles (an array) is
>> huge, and keeps growing.  What I'm confused by is why setting it to
>> something else in the next iteration of the foreach loop doesn't free the
>> previously allocated array, since there shouldn't be any more references
>> to
>> it.  I've worked around it by explicitly calling unset($getBundles), but
>> just wanted to understand why it's working the way it does.
>>
>>
> Aside from on the 1st iteration, each time getBundles() is called, it
> creates an array within its own scope. Until PHP assigns the returned array
> to $getBundles, there are two in memory. Maybe that's the problem.
>
But that means that as soon as you move to the next iteration of the foreach
loop, both arrays should still go away, right?  Instead, it looks like the
memory is continuously increasing on each iteration.


>
> I don't know how you can stand naming structures or variables the same as
> functions, btw. That would drive me nuts. Where is this $registeredBundles
> coming from? Perhaps you meant:
>
> $registeredBundles = getBundles($appId);
>
Ah, my fault.  Yeah, I don't usually name variables the same as functions,
but was coming up with names on the fly as I tried to shrink my script down
to a reproducible test case.

It should have been



and unset($registeredBundles).

Re: [PHP] PHP Memory Management

[Chacha C]

is it possible because you can assign $func = foo and call $func() and it
will call foo(), maybe that its creating an endless loop of assigning the
function to itself?

On Sun, Aug 3, 2008 at 11:17 AM, brian <[EMAIL PROTECTED]> wrote:

> Waynn Lue wrote:
>
>> I've been running the script below:
>>
>> >  $appIds = getLotsOfAppIds();
>>  foreach ($appIds as $appId) {
>>echo "$appId\n";
>>//echo memory_get_usage() . "\n";
>>try {
>>  $getBundles = getBundles($appId);
>>  $numBundles = count($registeredBundles);
>>  echo $numBundles . "\n";
>>  continue;
>>}
>>  }
>> ?>
>>
>> And I get PHP Fatal Error: Allowed Memory Size Exhausted after it runs for
>> a
>> bit.  Looking at the memory usage, it's because $getBundles (an array) is
>> huge, and keeps growing.  What I'm confused by is why setting it to
>> something else in the next iteration of the foreach loop doesn't free the
>> previously allocated array, since there shouldn't be any more references
>> to
>> it.  I've worked around it by explicitly calling unset($getBundles), but
>> just wanted to understand why it's working the way it does.
>>
>>
> Aside from on the 1st iteration, each time getBundles() is called, it
> creates an array within its own scope. Until PHP assigns the returned array
> to $getBundles, there are two in memory. Maybe that's the problem.
>
> I don't know how you can stand naming structures or variables the same as
> functions, btw. That would drive me nuts. Where is this $registeredBundles
> coming from? Perhaps you meant:
>
> $registeredBundles = getBundles($appId);
>
> b
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>


-- 
http://twit.tv
- What I listen to All Week

Re: [PHP] PHP Memory Management

[brian]

Waynn Lue wrote:

I've been running the script below:



And I get PHP Fatal Error: Allowed Memory Size Exhausted after it runs for a
bit.  Looking at the memory usage, it's because $getBundles (an array) is
huge, and keeps growing.  What I'm confused by is why setting it to
something else in the next iteration of the foreach loop doesn't free the
previously allocated array, since there shouldn't be any more references to
it.  I've worked around it by explicitly calling unset($getBundles), but
just wanted to understand why it's working the way it does.



Aside from on the 1st iteration, each time getBundles() is called, it 
creates an array within its own scope. Until PHP assigns the returned 
array to $getBundles, there are two in memory. Maybe that's the problem.


I don't know how you can stand naming structures or variables the same 
as functions, btw. That would drive me nuts. Where is this 
$registeredBundles coming from? Perhaps you meant:


$registeredBundles = getBundles($appId);

b

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] PHP Memory Management

[Waynn Lue]

I've been running the script below:



And I get PHP Fatal Error: Allowed Memory Size Exhausted after it runs for a
bit.  Looking at the memory usage, it's because $getBundles (an array) is
huge, and keeps growing.  What I'm confused by is why setting it to
something else in the next iteration of the foreach loop doesn't free the
previously allocated array, since there shouldn't be any more references to
it.  I've worked around it by explicitly calling unset($getBundles), but
just wanted to understand why it's working the way it does.

Thanks,
Waynn

Re: [PHP] Memory profiling tools

[Chris]

Larry Garfield wrote:
> Hi all.  I have a rather large application on which I need to do some memory 
> performance profiling and optimization.  Basically it's eating up more RAM 
> than it should and I'm not sure why.  I have some suspects, but nothing 
> concrete.
> 
> Are there any (open source) tools that people can recommend for such a task?  
> Or any programming tricks one can recommend to identify the size of a given 
> data structure?  Windows or Linux are both fine; I have access to both.

You could use ticks:

http://www.davedevelopment.co.uk/2008/05/12/log-memory-usage-using-declare-and-ticks-in-php/

though see my comments (#7) about why you don't want to store the info
in php itself :P

-- 
Postgresql & php tutorials
http://www.designmagick.com/

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory profiling tools

[Larry Garfield]

Hi all.  I have a rather large application on which I need to do some memory 
performance profiling and optimization.  Basically it's eating up more RAM 
than it should and I'm not sure why.  I have some suspects, but nothing 
concrete.

Are there any (open source) tools that people can recommend for such a task?  
Or any programming tricks one can recommend to identify the size of a given 
data structure?  Windows or Linux are both fine; I have access to both.

-- 
Larry Garfield
[EMAIL PROTECTED]

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory cache problem

[Stut]

On 14 Jun 2008, at 01:02, tedd wrote:

At 11:47 PM +0100 6/13/08, Stut wrote:

On 13 Jun 2008, at 23:20, R B wrote:

I search in both caches, and the video appears in the memory cache.


Sorry, but that doesn't answer the question of why you/he doesn't  
want it to be cached. If he's trying to protect the videos from  
being copied then you're going to need to make it clear to him that  
you can't stop that from happening without DRM which is a whole  
other kettle of fish and usually not worth it.


Aside from that I'm not aware of anything else you can do to stop  
clients caching your content. At the end of the day you have no  
reliable control over what happens after the data leaves the server.


-Stut


Caching is one thing, but viewing the video is another -- unless I  
am mistaken.


While it's true that you have no control over the video file once it  
leaves the server, however, what the file does in a foreign  
environment is something else.


Typically, DRM schemes are needed when someone wants to protect  
digital media from being copied to play in standard players. But, if  
you want to restrict playing a video to just your site, then that's  
another matter.


One can place actionscript in the video that will stop it from  
playing if certain conditions are not met. For example, you can have  
a file that exits in your site, but no where else. If the video is  
downloaded and an attempt is made to play it, then the video simply  
looks for the companion file and if it's not there, then it won't  
play.


Is this not true?


Indeed, but things like that are easily stripped. In addition to that  
most DRM schemes have been broken. It also tends to create issues for  
some users who may want to view the videos at a time that suits them  
in a place where it's not feasible to stream it from your website. At  
the end of the day it comes down to the website and what limitations  
the users will put up with. Either way it's beyond the scope of this  
list.


-Stut

--
http://stut.net/

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory cache problem

[Nathan Nobbe]

On Fri, Jun 13, 2008 at 8:02 PM, tedd <[EMAIL PROTECTED]> wrote:

> At 11:47 PM +0100 6/13/08, Stut wrote:
>
>> On 13 Jun 2008, at 23:20, R B wrote:
>>
>>> I search in both caches, and the video appears in the memory cache.
>>>
>>
>> Sorry, but that doesn't answer the question of why you/he doesn't want it
>> to be cached. If he's trying to protect the videos from being copied then
>> you're going to need to make it clear to him that you can't stop that from
>> happening without DRM which is a whole other kettle of fish and usually not
>> worth it.
>>
>> Aside from that I'm not aware of anything else you can do to stop clients
>> caching your content. At the end of the day you have no reliable control
>> over what happens after the data leaves the server.
>>
>> -Stut
>>
>
> Caching is one thing, but viewing the video is another -- unless I am
> mistaken.
>
> While it's true that you have no control over the video file once it leaves
> the server, however, what the file does in a foreign environment is
> something else.
>
> Typically, DRM schemes are needed when someone wants to protect digital
> media from being copied to play in standard players. But, if you want to
> restrict playing a video to just your site, then that's another matter.
>
> One can place actionscript in the video that will stop it from playing if
> certain conditions are not met. For example, you can have a file that exits
> in your site, but no where else. If the video is downloaded and an attempt
> is made to play it, then the video simply looks for the companion file and
> if it's not there, then it won't play.
>
> Is this not true?


are you asking a rhetorical question here tedd?  i thought you implemented
something like that w/ help of the list some months back.

-nathan

Re: [PHP] Memory cache problem

[tedd]

At 11:47 PM +0100 6/13/08, Stut wrote:

On 13 Jun 2008, at 23:20, R B wrote:

I search in both caches, and the video appears in the memory cache.


Sorry, but that doesn't answer the question of why you/he doesn't 
want it to be cached. If he's trying to protect the videos from 
being copied then you're going to need to make it clear to him that 
you can't stop that from happening without DRM which is a whole 
other kettle of fish and usually not worth it.


Aside from that I'm not aware of anything else you can do to stop 
clients caching your content. At the end of the day you have no 
reliable control over what happens after the data leaves the server.


-Stut


Caching is one thing, but viewing the video is another -- unless I am mistaken.

While it's true that you have no control over the video file once it 
leaves the server, however, what the file does in a foreign 
environment is something else.


Typically, DRM schemes are needed when someone wants to protect 
digital media from being copied to play in standard players. But, if 
you want to restrict playing a video to just your site, then that's 
another matter.


One can place actionscript in the video that will stop it from 
playing if certain conditions are not met. For example, you can have 
a file that exits in your site, but no where else. If the video is 
downloaded and an attempt is made to play it, then the video simply 
looks for the companion file and if it's not there, then it won't 
play.


Is this not true?

Cheers,

tedd

--
---
http://sperling.com  http://ancientstones.com  http://earthstones.com

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory cache problem

[Stut]

On 13 Jun 2008, at 23:20, R B wrote:
The video manual it´s not for me, it´s for a customer and he don´t  
want to be cached in the client machine...


If you put in firefoxin the url area:   about:cache , firefox  
display the Memory cache and the Disk cache...


I search in both caches, and the video appears in the memory cache.


Sorry, but that doesn't answer the question of why you/he doesn't want  
it to be cached. If he's trying to protect the videos from being  
copied then you're going to need to make it clear to him that you  
can't stop that from happening without DRM which is a whole other  
kettle of fish and usually not worth it.


Aside from that I'm not aware of anything else you can do to stop  
clients caching your content. At the end of the day you have no  
reliable control over what happens after the data leaves the server.


-Stut

--
http://stut.net/


On Fri, Jun 13, 2008 at 4:17 PM, Stut <[EMAIL PROTECTED]> wrote:
On 13 Jun 2008, at 23:12, R B wrote:
I´m making a video manual, but i don´t want to be cached in the client
machine.

I make a script like this:



With Internet explorer the script works fine. But Firefox have two  
types of

cache: Memory cache and disk cache.
With firefox disk cache, the script works fine, because don´t  
appears in the

disk cache, but i have problems with the Memory cache.

How can i avoid to appears in the firefox memory cache?

Why don't you want it cached?

What makes you think it's cached in memory on FF?

-Stut

--
http://stut.net/




--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory cache problem

[R B]

The video manual it´s not for me, it´s for a customer and he don´t want to
be cached in the client machine...

If you put in firefoxin the url area:   about:cache , firefox display the
Memory cache and the Disk cache...

I search in both caches, and the video appears in the memory cache.

On Fri, Jun 13, 2008 at 4:17 PM, Stut <[EMAIL PROTECTED]> wrote:

>  On 13 Jun 2008, at 23:12, R B wrote:
>
>> I´m making a video manual, but i don´t want to be cached in the client
>> machine.
>>
>> I make a script like this:
>>
>> > header("Expires: Mon, 26 Jul 1997 05:00:00 GMT");
>> header("Last-Modified: " . gmdate("D, d M Y H:i:s") . " GMT");
>> header("Cache-Control: no-store, no-cache, must-revalidate");
>> header("Cache-Control: post-check=0, pre-check=0", false);
>> header("Pragma: no-cache");
>>
>> readfile($video_name);
>> ?>
>>
>> With Internet explorer the script works fine. But Firefox have two types
>> of
>> cache: Memory cache and disk cache.
>> With firefox disk cache, the script works fine, because don´t appears in
>> the
>> disk cache, but i have problems with the Memory cache.
>>
>> How can i avoid to appears in the firefox memory cache?
>>
>
> Why don't you want it cached?
>
> What makes you think it's cached in memory on FF?
>
> -Stut
>
> --
> http://stut.net/

Re: [PHP] Memory cache problem

[Stut]

On 13 Jun 2008, at 23:12, R B wrote:

I´m making a video manual, but i don´t want to be cached in the client
machine.

I make a script like this:



With Internet explorer the script works fine. But Firefox have two  
types of

cache: Memory cache and disk cache.
With firefox disk cache, the script works fine, because don´t  
appears in the

disk cache, but i have problems with the Memory cache.

How can i avoid to appears in the firefox memory cache?


Why don't you want it cached?

What makes you think it's cached in memory on FF?

-Stut

--
http://stut.net/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory cache problem

[R B]

Hello,

I´m making a video manual, but i don´t want to be cached in the client
machine.

I make a script like this:



With Internet explorer the script works fine. But Firefox have two types of
cache: Memory cache and disk cache.
With firefox disk cache, the script works fine, because don´t appears in the
disk cache, but i have problems with the Memory cache.

How can i avoid to appears in the firefox memory cache?

RE: [PHP] Command-line PHP memory limit

[Boyd, Todd M.]

> -Original Message-
> From: Shawn McKenzie [mailto:[EMAIL PROTECTED]
> Sent: Thursday, June 12, 2008 11:06 AM
> To: php-general@lists.php.net
> Subject: Re: [PHP] Command-line PHP memory limit
> 
> Per Jessen wrote:
> > Rene Fournier wrote:
> >
> >> Is it possible to set a unique memory limit for PHP scripts that are
> >> run from the command line? (That is, different from what's specified
> >> in php.ini.)
> >
> > This might specific to openSUSE, but the typical installation comes
> with
> > separate php.inis for apache and cli.
> >
> > /etc/php5/cli/php.ini
> > /etc/php5/apache2/php.ini
> >
> >
> > /Per Jessen, Zürich
> >
> 
> K/Ubuntu, I would assume Debian as well.

Correct. I am running Debian etch, and I have two separate php.ini files 
(automatically configured as such)--one for Apache, one for command line.


Todd Boyd
Web Programmer

Re: [PHP] Command-line PHP memory limit

[Shawn McKenzie]

Per Jessen wrote:

Rene Fournier wrote:


Is it possible to set a unique memory limit for PHP scripts that are
run from the command line? (That is, different from what's specified
in php.ini.)


This might specific to openSUSE, but the typical installation comes with
separate php.inis for apache and cli. 


/etc/php5/cli/php.ini
/etc/php5/apache2/php.ini


/Per Jessen, Zürich



K/Ubuntu, I would assume Debian as well.

-Shawn

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Command-line PHP memory limit

[Per Jessen]

Rene Fournier wrote:

> Is it possible to set a unique memory limit for PHP scripts that are
> run from the command line? (That is, different from what's specified
> in php.ini.)

This might specific to openSUSE, but the typical installation comes with
separate php.inis for apache and cli. 

/etc/php5/cli/php.ini
/etc/php5/apache2/php.ini


/Per Jessen, Zürich


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Command-line PHP memory limit

[Robert Cummings]

On Wed, 2008-06-11 at 23:48 +0200, Rene Fournier wrote:
> Is it possible to set a unique memory limit for PHP scripts that are  
> run from the command line? (That is, different from what's specified  
> in php.ini.)

In your script:

ini_set( 'memory_limit', -1 );

Cheers,
Rob.
-- 
http://www.interjinn.com
Application and Templating Framework for PHP


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Command-line PHP memory limit

[Rene Fournier]

Is it possible to set a unique memory limit for PHP scripts that are  
run from the command line? (That is, different from what's specified  
in php.ini.)


...Rene

Re: [PHP] Memory limit of 8 MB not enough

[Andrew Ballard]

On Wed, Apr 16, 2008 at 12:00 PM, David Giragosian
<[EMAIL PROTECTED]> wrote:
>
> On 4/16/08, tedd <[EMAIL PROTECTED]> wrote:
>  >
>  > At 10:53 AM -0400 4/16/08, Andrew Ballard wrote:
>  >
>  > > On Wed, Apr 16, 2008 at 9:59 AM, tedd <[EMAIL PROTECTED]> wrote:
>  > >
>  > >  >  I saw one the other day that caught my eye -- will look into it.
>  > >
>  > > >
>  > > > The first problem I see implementing the approach is the JavaScript
>  > > sandbox. JavaScript is allowed to read the text of the input file
>  > > field (and thus know the file name and path once a file has been
>  > > selected); it cannot access the disk to actually get to the image or
>  > > do anything with it. From here, I'm thinking there are a couple things
>  > > you could do with JavaScript. The first would be to embed the image
>  > > into the web page using an IMG tag, which would allow you to determine
>  > > the pixel dimensions (but not the actual file size). The second would
>  > > be what Gmail seems to do, and actually upload the form to their
>  > > server with some sort of AJAX request, at which time your server could
>  > > return information on the size of the file. However, if the file is
>  > > too large, you're back to the original problem of needing to increase
>  > > the RAM limit in PHP.
>  > >
>  > > The second problem is, as I said, I'm not aware of any JavaScript that
>  > > can manipulate the image. JavaScript can cause the display of an image
>  > > to be resized within the browser, but that doesn't actually affect the
>  > > stored file. I don't know, but I guess if you could actually get to
>  > > the bits, you could probably write an algorithm in JavaScript to
>  > > resize an image by manipulating the bits, but I think it would be
>  > > dreadfully slow if it would even run without running out of resources.
>  > >
>  > > I think the only pure-client options are either Java (which would
>  > > require special permissions to get out of its own sandbox) or ActiveX.
>  > >
>  > > If you know of some other approach, I'd be interested to see how it
>  > > works.
>  > >
>  > > Andrew
>  > >
>  >
>  >
>  > Andrew:
>  >
>  > I think you are right -- I wasn't able to resize the image. Here's the
>  > link that caught my eye, but it doesn't work as described:
>  >
>  > http://javascript.internet.com/forms/image-upload-preview.html
>  >
>  > Cheers,
>  >
>  > tedd
>  > --
>  >
>
>
>  It worked for me using IE on XP.
>
>  David
>

I get nothing in either browser.

FF: only the first header line; no content
HTTP/0.9 200 OK

IE:
 The page cannot be displayed
The page you are looking for is currently unavailable. The Web site
might be experiencing technical difficulties, or you may need to
adjust your browser settings.
[snip]
Cannot find server or DNS Error



Andrew

HTTP/0.9 200 OK

Andrew

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[David Giragosian]

On 4/16/08, tedd <[EMAIL PROTECTED]> wrote:
>
> At 10:53 AM -0400 4/16/08, Andrew Ballard wrote:
>
> > On Wed, Apr 16, 2008 at 9:59 AM, tedd <[EMAIL PROTECTED]> wrote:
> >
> >  >  I saw one the other day that caught my eye -- will look into it.
> >
> > >
> > > The first problem I see implementing the approach is the JavaScript
> > sandbox. JavaScript is allowed to read the text of the input file
> > field (and thus know the file name and path once a file has been
> > selected); it cannot access the disk to actually get to the image or
> > do anything with it. From here, I'm thinking there are a couple things
> > you could do with JavaScript. The first would be to embed the image
> > into the web page using an IMG tag, which would allow you to determine
> > the pixel dimensions (but not the actual file size). The second would
> > be what Gmail seems to do, and actually upload the form to their
> > server with some sort of AJAX request, at which time your server could
> > return information on the size of the file. However, if the file is
> > too large, you're back to the original problem of needing to increase
> > the RAM limit in PHP.
> >
> > The second problem is, as I said, I'm not aware of any JavaScript that
> > can manipulate the image. JavaScript can cause the display of an image
> > to be resized within the browser, but that doesn't actually affect the
> > stored file. I don't know, but I guess if you could actually get to
> > the bits, you could probably write an algorithm in JavaScript to
> > resize an image by manipulating the bits, but I think it would be
> > dreadfully slow if it would even run without running out of resources.
> >
> > I think the only pure-client options are either Java (which would
> > require special permissions to get out of its own sandbox) or ActiveX.
> >
> > If you know of some other approach, I'd be interested to see how it
> > works.
> >
> > Andrew
> >
>
>
> Andrew:
>
> I think you are right -- I wasn't able to resize the image. Here's the
> link that caught my eye, but it doesn't work as described:
>
> http://javascript.internet.com/forms/image-upload-preview.html
>
> Cheers,
>
> tedd
> --
>


It worked for me using IE on XP.

David

Re: [PHP] Memory limit of 8 MB not enough

[Daniel Brown]

On Wed, Apr 16, 2008 at 11:42 AM, Per Jessen <[EMAIL PROTECTED]> wrote:
> Daniel Brown wrote:
>
>  I guess it's a matter of preference - I tend to think that a shared
>  hosting user is best restricted to whatever changes he can do
>  in .htaccess (with php_admin_flag etc.).

I allow overrides in .htaccess, too.  Not sure what your thinking
is on that.

>  Mind you, it's not like I have hundreds of hosting customers, so maybe
>  my horizont is a little limited here.

I don't have hundreds right now either.  I had a company several
years ago, which I've since sold, that had about three hundred
accounts, but any more, I like to keep lower numbers per server.

-- 

Ask me about:
Dedicated servers starting @ $59.99/mo., VPS starting @ $19.99/mo.,
and shared hosting starting @ $2.50/mo.
Unmanaged, managed, and fully-managed!

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[tedd]

At 10:53 AM -0400 4/16/08, Andrew Ballard wrote:

On Wed, Apr 16, 2008 at 9:59 AM, tedd <[EMAIL PROTECTED]> wrote:

 >  I saw one the other day that caught my eye -- will look into it.



The first problem I see implementing the approach is the JavaScript
sandbox. JavaScript is allowed to read the text of the input file
field (and thus know the file name and path once a file has been
selected); it cannot access the disk to actually get to the image or
do anything with it. From here, I'm thinking there are a couple things
you could do with JavaScript. The first would be to embed the image
into the web page using an IMG tag, which would allow you to determine
the pixel dimensions (but not the actual file size). The second would
be what Gmail seems to do, and actually upload the form to their
server with some sort of AJAX request, at which time your server could
return information on the size of the file. However, if the file is
too large, you're back to the original problem of needing to increase
the RAM limit in PHP.

The second problem is, as I said, I'm not aware of any JavaScript that
can manipulate the image. JavaScript can cause the display of an image
to be resized within the browser, but that doesn't actually affect the
stored file. I don't know, but I guess if you could actually get to
the bits, you could probably write an algorithm in JavaScript to
resize an image by manipulating the bits, but I think it would be
dreadfully slow if it would even run without running out of resources.

I think the only pure-client options are either Java (which would
require special permissions to get out of its own sandbox) or ActiveX.

If you know of some other approach, I'd be interested to see how it works.

Andrew



Andrew:

I think you are right -- I wasn't able to resize the image. Here's 
the link that caught my eye, but it doesn't work as described:


http://javascript.internet.com/forms/image-upload-preview.html

Cheers,

tedd
--
---
http://sperling.com  http://ancientstones.com  http://earthstones.com

Re: [PHP] Memory limit of 8 MB not enough

[Per Jessen]

Daniel Brown wrote:

> On Wed, Apr 16, 2008 at 10:36 AM, Per Jessen <[EMAIL PROTECTED]> wrote:
>>
>>  I'm curious - why?  To me php.ini seems to be exactly the kind of
>>  thing you wouldn't the user to fiddle with - in a shared
>>  environment.
> 
> To allow flexibility for the user, and give them the opportunity
> to customize the account to their needs. 

I guess it's a matter of preference - I tend to think that a shared
hosting user is best restricted to whatever changes he can do
in .htaccess (with php_admin_flag etc.).  
Mind you, it's not like I have hundreds of hosting customers, so maybe
my horizont is a little limited here. 


/Per Jessen, Zürich


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Daniel Brown]

On Wed, Apr 16, 2008 at 10:36 AM, Per Jessen <[EMAIL PROTECTED]> wrote:
>
>  I'm curious - why?  To me php.ini seems to be exactly the kind of thing
>  you wouldn't the user to fiddle with - in a shared environment.

To allow flexibility for the user, and give them the opportunity
to customize the account to their needs.  The rest of the monitors and
filters cover the server in case they do anything extremely stupid,
but other than damaging their own things by turning register_globals
on, or running their quotas to the limit allowed by the balancers
(exceeding their allotted memory usage, for example), they can't
damage the rest of the server.

It should also be mentioned that all accounts on my servers are
chroot'd (jailed), with suExec running for each.  It's as if they're
the only one on the server (such as with a VPS), but without root
access, and with more limited resources.

-- 

Ask me about:
Dedicated servers starting @ $59.99/mo., VPS starting @ $19.99/mo.,
and shared hosting starting @ $2.50/mo.
Unmanaged, managed, and fully-managed!

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Andrew Ballard]

On Wed, Apr 16, 2008 at 9:59 AM, tedd <[EMAIL PROTECTED]> wrote:
>
> At 9:41 AM -0400 4/16/08, Andrew Ballard wrote:
>
> > On Wed, Apr 16, 2008 at 7:56 AM, tedd <[EMAIL PROTECTED]> wrote:
> >
> > >  At 6:48 PM +0900 4/16/08, Dave M G wrote:
> > >
> > >  > PHP list,
> > >  >
> > >  > I have a PHP script that resizes an image. It takes just about
> whatever
> > >  size and shrinks and crops it down to 320X240.
> > >  >
> > >  > I've found that these days, it's not uncommon for people to take
> images
> > >  straight off their digital camera, which can be 3000X2000 pixels in
> image
> > >  size, even if the jpeg file containing the image is only 200 KB. When
> > >  working with the image, it gets uncompressed and takes up a lot of
> memory.
> > >  >
> > >
> > >  You might want to look into using javascript to trim the file down to a
> > >  respectable size before uploading.
> > >
> > >  Cheers,
> > >
> > >  tedd
> > >
> >
> > I'd be interested in seeing an example of how you actually plan on
> > doing that, Tedd. JavaScript can't access the file from an input
> > field, and I'm not aware of any JavaScript image handling functions
> > either.
> >
> > Andrew
> >
>
>
>  I saw one the other day that caught my eye -- will look into it.
>
>
>
>  Cheers,
>
>  tedd

The first problem I see implementing the approach is the JavaScript
sandbox. JavaScript is allowed to read the text of the input file
field (and thus know the file name and path once a file has been
selected); it cannot access the disk to actually get to the image or
do anything with it. From here, I'm thinking there are a couple things
you could do with JavaScript. The first would be to embed the image
into the web page using an IMG tag, which would allow you to determine
the pixel dimensions (but not the actual file size). The second would
be what Gmail seems to do, and actually upload the form to their
server with some sort of AJAX request, at which time your server could
return information on the size of the file. However, if the file is
too large, you're back to the original problem of needing to increase
the RAM limit in PHP.

The second problem is, as I said, I'm not aware of any JavaScript that
can manipulate the image. JavaScript can cause the display of an image
to be resized within the browser, but that doesn't actually affect the
stored file. I don't know, but I guess if you could actually get to
the bits, you could probably write an algorithm in JavaScript to
resize an image by manipulating the bits, but I think it would be
dreadfully slow if it would even run without running out of resources.

I think the only pure-client options are either Java (which would
require special permissions to get out of its own sandbox) or ActiveX.

If you know of some other approach, I'd be interested to see how it works.

Andrew

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Per Jessen]

Daniel Brown wrote:

> On Wed, Apr 16, 2008 at 9:57 AM, Per Jessen <[EMAIL PROTECTED]> wrote:
>> Daniel Brown wrote:
>>  >
>>  > And on a shared host, the likelihood of increased billing for
>>  > overuse of memory.
>>
>>  Except a shared hoster would probably not permit anyone to change
>>  php.ini :-)
> 
> I do.  A lot of them do.

I'm curious - why?  To me php.ini seems to be exactly the kind of thing
you wouldn't the user to fiddle with - in a shared environment.  



/Per Jessen, Zürich


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Daniel Brown]

On Wed, Apr 16, 2008 at 9:57 AM, Per Jessen <[EMAIL PROTECTED]> wrote:
> Daniel Brown wrote:
>  >
>  > And on a shared host, the likelihood of increased billing for
>  > overuse of memory.
>
>  Except a shared hoster would probably not permit anyone to change
>  php.ini :-)

I do.  A lot of them do.

-- 

Ask me about:
Dedicated servers starting @ $59.99/mo., VPS starting @ $19.99/mo.,
and shared hosting starting @ $2.50/mo.
Unmanaged, managed, and fully-managed!

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[tedd]

At 9:41 AM -0400 4/16/08, Andrew Ballard wrote:

On Wed, Apr 16, 2008 at 7:56 AM, tedd <[EMAIL PROTECTED]> wrote:

 At 6:48 PM +0900 4/16/08, Dave M G wrote:

 > PHP list,
 >
 > I have a PHP script that resizes an image. It takes just about whatever
 size and shrinks and crops it down to 320X240.
 >
 > I've found that these days, it's not uncommon for people to take images
 straight off their digital camera, which can be 3000X2000 pixels in image
 size, even if the jpeg file containing the image is only 200 KB. When
 working with the image, it gets uncompressed and takes up a lot of memory.
 >

  You might want to look into using javascript to trim the file down to a
 respectable size before uploading.

  Cheers,

  tedd


I'd be interested in seeing an example of how you actually plan on
doing that, Tedd. JavaScript can't access the file from an input
field, and I'm not aware of any JavaScript image handling functions
either.

Andrew



I saw one the other day that caught my eye -- will look into it.

Cheers,

tedd

--
---
http://sperling.com  http://ancientstones.com  http://earthstones.com

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Per Jessen]

Daniel Brown wrote:

> On Wed, Apr 16, 2008 at 6:07 AM, Per Jessen <[EMAIL PROTECTED]> wrote:
>>
>>  > Actually, I have no idea what the potential dangers are.
>>
>>  Using up all available memory is the only real "risk".  It might
>>  lead to swapping which in turn will most likely increase response
>>  times.
> 
> And on a shared host, the likelihood of increased billing for
> overuse of memory. 

Except a shared hoster would probably not permit anyone to change
php.ini :-)


/Per Jessen, Zürich


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Daniel Brown]

On Wed, Apr 16, 2008 at 7:56 AM, tedd <[EMAIL PROTECTED]> wrote:
> At 6:48 PM +0900 4/16/08, Dave M G wrote:
>
> > PHP list,
> >
> > I have a PHP script that resizes an image. It takes just about whatever
> size and shrinks and crops it down to 320X240.
> >
> > I've found that these days, it's not uncommon for people to take images
> straight off their digital camera, which can be 3000X2000 pixels in image
> size, even if the jpeg file containing the image is only 200 KB. When
> working with the image, it gets uncompressed and takes up a lot of memory.
> >
>
>  You might want to look into using javascript to trim the file down to a
> respectable size before uploading.

I've never seen JavaScript's ability to work with image resizing
and such.  Are you suggesting using it instead to say, "that file is
too big," before it's uploaded?  If that's the case, it may not work
in Dave's situation, since he expects large files already, and
therefore has the resizing script(s) in place.

If I'm not totally off-point, I think the memory issue is more of
GD/Imagemagick being a resource hog than the actual file size, since
he pointed out that a megapixel image is sometimes still only 200K.

-- 

Ask me about:
Dedicated servers starting @ $59.99/mo., VPS starting @ $19.99/mo.,
and shared hosting starting @ $2.50/mo.
Unmanaged, managed, and fully-managed!

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Daniel Brown]

On Wed, Apr 16, 2008 at 6:07 AM, Per Jessen <[EMAIL PROTECTED]> wrote:
>
>  > Actually, I have no idea what the potential dangers are.
>
>  Using up all available memory is the only real "risk".  It might lead to
>  swapping which in turn will most likely increase response times.

And on a shared host, the likelihood of increased billing for
overuse of memory.  If it's going to be that resource-intensive, it
may be better to consider moving to a VPS or dedicated server.

I removed my signature from this email so it doesn't look like I'm
stating this to try to sell my own stuff.  ;-P

-- 


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Andrew Ballard]

On Wed, Apr 16, 2008 at 7:56 AM, tedd <[EMAIL PROTECTED]> wrote:
> At 6:48 PM +0900 4/16/08, Dave M G wrote:
>
> > PHP list,
> >
> > I have a PHP script that resizes an image. It takes just about whatever
> size and shrinks and crops it down to 320X240.
> >
> > I've found that these days, it's not uncommon for people to take images
> straight off their digital camera, which can be 3000X2000 pixels in image
> size, even if the jpeg file containing the image is only 200 KB. When
> working with the image, it gets uncompressed and takes up a lot of memory.
> >
>
>  You might want to look into using javascript to trim the file down to a
> respectable size before uploading.
>
>  Cheers,
>
>  tedd

I'd be interested in seeing an example of how you actually plan on
doing that, Tedd. JavaScript can't access the file from an input
field, and I'm not aware of any JavaScript image handling functions
either.

Andrew

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Daniel Brown]

On Wed, Apr 16, 2008 at 5:48 AM, Dave M G <[EMAIL PROTECTED]> wrote:
> PHP list,
>
>  I have a PHP script that resizes an image. It takes just about whatever
> size and shrinks and crops it down to 320X240.
>
[snip!]
>
>  So sometimes I've seen an error in my logs that says:
>
>  Fatal error: Allowed memory size of 8388608 bytes
>  exhausted (tried to allocate 8000 bytes) in
>  /home/public_html/Action.php on line 153.
[snip!]
>
>  What would be a recommended size? While I want to have enough to work with
> large images, I don't want to overload the server.

I set the server-wide setting on most of my servers to 64MB.  This
includes my shared hosting servers, where resources are then monitored
otherwise and then clients who may overuse or abuse resources may be
bumped to a more powerful system (VPS, dedicated, or even just a
higher shared plan).  So I'd recommend using 64MB, but monitoring
traffic as well --- because that 64MB max limit will be for each
execution of each script.  So if you have 100 simultaneous connections
each using the maximum memory allowed by configuration, you're looking
at ~6.4GB memory --- clearly into swap space, and not an optimal
situation.

-- 

Ask me about:
Dedicated servers starting @ $59.99/mo., VPS starting @ $19.99/mo.,
and shared hosting starting @ $2.50/mo.
Unmanaged, managed, and fully-managed!

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[tedd]

At 6:48 PM +0900 4/16/08, Dave M G wrote:

PHP list,

I have a PHP script that resizes an image. It takes just about 
whatever size and shrinks and crops it down to 320X240.


I've found that these days, it's not uncommon for people to take 
images straight off their digital camera, which can be 3000X2000 
pixels in image size, even if the jpeg file containing the image is 
only 200 KB. When working with the image, it gets uncompressed and 
takes up a lot of memory.


You might want to look into using javascript to trim the file down to 
a respectable size before uploading.


Cheers,

tedd

--
---
http://sperling.com  http://ancientstones.com  http://earthstones.com

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory limit of 8 MB not enough

[Per Jessen]

Dave M G wrote:

> I contacted my web host provider, and they recommended increasing the
> allowed allocation limit in "/etc/php.ini".
> 
> Right now my allocation limit, assuming I'm looking at the right
> setting is 8 megabytes:
> 
> memory_limit = 8M  ; Maximum amount of memory a script may consume
> (8MB)
> 
> What would be a recommended size? While I want to have enough to work
> with large images, I don't want to overload the server.

Unless you are constrained by memory in general, I would say 64M.  But
check how much memory you have and how many pictures you expect to
process concurrently. 

> Actually, I have no idea what the potential dangers are.  

Using up all available memory is the only real "risk".  It might lead to
swapping which in turn will most likely increase response times. 


/Per Jessen, Zürich


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory limit of 8 MB not enough

[Dave M G]

PHP list,

I have a PHP script that resizes an image. It takes just about whatever 
size and shrinks and crops it down to 320X240.


I've found that these days, it's not uncommon for people to take images 
straight off their digital camera, which can be 3000X2000 pixels in 
image size, even if the jpeg file containing the image is only 200 KB. 
When working with the image, it gets uncompressed and takes up a lot of 
memory.


So sometimes I've seen an error in my logs that says:

Fatal error: Allowed memory size of 8388608 bytes
exhausted (tried to allocate 8000 bytes) in
/home/public_html/Action.php on line 153.

I contacted my web host provider, and they recommended increasing the 
allowed allocation limit in "/etc/php.ini".


Right now my allocation limit, assuming I'm looking at the right setting 
is 8 megabytes:


memory_limit = 8M  ; Maximum amount of memory a script may consume (8MB)

What would be a recommended size? While I want to have enough to work 
with large images, I don't want to overload the server.


Actually, I have no idea what the potential dangers are. Just on 
principle I think it's a bad idea to edit php.ini without some guidance.


What should I set the memory limit to?

--
Dave M G

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

RES: [PHP] Memory usage very high under AMD64?

[Thiago Pojda]

Try those Internal lists.

You can check them out @ http://www.php.net/mailing-lists.php


Good luck! 

-Mensagem original-
De: Ed W [mailto:[EMAIL PROTECTED] 
Enviada em: quinta-feira, 3 de abril de 2008 07:08
Para: Aschwin Wesselius
Cc: Robert Cummings; php-general@lists.php.net
Assunto: Re: [PHP] Memory usage very high under AMD64?

Aschwin Wesselius wrote:
> Ed W wrote:
>> RSS is staying approximately constant, ie the memory in use has not 
>> changed much
>>
>> VSZ, ie virtual memory has increased by more than 2x2=4.  If someone 
>> has some hard experience of both platforms then please add some 
>> experience to this - however, I'm looking for some hard debugging 
>> knowhow on this, not just some handwaving estimates please
>>
>> Ed W
>
> Maybe still some handwaving, but my colleague forwarded me this link:
>
> http://www.bit-tech.net/bits/2007/10/16/64-bit_more_than_just_the_ram/
> 1
>
> I hope that's still in your interest.
>

Interesting, but I think I have come to the wrong forum?  Where 
do the programmers who implement PHP hang out?  I really need 
to get under the bonnet on this and find out what's happening - 
I'm not after speculation (thanks to everyone who responded though)

Anyone got some hard details on how to debug this further - I'm 
looking for under the hood kind of thoughts please

Ed W


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory usage very high under AMD64?

[Ed W]

Aschwin Wesselius wrote:

Ed W wrote:
RSS is staying approximately constant, ie the memory in use has not 
changed much


VSZ, ie virtual memory has increased by more than 2x2=4.  If someone 
has some hard experience of both platforms then please add some 
experience to this - however, I'm looking for some hard debugging 
knowhow on this, not just some handwaving estimates please


Ed W


Maybe still some handwaving, but my colleague forwarded me this link:

http://www.bit-tech.net/bits/2007/10/16/64-bit_more_than_just_the_ram/1

I hope that's still in your interest.



Interesting, but I think I have come to the wrong forum?  Where do the 
programmers who implement PHP hang out?  I really need to get under the 
bonnet on this and find out what's happening - I'm not after speculation 
(thanks to everyone who responded though)


Anyone got some hard details on how to debug this further - I'm looking 
for under the hood kind of thoughts please


Ed W

Re: [PHP] Memory usage very high under AMD64?

[Aschwin Wesselius]

Ed W wrote:
RSS is staying approximately constant, ie the memory in use has not 
changed much


VSZ, ie virtual memory has increased by more than 2x2=4.  If someone 
has some hard experience of both platforms then please add some 
experience to this - however, I'm looking for some hard debugging 
knowhow on this, not just some handwaving estimates please


Ed W


Maybe still some handwaving, but my colleague forwarded me this link:

http://www.bit-tech.net/bits/2007/10/16/64-bit_more_than_just_the_ram/1

I hope that's still in your interest.


--

Aschwin Wesselius

/'What you would like to be done to you, do that to the other'/

Re: [PHP] Memory usage very high under AMD64?

[Ed W]

Aschwin Wesselius wrote:

Ed W wrote:

45MB x2 is a lot less than 215MB...

Also, I would expect the actual consumption to be less than 2x since 
not all the data will be doubled in size..?


Any other suggestions on how to debug this 5x jump in memory usage?

Thanks

Ed W


Robert Cummings wrote:

64 bit integers are twice as big as 32 bit integers.


Hi all,

I'm not an expert and have no real experience with programming in a 
64-bit environment.


That said, I think a system will allocate not only space for 64-bit 
values (in simple theory twice as much as 32-bit). 64-bit is simple a 
factor, so calculations increase with this factor.


I think the memory goes into a square ratio (NxN) instead of just 
saying double (Nx2). Again, I'm not a wizzkid with enough math 
experience. This is just my simple and humble reasoning.

--


RSS is staying approximately constant, ie the memory in use has not 
changed much


VSZ, ie virtual memory has increased by more than 2x2=4.  If someone has 
some hard experience of both platforms then please add some experience 
to this - however, I'm looking for some hard debugging knowhow on this, 
not just some handwaving estimates please


Ed W

Re: [PHP] Memory usage very high under AMD64?

[Aschwin Wesselius]

Ed W wrote:

45MB x2 is a lot less than 215MB...

Also, I would expect the actual consumption to be less than 2x since 
not all the data will be doubled in size..?


Any other suggestions on how to debug this 5x jump in memory usage?

Thanks

Ed W


Robert Cummings wrote:

64 bit integers are twice as big as 32 bit integers.


Hi all,

I'm not an expert and have no real experience with programming in a 
64-bit environment.


That said, I think a system will allocate not only space for 64-bit 
values (in simple theory twice as much as 32-bit). 64-bit is simple a 
factor, so calculations increase with this factor.


I think the memory goes into a square ratio (NxN) instead of just saying 
double (Nx2). Again, I'm not a wizzkid with enough math experience. This 
is just my simple and humble reasoning.

--

Aschwin Wesselius

/'What you would like to be done to you, do that to the other'/

Re: [PHP] Memory usage very high under AMD64?

[Ed W]

45MB x2 is a lot less than 215MB...

Also, I would expect the actual consumption to be less than 2x since not 
all the data will be doubled in size..?


Any other suggestions on how to debug this 5x jump in memory usage?

Thanks

Ed W


Robert Cummings wrote:

64 bit integers are twice as big as 32 bit integers.

Cheers,
Rob.


On Wed, 2008-04-02 at 20:16 +0100, Ed W wrote:
  

Hi

I am trying to figure out expected memory usage of PHP

Under my 32bit install Apache 2 processes are drawing around 45MB
virtual and 25MB RSS. (XCache enabled)

However, under 64bit, same PHP and Apache versions, FastCGI is consuming
around 110MB virt and 25MB RSS.  However, Apache2 processes are
consuming around 215MB virt and 20MB RSS

Can anyone please help debug why the virtual consumption is so high?

Actually the RSS numbers look fine - it's the vsz numbers that I am
trying to understand why so high?

Thanks

Ed W






--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] Memory usage very high under AMD64?

[Robert Cummings]

64 bit integers are twice as big as 32 bit integers.

Cheers,
Rob.


On Wed, 2008-04-02 at 20:16 +0100, Ed W wrote:
> Hi
> 
> I am trying to figure out expected memory usage of PHP
> 
> Under my 32bit install Apache 2 processes are drawing around 45MB
> virtual and 25MB RSS. (XCache enabled)
> 
> However, under 64bit, same PHP and Apache versions, FastCGI is consuming
> around 110MB virt and 25MB RSS.  However, Apache2 processes are
> consuming around 215MB virt and 20MB RSS
> 
> Can anyone please help debug why the virtual consumption is so high?
> 
> Actually the RSS numbers look fine - it's the vsz numbers that I am
> trying to understand why so high?
> 
> Thanks
> 
> Ed W
> 
> 
-- 
http://www.interjinn.com
Application and Templating Framework for PHP


-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory usage very high under AMD64?

[Ed W]

Hi

I am trying to figure out expected memory usage of PHP

Under my 32bit install Apache 2 processes are drawing around 45MB
virtual and 25MB RSS. (XCache enabled)

However, under 64bit, same PHP and Apache versions, FastCGI is consuming
around 110MB virt and 25MB RSS.  However, Apache2 processes are
consuming around 215MB virt and 20MB RSS

Can anyone please help debug why the virtual consumption is so high?

Actually the RSS numbers look fine - it's the vsz numbers that I am
trying to understand why so high?

Thanks

Ed W


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Memory Leaks

[Nilesh Govindrajan]

I am having 2 memory leaks while calling phpinfo() via FastCGI 
(lighttpd) when session.auto_start is 1. Otherwise there is no memory leak.


Here are the two memory leaks -

[Sat Mar 22 14:07:49 2008]  Script:  '/var/www/lighttpd/test.php'
/root/php-5.2.5/ext/standard/url_scanner_ex.c(305) :  Freeing 0x097BD7E0 
(79 bytes), script=/var/www/lighttpd/test.php

[Sat Mar 22 14:07:50 2008]  Script:  '/var/www/lighttpd/test.php'
/root/php-5.2.5/ext/standard/url_scanner_ex.c(316) :  Freeing 0x097BD85C 
(79 bytes), script=/var/www/lighttpd/test.php


Is this a security threat ?

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] PHP Memory Leak

[Jim Lucas]

Sascha Braun wrote:

Hi Everybody,

I have a couple of foreach loops which are ending in a for loop,
which causes the apache to consume the complete memory of the
server system the php engine is running on.

The nesting level is at round about three and looking like that:

$num_new = 4;
if (is_array($array)) {
foreach ($array as $key => value) {
if ($value['element'] == 'test1') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
} elseif ($value['element'] == 'test2') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
}


I would do a in_array here..

if (in_array($value['element'], array('test1', 'test2') ) ) {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
}



if ($num_new > 0) {
where are you lowering this number?  From what I can tell, it is always 
going to be 4




// this part causes the memory leak

for ($i = 0; $i < $num_new; $i++) {


you have a $i - 0   (a minus sign) not equals


echo "sgasdgga";
}
}   
}
}

I dont know if the above code is causing the memory leak the source
is a little more complex, if nessessary I will provide some more code,

Thank you!


np



I hope for a solution



How about that?

--
Jim Lucas


"Perseverance is not a long race;
it is many short races one after the other"

Walter Elliot



"Some men are born to greatness, some achieve greatness,
and some have greatness thrust upon them."

Twelfth Night, Act II, Scene V
by William Shakespeare

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] PHP Memory Leak

[Casey]

On Dec 6, 2007, at 3:15 PM, Sascha Braun <[EMAIL PROTECTED]>  
wrote:



Hi Everybody,

I have a couple of foreach loops which are ending in a for loop,
which causes the apache to consume the complete memory of the
server system the php engine is running on.

The nesting level is at round about three and looking like that:

$num_new = 4;
if (is_array($array)) {
   foreach ($array as $key => value)

Typo on above line?

{
   if ($value['element'] == 'test1') {
   foreach ($value['data'] as $skey => $svalue) {
   echo $svalue;
   }
   } elseif ($value['element'] == 'test2') {
   foreach ($value['data'] as $skey => $svalue) {
   echo $svalue;
   }
   }
   if ($num_new > 0) {

   // this part causes the memory leak

   for ($i = 0; $i < $num_new; $i++) {
   echo "sgasdgga";
   }
   }
   }
}

I dont know if the above code is causing the memory leak the source
is a little more complex, if nessessary I will provide some more code,

Thank you!

I hope for a solution

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php



--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] PHP Memory Leak

[Chris]

Sascha Braun wrote:

Hi Everybody,

I have a couple of foreach loops which are ending in a for loop,
which causes the apache to consume the complete memory of the
server system the php engine is running on.

The nesting level is at round about three and looking like that:

$num_new = 4;
if (is_array($array)) {
foreach ($array as $key => value) {
if ($value['element'] == 'test1') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
} elseif ($value['element'] == 'test2') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
}
if ($num_new > 0) {

// this part causes the memory leak

for ($i = 0; $i < $num_new; $i++) {
echo "sgasdgga";
}
}   
}
}

I dont know if the above code is causing the memory leak the source
is a little more complex, if nessessary I will provide some more code,


If you don't know how are we supposed to know? :)

Add

memory_get_usage() calls all over the place and see what's going on.

eg:

error_log(__LINE__ . "\t" . memory_get_usage() . "\n", 3, 
'/path/to/log.file');


at various spots and go from there.

--
Postgresql & php tutorials
http://www.designmagick.com/

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] PHP Memory Leak

[Sascha Braun]

Hi Everybody,

I have a couple of foreach loops which are ending in a for loop,
which causes the apache to consume the complete memory of the
server system the php engine is running on.

The nesting level is at round about three and looking like that:

$num_new = 4;
if (is_array($array)) {
foreach ($array as $key => value) {
if ($value['element'] == 'test1') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
} elseif ($value['element'] == 'test2') {
foreach ($value['data'] as $skey => $svalue) {
echo $svalue;
}
}
if ($num_new > 0) {

// this part causes the memory leak

for ($i = 0; $i < $num_new; $i++) {
echo "sgasdgga";
}
}   
}
}

I dont know if the above code is causing the memory leak the source
is a little more complex, if nessessary I will provide some more code,

Thank you!

I hope for a solution

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php