Hi,
in a mod_perl package I load a CSV file on apache startup into a simple
hash as read-only class data to be shared by all childs.
A loading routine reads the file line by line and uses one numeric field
as hash entry (error checks etc. omitted):
package Data;
my $class_data = {};
ReadFile ( 'data.txt', $class_data, 4 );
sub ReadFile
{
my $filename = shift; # path string
my $data = shift; # ref to hash
my $ndx_field = shift; # field number
my ( @record, $num_key );
local $_;
open ( INFILE, "<$filename" );
while ( <INFILE> )
{
chomp;
@record = split "\t";
$num_key = $record[$ndx_field];
$data->{$num_key} = [ @record ];
}
close ( INFILE );
}
sub new...
creates an object for searching the data, last result, errors etc.
sub find...
method with something like:
if exists $class_data->{$key} return...
etc.
Now I'm scared about the memory consumption:
The CSV file has 14.000 records with 18 fields and a size of 2 MB
(approx. 150 Bytes per record).
Omitting the loading, top shows, that each httpd instance has 10 MB (all
shared as it should be).
But reading in the file explodes the memory to 36 MB (ok, shared as
well)!
So, how comes, that 2 MB data need 26 MB of memory, if it is stored as a
hash?
Reading perldebguts.pod I did not expect such an increasing:
Description (avg.) CSV Perl
4 string fields (4 chars) 16 bytes (32 bytes) 128 bytes
9 float fields (5 chars) 45 bytes (24 bytes) 216 bytes
5 string fields (rest) 89 bytes (32 bytes) 160 bytes
the integer key (20 bytes) 20 bytes
150 bytes 524 bytes
That will give 14.000 x 524 = approx. 7 MB, but not 26 MB !?
Lost in space...
Ernest
P.S.:
Apache/1.3.19 (Unix) (SuSE/Linux) mod_perl/1.25
This is perl, v5.6.0 built for i586-linux
--
*********************************************************************
* VIRTUALITAS Inc. * *
* * *
* European Consultant Office * http://www.virtualitas.net *
* Internationales Handelszentrum * contact:Ernest Lergon *
* Friedrichstraße 95 * mailto:[EMAIL PROTECTED] *
* 10117 Berlin / Germany * ums:+49180528132130266 *
*********************************************************************
PGP-Key http://www.virtualitas.net/Ernest_Lergon.asc
