Re: Data structure question
"Drew" == Drew Taylor [EMAIL PROTECTED] writes:
Drew I would like to return a single data structure, but order IS
Drew important (hence the current setup). I was thinking of using
Drew an array, where each element is a hash reference. So I would
Drew return something like this:
In this case pseudohashes are absolutely what you're looking for.
They'll also have the smallest impact on your code as you can walk
@{$ref}[1..foo] when you need the items in order and grab $ref-{key}
when you need a particular value. Just remember that $ref-[0] is
special.
--
Stephen
"Farcical aquatic ceremonies are no basis for a system of government!"
Re: Data structure question
On 8 Jun 2000, Stephen Zander wrote:
"Drew" == Drew Taylor [EMAIL PROTECTED] writes:
Drew I would like to return a single data structure, but order IS
Drew important (hence the current setup). I was thinking of using
Drew an array, where each element is a hash reference. So I would
Drew return something like this:
In this case pseudohashes are absolutely what you're looking for.
They'll also have the smallest impact on your code as you can walk
@{$ref}[1..foo] when you need the items in order and grab $ref-{key}
when you need a particular value. Just remember that $ref-[0] is
special.
Ugh. Psuedo hash references are badly broken even in 5.6. Anyone who's
done extensive work with them (or tried to) can tell you that.
Instead, write a class for your objects, and use arrays internally. Define
constants for the indexes of the arrays.
--
Matt/
Fastnet Software Ltd. High Performance Web Specialists
Providing mod_perl, XML, Sybase and Oracle solutions
Email for training and consultancy availability.
http://sergeant.org http://xml.sergeant.org
Re: Data structure question
On Tue, 6 Jun 2000, Drew Taylor wrote: I have a question about data structures. Currently, I am doing SQL queries and returning an array ref and a hash ref. The array is to preserve order, and the hash contains various bits of data about that not to be dense, but can't you just issue an "order by" clause as part of your SQL statement? ky
Re: Data structure question
Using tied hashes, you could conceivably make your own ordered hash class
and use that as the data structure you return. You'd still basically have
two data structures (for performance) but the fact that it is two data
structures would be hidden behind the tied hash which would be programmed
to iterate the keys using the array rather than the keys function on the
hash part.
I think there is source code for this publicly available, but I forget
where I saw it. You can get some docs from perldoc perltie though.
At 12:39 PM 6/6/00 -0400, Drew Taylor wrote:
Hello,
This doesn't directly relate to mod_perl, but I'd like to make this as
memory efficient as possible since it runs under mod_perl. :-)
I have a question about data structures. Currently, I am doing SQL
queries and returning an array ref and a hash ref. The array is to
preserve order, and the hash contains various bits of data about that
particular product ( it could be another hash ref, and often is). After
getting the two references, I use foreach to loop through the array, and
within that loop I access various data from the hash where the productID
is the key. It looks like this:
Common.pm:
sub getdata {
my $AR = [123, 456, 243, ... ]
my $HR = { 123 = {foo=bar, name='name', price='123'}, ... }
return ($AR, $HR);
}
Otherstuff.pm:
my ($AR, $HR) = $self-getdata();
foreach (@{$AR}) {
my $name = $HR-{$_}{name};
...
}
I would like to return a single data structure, but order IS important
(hence the current setup). I was thinking of using an array, where each
element is a hash reference. So I would return something like this:
[ {ID=123, name='name123', foor='bar'}, {ID=321, name='name321',
bar='foo'}, ... ]
Are there any de-referenceing issues (performance wise) that would make
this less efficient than the 2 structures? TIA for any pointers.
--
Drew Taylor
Vialogix Communications, Inc.
501 N. College Street
Charlotte, NC 28202
704 370 0550
http://www.vialogix.com/
Re: Data structure question
Using tied hashes, you could conceivably make your own ordered hash class and use that as the data structure you return. You'd still basically have two data structures (for performance) but the fact that it is two data structures would be hidden behind the tied hash which would be programmed to iterate the keys using the array rather than the keys function on the hash part. I think there is source code for this publicly available, but I forget where I saw it. You can get some docs from perldoc perltie though. Tie::IxHash -- Eric
Re: Data structure question
At 12:39 PM 6/6/00 -0400, Drew Taylor wrote:
Hello,
This doesn't directly relate to mod_perl, but I'd like to make this as
memory efficient as possible since it runs under mod_perl. :-)
I have a question about data structures. Currently, I am doing SQL
queries and returning an array ref and a hash ref. The array is to
preserve order, and the hash contains various bits of data about that
particular product ( it could be another hash ref, and often is). After
getting the two references, I use foreach to loop through the array, and
within that loop I access various data from the hash where the productID
is the key. It looks like this:
Common.pm:
sub getdata {
my $AR = [123, 456, 243, ... ]
my $HR = { 123 = {foo=bar, name='name', price='123'}, ... }
return ($AR, $HR);
}
Otherstuff.pm:
my ($AR, $HR) = $self-getdata();
foreach (@{$AR}) {
my $name = $HR-{$_}{name};
...
}
I would like to return a single data structure, but order IS important
(hence the current setup). I was thinking of using an array, where each
element is a hash reference. So I would return something like this:
[ {ID=123, name='name123', foor='bar'}, {ID=321, name='name321',
bar='foo'}, ... ]
Well, if the keys are unique, you could just return a hashref, and then
access it using sorted keys:
foreach( sort keys %$HR ) {
## insert useful stuff here
}
Are there any de-referenceing issues (performance wise) that would make
this less efficient than the 2 structures? TIA for any pointers.
Probably not, except your method takes more mems, since you're returning an
extra array. 'Course, the sort takes mems as well, but not as much as the
extra array.
And, there is the overhead of sorting the keys.
I think an array of hashref's is probably the best bet. Then you can use
the DBs sort, and just build the array on the fly, once.
For the site I'm working on, I return a reference to a ResultSet object
which through the next() method returns the next row in the result set:
my $account = $dbs-get_account( "123456789" );
my $rs = $account-get_cards;
while( my $unit = $rs-next ) {
# do something
}
so this enforces the order (due to the order by in the SQL query). This is
a bit slower, since next() returns allocated objects, but it works.
Cheers!
-klm.
---
Ken Miller, Consultant
Shetland Software Services Inc.
Re: Data structure question
On Tue, 6 Jun 2000, Eric Cholet wrote: Using tied hashes, you could conceivably make your own ordered hash class and use that as the data structure you return. You'd still basically have two data structures (for performance) but the fact that it is two data structures would be hidden behind the tied hash which would be programmed to iterate the keys using the array rather than the keys function on the hash part. I think there is source code for this publicly available, but I forget where I saw it. You can get some docs from perldoc perltie though. Tie::IxHash and in perl5.6 it's called pseudohash (well it was known before but is supported in 5.6) http://www.perl.com/pub/doc/manual/html/pod/perldelta.html#Pseudo_hashes_are_supported also take a look at this: Building a Better Hash http://www.dfan.org/real/tpj_hash.html _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: Data structure question
"Ken Y. Clark" wrote: On Tue, 6 Jun 2000, Drew Taylor wrote: I have a question about data structures. Currently, I am doing SQL queries and returning an array ref and a hash ref. The array is to preserve order, and the hash contains various bits of data about that not to be dense, but can't you just issue an "order by" clause as part of your SQL statement? I do that already. But hashes don't preserve order on their own, otherwise I'd just use that alone. The array keeps things in the proper order, and the hash stores data about each individual record. -- Drew Taylor Vialogix Communications, Inc. 501 N. College Street Charlotte, NC 28202 704 370 0550 http://www.vialogix.com/
Re: Data structure question
Gunther Birznieks wrote:
Using tied hashes, you could conceivably make your own ordered hash class
and use that as the data structure you return. You'd still basically have
two data structures (for performance) but the fact that it is two data
structures would be hidden behind the tied hash which would be programmed
to iterate the keys using the array rather than the keys function on the
hash part.
I think there is source code for this publicly available, but I forget
where I saw it. You can get some docs from perldoc perltie though.
I know about tied hashes - Thanks Damien for your excellent book! - but
there is a performance penalty. How big is this penalty? Is it worth
using tied hashes? Versus an array of hash refs?
At 12:39 PM 6/6/00 -0400, Drew Taylor wrote:
Hello,
This doesn't directly relate to mod_perl, but I'd like to make this as
memory efficient as possible since it runs under mod_perl. :-)
I have a question about data structures. Currently, I am doing SQL
queries and returning an array ref and a hash ref. The array is to
preserve order, and the hash contains various bits of data about that
particular product ( it could be another hash ref, and often is). After
getting the two references, I use foreach to loop through the array, and
within that loop I access various data from the hash where the productID
is the key. It looks like this:
Common.pm:
sub getdata {
my $AR = [123, 456, 243, ... ]
my $HR = { 123 = {foo=bar, name='name', price='123'}, ... }
return ($AR, $HR);
}
Otherstuff.pm:
my ($AR, $HR) = $self-getdata();
foreach (@{$AR}) {
my $name = $HR-{$_}{name};
...
}
I would like to return a single data structure, but order IS important
(hence the current setup). I was thinking of using an array, where each
element is a hash reference. So I would return something like this:
[ {ID=123, name='name123', foor='bar'}, {ID=321, name='name321',
bar='foo'}, ... ]
Are there any de-referenceing issues (performance wise) that would make
this less efficient than the 2 structures? TIA for any pointers.
--
Drew Taylor
Vialogix Communications, Inc.
501 N. College Street
Charlotte, NC 28202
704 370 0550
http://www.vialogix.com/
--
Drew Taylor
Vialogix Communications, Inc.
501 N. College Street
Charlotte, NC 28202
704 370 0550
http://www.vialogix.com/
Re: Data structure question
Eric Cholet wrote: Using tied hashes, you could conceivably make your own ordered hash class and use that as the data structure you return. You'd still basically have two data structures (for performance) but the fact that it is two data structures would be hidden behind the tied hash which would be programmed to iterate the keys using the array rather than the keys function on the hash part. I think there is source code for this publicly available, but I forget where I saw it. You can get some docs from perldoc perltie though. Tie::IxHash How much overhead does this module impose? I've heard about it in my readings, but never looked into it very much. Does anyone have experience using Tie::IxHash? -- Drew Taylor Vialogix Communications, Inc. 501 N. College Street Charlotte, NC 28202 704 370 0550 http://www.vialogix.com/
Re: Data structure question
On Tue, 6 Jun 2000, Drew Taylor wrote: "Ken Y. Clark" wrote: On Tue, 6 Jun 2000, Drew Taylor wrote: I have a question about data structures. Currently, I am doing SQL queries and returning an array ref and a hash ref. The array is to preserve order, and the hash contains various bits of data about that not to be dense, but can't you just issue an "order by" clause as part of your SQL statement? I do that already. But hashes don't preserve order on their own, otherwise I'd just use that alone. The array keeps things in the proper order, and the hash stores data about each individual record. doh, but if you fetch the data using ORDER BY and then read in one hashref at a time you are getting an ordered array, not a hash ... this is way OT now ... -- Drew Taylor Vialogix Communications, Inc. 501 N. College Street Charlotte, NC 28202 704 370 0550 http://www.vialogix.com/ - nick
Re: Data structure question
Ken Miller wrote:
Well, if the keys are unique, you could just return a hashref, and then
access it using sorted keys:
foreach( sort keys %$HR ) {
## insert useful stuff here
}
If only I could just use sort. :-) The order could be completely
arbitrary, based on search parameters, individual rankings, etc.
Are there any de-referenceing issues (performance wise) that would make
this less efficient than the 2 structures? TIA for any pointers.
My guess was that whatever overhead there was with de-referencing, it
would more than make up for it in the memory usage. And since I'm
running mod_perl with perl's malloc(), the extra memory doesn't get
returned until the child exists.
Probably not, except your method takes more mems, since you're returning an
extra array. 'Course, the sort takes mems as well, but not as much as the
extra array.
And, there is the overhead of sorting the keys.
I think an array of hashref's is probably the best bet. Then you can use
the DBs sort, and just build the array on the fly, once.
For the site I'm working on, I return a reference to a ResultSet object
which through the next() method returns the next row in the result set:
That is a very neat idea. From a logical point of view, I like it.
However, in my case that would be unnecessary overkill. I'll file it
away for future use. :-)
--
Drew Taylor
Vialogix Communications, Inc.
501 N. College Street
Charlotte, NC 28202
704 370 0550
http://www.vialogix.com/
RE: Data structure question
you can use sort, of the values are hashes or indexes:
foreach ( sort { $a-{name} cmp $b-{name} keys %hash )
or
foreach ( sort { $a-[0] cmp $b-[0] keys %hash )
o _
/|/ | Jerrad Pierce \ | __|_ _|
/||/ http://pthbb.org . | _| |
\|| _.-~-._.-~-._.-~-._@" _|\_|___|___|
-Original Message-
From: Drew Taylor [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, June 06, 2000 14:56
To: Ken Miller
Cc: modperl
Subject: Re: Data structure question
Ken Miller wrote:
Well, if the keys are unique, you could just return a
hashref, and then
access it using sorted keys:
foreach( sort keys %$HR ) {
## insert useful stuff here
}
If only I could just use sort. :-) The order could be completely
arbitrary, based on search parameters, individual rankings, etc.
Are there any de-referenceing issues (performance wise)
that would make
this less efficient than the 2 structures? TIA for any pointers.
My guess was that whatever overhead there was with de-referencing, it
would more than make up for it in the memory usage. And since I'm
running mod_perl with perl's malloc(), the extra memory doesn't get
returned until the child exists.
Probably not, except your method takes more mems, since
you're returning an
extra array. 'Course, the sort takes mems as well, but not
as much as the
extra array.
And, there is the overhead of sorting the keys.
I think an array of hashref's is probably the best bet.
Then you can use
the DBs sort, and just build the array on the fly, once.
For the site I'm working on, I return a reference to a
ResultSet object
which through the next() method returns the next row in the
result set:
That is a very neat idea. From a logical point of view, I like it.
However, in my case that would be unnecessary overkill. I'll file it
away for future use. :-)
--
Drew Taylor
Vialogix Communications, Inc.
501 N. College Street
Charlotte, NC 28202
704 370 0550
http://www.vialogix.com/
Re: Data structure question
Stas Bekman wrote: and in perl5.6 it's called pseudohash (well it was known before but is supported in 5.6) http://www.perl.com/pub/doc/manual/html/pod/perldelta.html#Pseudo_hashes_are_supported I know about pseudohashes - thanks to Damien again! :-). They look very cool, but to be honest I'm afraid to implement them yet until I've had time to play with them more. also take a look at this: Building a Better Hash http://www.dfan.org/real/tpj_hash.html I'll definately take a look at this article. -- Drew Taylor Vialogix Communications, Inc. 501 N. College Street Charlotte, NC 28202 704 370 0550 http://www.vialogix.com/
Re: Data structure question
Jerrad Pierce wrote:
you can use sort, of the values are hashes or indexes:
foreach ( sort { $a-{name} cmp $b-{name} keys %hash )
or
foreach ( sort { $a-[0] cmp $b-[0] keys %hash )
In this case I can't use sort since the order is completely arbitrary,
based on the SQL issued. Hence the need for the array. ;-)
--
Drew Taylor
Vialogix Communications, Inc.
501 N. College Street
Charlotte, NC 28202
704 370 0550
http://www.vialogix.com/
Re: Data structure question
On Tue, 6 Jun 2000, Drew Taylor wrote: I know about tied hashes - Thanks Damien for your excellent book! - but there is a performance penalty. How big is this penalty? Is it worth using tied hashes? Versus an array of hash refs? They're a lot slower than normal data structures, or even normal object methods. Whether that slowness will be noticeable next to the slowness of accessing a database is questionable. There were a few benchmarks posted to this list that you could dig out of the archive. - Perrin
Re: Data structure question
On Tue, 6 Jun 2000, Perrin Harkins wrote: On Tue, 6 Jun 2000, Drew Taylor wrote: I know about tied hashes - Thanks Damien for your excellent book! - but there is a performance penalty. How big is this penalty? Is it worth using tied hashes? Versus an array of hash refs? They're a lot slower than normal data structures, or even normal object methods. Whether that slowness will be noticeable next to the slowness of accessing a database is questionable. There were a few benchmarks posted to this list that you could dig out of the archive. If you are going to run it with apache benchmarks try a fresh version of Apache::Benchmark http://stason.org/works/modules/Apache-Benchmark-0.01.tar.gz but actually there is no reason, Benchmark is perfect for that... I have posted a few examples, so you can roll your own benchmark. perldoc Benchmark will be no less helpful ... just plugged this note about Apache::Benchmark so you'd go grab and try the package before I release it... _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: Data structure question
Hi Drew,
How about writing a custom sort routine, based on the order you would be
using in the array, and returning that as a code ref? Sorting the hash
would be as simple as:
Common.pm:
sub getdata {
my $CR = sub { # generate code ref here };
my $HR = { 'sort_sub' = $CR, 123 = {foo=bar, name='name', price='123'}};
return ($HR);
}
Otherstuff.pm:
my ($HR) = $self-getdata();
foreach (sort {$HR-{'sort_sub'}} grep !/sort_sub/ keys %{$HR}) {
my $name = $HR-{$_}{name};
...
}
I'm assuming here that $AR's ordering may change based on a db select
or something similar; if it doesn't, then write it as a regular subroutine.
If sorting that way is really not an option, then just make $AR an array ref
as you had it, make it an element of $HR called 'sort', and change the foreach
Otherstuff.pm to read something like:
foreach (grep !/sort/ @{$HR-{'sort'}}) {
# la la la
}
darren
--
One man's "magic" is another man's engineering. "Supernatural" is a null word.
-- Robert Heinlein
Re: Data structure question
darren chamberlain wrote: Hi Drew, How about writing a custom sort routine, based on the order you would be using in the array, and returning that as a code ref? Sorting the hash would be as simple as: In this case, it's overkill: the DB has already put the data together in the order I need it. I just have to transfer that structure/order to the rest of my code. -- Drew Taylor Vialogix Communications, Inc. 501 N. College Street Charlotte, NC 28202 704 370 0550 http://www.vialogix.com/
Re: Data structure question
Stas Bekman wrote: On Tue, 6 Jun 2000, Perrin Harkins wrote: On Tue, 6 Jun 2000, Drew Taylor wrote: I know about tied hashes - Thanks Damien for your excellent book! - but there is a performance penalty. How big is this penalty? Is it worth using tied hashes? Versus an array of hash refs? They're a lot slower than normal data structures, or even normal object methods. Whether that slowness will be noticeable next to the slowness of accessing a database is questionable. There were a few benchmarks posted to this list that you could dig out of the archive. I knew they were slower. I'll look for the benchmarks in the archives. Unless I find something really cool that justifies tie()ing, I'm just going to go withmy original idea. The DB already gives me the order I want - I just need to transfer it. But I really like the ideas behind tie'ing things. You can do some really neat stuff behind the scenes. :-) If you are going to run it with apache benchmarks try a fresh version of Apache::Benchmark http://stason.org/works/modules/Apache-Benchmark-0.01.tar.gz ... just plugged this note about Apache::Benchmark so you'd go grab and try the package before I release it... Nothing like a good plug. ;-) -- Drew Taylor Vialogix Communications, Inc. 501 N. College Street Charlotte, NC 28202 704 370 0550 http://www.vialogix.com/
