Re: [GENERAL] Working with huge amount of data. RESULTS!

2008-02-12 Thread Oleg Bartunov
On Tue, 12 Feb 2008, Mario Lopez wrote: Hi!, I optimized the LIKE 'keyword%' and LIKE '%keyword' with the following results: # time /Library/PostgreSQL8/bin/psql -U postgres -d testdb -c "select * from table1 where varchar_reverse(data) like varchar_reverse('%keyword');" real0m0.055s u

Re: [GENERAL] Working with huge amount of data. RESULTS!

2008-02-12 Thread hubert depesz lubaczewski
On Tue, Feb 12, 2008 at 03:45:51PM +0100, Mario Lopez wrote: > the reversed index which takes like 20 minutes, I guess it has to do > with the plperl function, perhaps a C function for inverting would make > it up in less time. sure. take a look at this: http://www.depesz.com/index.php/2007/09/0

Re: [GENERAL] Working with huge amount of data. RESULTS!

2008-02-12 Thread Alvaro Herrera
Mario Lopez wrote: > The problem is still with the LIKE '%keyword%', my problem is that I am > not searching for Words in a dictionary fashion, suppose my "data" is > random garbage, that it has common consecutive bytes. How could I > generate a dictionary from this random garbage to make it

Re: [GENERAL] Working with huge amount of data. RESULTS!

2008-02-12 Thread Mario Lopez
Hi!, I optimized the LIKE 'keyword%' and LIKE '%keyword' with the following results: # time /Library/PostgreSQL8/bin/psql -U postgres -d testdb -c "select * from table1 where varchar_reverse(data) like varchar_reverse('%keyword');" real0m0.055s user0m0.011s sys 0m0.006s # time

Re: [GENERAL] Working with huge amount of data.

2008-02-11 Thread Jeff
On Feb 11, 2008, at 10:37 AM, Mario Lopez wrote: SELECT * FROM names WHERE name LIKE ‘keyword%’ If you use the C locale, PG can use an index for this query Or SELECT * FROM names WHERE name LIKE ‘%keyword%’ But not this one - substring searches are painful. However, there is some hope -

Re: [GENERAL] Working with huge amount of data.

2008-02-11 Thread Jonathan Ballet
Mario Lopez wrote: Hi guys :-), I am working on a personal project in which I am trying to make sense on a huge (at least for me) amount of data. I have approximately 150 million rows of unique words (they are not exactly words it is just for explaining the situation). The table I am insertin

Re: [GENERAL] Working with huge amount of data.

2008-02-11 Thread Mario Lopez
Erik, Thanks for your answers, actually this is a workable solution because my data does not get updated so frequently (every 24 hours). The problem is that I would like a more advanced version of this, there must be something I can do, I am going to try what Hubert Despez explained in his a

Re: [GENERAL] Working with huge amount of data.

2008-02-11 Thread Erik Jones
On Feb 11, 2008, at 9:37 AM, Mario Lopez wrote: Hi guys :-), I am working on a personal project in which I am trying to make sense on a huge (at least for me) amount of data. I have approximately 150 million rows of unique words (they are not exactly words it is just for explaining the sit

Re: [GENERAL] Working with huge amount of data.

2008-02-11 Thread Mario Lopez
Hubert, Your two posts look pretty cool :), I would read them tonight and answer you back :) Thanks! On Mon, Feb 11, 2008 at 04:37:24PM +0100, Mario Lopez wrote: SELECT * FROM names WHERE name LIKE ‘keyword%’ Or SELECT * FROM names WHERE name LIKE ‘%keyword%’ check this: http://www

Re: [GENERAL] Working with huge amount of data.

2008-02-11 Thread hubert depesz lubaczewski
On Mon, Feb 11, 2008 at 04:37:24PM +0100, Mario Lopez wrote: > SELECT * FROM names WHERE name LIKE ‘keyword%’ > Or > SELECT * FROM names WHERE name LIKE ‘%keyword%’ check this: http://www.depesz.com/index.php/2007/07/30/indexable-field-like-something/ and this: http://www.depesz.com/index.php/2007

Re: [GENERAL] Working with huge amount of data.

2008-02-11 Thread Vivek Khera
On Feb 11, 2008, at 10:37 AM, Mario Lopez wrote: The problem arises with the second type of queries, where there are no possible partitions and that the search keywords are not known, I have tried making indexes on the letter it ends with, or indexes that specify that it contains the lette

[GENERAL] Working with huge amount of data.

2008-02-11 Thread Mario Lopez
Hi guys :-), I am working on a personal project in which I am trying to make sense on a huge (at least for me) amount of data. I have approximately 150 million rows of unique words (they are not exactly words it is just for explaining the situation). The table I am inserting this is a quite si