On 10/23/13 9:05 PM, Alvaro Herrera wrote:
Gordon Mohr wrote:

Thanks for this! I decided to give the patch a try at the bleeding
edge with some high-dimensional vectors, specifically the 1.4
million 1000-dimensional Freebase entity vectors from the Google
'word2vec' project:

https://code.google.com/p/word2vec/#Pre-trained_entity_vectors_with_Freebase_naming

Unfortunately, here's what I found:

I wonder if these results would improve with this patch:
http://www.postgresql.org/message-id/efedc2bf-ab35-4e2c-911f-fc88da647...@gmail.com

Thanks for the pointer; I'd missed that relevant update from Stas Kelvich. I applied that patch, and reindexed.

On the 100-dimension, 850K vector set:

indexing:  1137s (vs. 1344s)
DATA size: 4.7G (vs 5.0G)
top-11-nearest-neighbor query: 32s (vs ~57s)

On the 500-dimension, 100K vector set:

indexing: 756s (vs. 977s)
DATA size: 4.5G (vs. 4.8G)
top-11-nearest-neighbor query: 18s (vs ~46s)

So, moderate (5-20%) improvements in indexing time and size, and larger (40-60%) speedups in index-assisted (<->) queries... but those index-assisted queries are still ~10X+ slower than the sequence-scan (distance_euclid()) queries, so the existence of the knn-GIST index is still harming rather than hurting performance.

Will update if my understanding changes; still interested to hear if I've missed a key factor/switch needed for these indexes to work well.

- Gordon Mohr



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply via email to