Hi Matthew
That's a pretty neat setup - didn't realise splitting things out
provides such a good increase in speed. As for your Sphinx syntax,
using local instead of agent is correct, because the indexes are
local, not remote.
As for pulling this into Sphinx, I've been working on some changes to
allow multiple indexes per model, so with some metaprogramming, it'd
be easy to create four indexes, filtering by a quarter of the results
each time. Hopefully these changes will go live within the next week
or so.
Cheers
--
Pat
On 20/11/2009, at 9:44 AM, mixonic wrote:
> Hi Pat, friendly folk,
>
> I've got a 375_000 row geo data set indexed by lat and lon. I can
> search just great, but queries usually take about 650ms on our
> production HW. I found there is a huge speed boost using the main
> index to walk a split set of sources and indexes. So instead of:
>
> index datapoint
> {
> type = distributed
> local = datapoint_core
> }
>
> I have:
>
> index datapoint
> {
> type = distributed
> local = datapoint_core_0
> local = datapoint_core_1
> local = datapoint_core_2
> local = datapoint_core_3
> }
>
> Each source then has a range by IDs:
>
> source datapoint_core_0
>
> sql_query_range = SELECT IFNULL(MIN(`id`), 1), IFNULL(MAX(`id`), 1)
> FROM `restaurant_inspection_datapoints` WHERE `id` BETWEEN 0 AND 93878
>
> Where 93878 is 1/4 of the records. Each index covers 1/4 of the total
> IDs. On my laptop alone, this gave me massive gains...queries that
> could take a second tool 300ms.
>
> I would love to get this into riddle or thinking_sphinx, but the
> riddle configuration code is really complex. Is anyone interested in
> working with me on this or giving me a starting point?
>
> Pat, does that use of "local" make sense? All the other examples out
> there use agents.
>
> See:
>
> http://www.sphinxsearch.com/docs/current.html#distributed
> http://www.sphinxsearch.com/bugs/view.php?id=407 <-- I didn't
> encounter that bug
> http://blog.wasimasif.com/sphinx-distributed-searching/
>
> Thanks all, I'm very excited to get this up!
>
> --
> Matthew Beale :: 607 227 0871
> Resume & Portfolio @ http://madhatted.com
>
> --
>
> You received this message because you are subscribed to the Google
> Groups "Thinking Sphinx" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected]
> .
> For more options, visit this group at
> http://groups.google.com/group/thinking-sphinx?hl=
> .
>
>
--
You received this message because you are subscribed to the Google Groups
"Thinking Sphinx" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/thinking-sphinx?hl=en.
