Does the VM have enough RAM? I'm running out of suggestions for the cause to be honest. And I don't have any similar sized datasets on hand to compare against.
-- Pat On 21/12/2010, at 3:56 AM, Simon wrote: > I'm running it on a Ubuntu VM. The speed does improve a bit when I > remove body, but the number of Mhits that get sorted when I do drops > from just over 200 to 1.7. > > On Dec 20, 1:37 am, Pat Allan <[email protected]> wrote: >> Ah, I just wanted to confirm whether there were joins or not. If not, then >> this definitely feels too slow. What machine are you running this on? And >> does the speed improve if you remove body from the index definition? >> >> -- >> Pat >> >> On 20/12/2010, at 12:38 AM, Simon wrote: >> >>> Hi, >> >>> Thanks for the reply. Not sure what you mean about the columns . >>> They are columns containing ids for other tables, that I am using to >>> limit my actual search queries. >> >>> Changing the sql_range_step has not seemed to make any noticeable >>> difference in the amount of time it takes. I have tried at the >>> default value, in 100,000 blocks, and with the huge value to try and >>> get all the values at once. I thought it seemed pretty slow too, >>> considering there are no joins or anything like that happening. >> >>> Thanks again, >> >>> Simon >> >>> On Dec 19, 2:33 am, Pat Allan <[email protected]> wrote: >>>> Hi Simon >> >>>> Is x_id and y_id the actual columns you're referencing? If not, can you >>>> provide exactly what your define_index block looks like? It will give me a >>>> better picture of whether your indexing is slow or not. >> >>>> Has changing the sql_range_step value made any difference? What happens if >>>> you put it back to the default of 1000? 30 minutes for 800,000 values does >>>> sound slow, for what appears to be quite a simple index definition. >> >>>> Cheers >> >>>> -- >>>> Pat >> >>>> On 18/12/2010, at 6:50 AM, Simon wrote: >> >>>>> Hi there, >> >>>>> I have a table I'm indexing that has roughly 800,000 rows. From >>>>> reading around online and in this group I feel like it's taking a long >>>>> time for my index to get generated. >> >>>>> I have the following in my model: >>>>> define_index do >>>>> indexes title, body >>>>> has x_id >>>>> has locked >>>>> has created_at, y_id >> >>>>> set_property :delta => true >>>>> end >> >>>>> In my sphinx.yml file, I have the following: >>>>> max_matches: 1000 >>>>> html_strip: 1 >>>>> sql_range_step: 10000000 >>>>> min_word_len: 3 >>>>> mem_limit: 256M >> >>>>> And here is sample output from running rake ts:index: >>>>> indexing index 'entry_core'... >>>>> collected 841492 docs, 1783.4 MB >>>>> collected 0 attr values >>>>> sorted 0.8 Mvalues, 100.0% done >>>>> sorted 205.5 Mhits, 100.0% done >>>>> total 841492 docs, 1783446653 bytes >>>>> total 1343.154 sec, 1327804.99 bytes/sec, 626.50 docs/sec >>>>> indexing index 'entry_delta'... >>>>> collected 0 docs, 0.0 MB >>>>> collected 0 attr values >>>>> sorted 0.0 Mvalues, nan% done >>>>> total 0 docs, 0 bytes >>>>> total 250.385 sec, 0.00 bytes/sec, 0.00 docs/sec >>>>> distributed index 'entry' can not be directly indexed; skipping. >> >>>>> It's taking around 25-30 minutes to run (without having any delta >>>>> indexes). That seems like quite a while compared to what I've seen as >>>>> sample times from other people. Does anybody have any suggestions for >>>>> what I could do to improve the performance, or any comments on the >>>>> speed of the indexing compared to what they have seen? >> >>>>> Thanks, >> >>>>> Simon >> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Thinking Sphinx" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group >>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Thinking Sphinx" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group >>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/thinking-sphinx?hl=en. > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
