Yeah, I was using 2.0.11 previously. There does not seem to be any difference with removing sortable: true from the index definition (for resumes.document), except that this line disappears from the generated configuration file: sql_field_string = document. This seems to at least let indexer complete properly, but the index size is still huge:
indexing index 'candidate_user_core'... collected 199704 docs, *8478.8 MB* It also takes a long time to go through the sorting "Mhits" step now. I see how TS2 added sql_attr_string for the sort columns whereas TS3 adds sql_field_string - that's what you're talking about right? Is there any way to either a) get around this issue, or b) force TS to use the ordinal type? (everything should still work that way, correct?) Here's the options I set in thinking_sphinx.yml: development: address: localhost version: 2.0.8-release mem_limit: 256M enable_star: true min_prefix_len: 2 blend_chars: "@, -, &" html_strip: true max_matches: 25000 Is there any way I can speed this up / reduce the size? On Thursday, July 18, 2013 1:59:11 PM UTC-4, Pat Allan wrote: > > I think with 2.0.11 (what you were using previously, right?) TS uses the > ordinal attribute type, which stores an integer for each string (calculated > by grabbing all known values, putting them in order, returning the index of > each value). > > With TS v3 (and later 2.x releases if I remember correctly) it'll use the > native string attribute type (a relatively recent addition to Sphinx), > which means Sphinx is storing the real string value - which is much better > if you're sorting across more than one index (say, if you're using deltas, > or searching across multiple models). In this case, it would mean Sphinx is > now storing potentially a ton of data, instead of a 32-bit integer per > record. > > -- > Pat > > On 19/07/2013, at 3:53 AM, Daniel Vandersluis wrote: > > > Thanks for the response, Pat - yes, it's the same index as the other > thread. Good point about sorting resumes, that shouldn't be there. However, > why would that make such a difference between TS2 and TS3 (see my other > post which I added at the same time as your response)? > > > > I will try removing the sortable on resumes and see what difference it > makes! > > > > On Thursday, July 18, 2013 1:49:13 PM UTC-4, Pat Allan wrote: > > Hi Daniel > > > > If this is the same index as in the other thread, I'm guessing it's the > fact that you've got resumes.document sortable. A record with many resumes > and/or large document values could end up with massive values for the > underlying string attribute (that you'd sort by) - are you actually sorting > by this? Generally I'd be surprised if there's much point sorting by large > amounts of text. > > > > -- > > Pat > > > > On 19/07/2013, at 3:09 AM, Daniel Vandersluis wrote: > > > > > Is there any reason that an index would grow in size when upgrading > from thinkingsphinx 2 to 3? The only differences in the configuration file > is changing port to mysql41, and changing version to 2.0.8-release, but an > index that used to be around 500MB is now resulting in this error: > > > > > > ERROR: index 'user_core': too many string attributes (current index > format allows up to 4 GB). > > > > > > Anyone have any idea why this would be? > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups "Thinking Sphinx" group. > > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > > To post to this group, send email to [email protected]. > > > Visit this group at http://groups.google.com/group/thinking-sphinx. > > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Thinking Sphinx" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] <javascript:>. > > To post to this group, send email to > > [email protected]<javascript:>. > > > Visit this group at http://groups.google.com/group/thinking-sphinx. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/groups/opt_out.
