Roger on the generation. Any high-level suggestions for optimization? I'll see if I can't figure out the exact record that's tripping up the index generation. In the meantime, here's a gist of our index definition: https://gist.github.com/garrettdimon/25f6c305541f30b3ce39
On Thu, Feb 27, 2014 at 9:21 PM, Pat Allan <[email protected]> wrote: > Hi Garrett > > Generation can be slow - at the end of the day, it really comes down to > how much data you’re dealing with, and if you’re using aggregation methods, > how quick they are. It’s all going through your Rails app (instead of just > SQL queries), so optimising for that is different to adding db indices and > such. > > As for the error though… without having a copy of the database, it’s a > little hard to debug, but it sounds like there’s a bug in TS with something > in the data being passed through. Having a look at your app log may help > identify the record in question… also, what does the index definition for > that model look like? > > — > Pat > > On 28 Feb 2014, at 9:47 am, Garrett Dimon <[email protected]> wrote: > > Howdy, Pat. We're in the process of upgrading from TS 2 with delayed > deltas to TS 3 with real time indexing. > > We've been able to get everything up and running, but we've run into a > couple of problems/questions around the indexing. These may ultimately be > Sphinx questions rather than Thinking Sphinx questions, but I thought I'd > start here since we're only changing Sphinx from 2.0 to 2.1. > > 1. Index Creation Performance > > Our production logs show about a 20 minute turnaround to do a complete > reindex of our production data with TS 2. Running some local tests, TS 3 > generate is taking at least an hour for that data. (The generate is > crashing, so it may ultimately take even longer.) Our indexing > configuration is setup so that a large portion of content is excluded from > the index. (Inactive accounts, archived projects, etc.) We've verified that > searching is correctly excluding the relevant records, but appears as if > that's happening when the query is run rather than when the indexing > occurs. Our only theory so far is that with TS 2 and traditional indexing, > those weren't included in the index at all, but that with real time > indexing, they're included in the index and filtered out when the query is > run. Can you provide any insight about whether this sounds like normal > behavior or whether we've likely screwed something up? :) > > 2. The Generate is crashing with "rake aborted! sphinxql: syntax error, > unexpected $undefined, expecting CONST_INT (or 4 other tokens) near ''..." > (where ... is content from our DB.) > > I've done some searching, but haven't had any luck. I've run the generate > rake task on two separate occasions, and both times it failed with the same > error message and content, so my gut is leading me to think that it's an > encoding or unescaped quotation mark problem. Does that problem ring any > bells? > > Thanks! > > > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/thinking-sphinx. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Thinking Sphinx" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/thinking-sphinx/7llAB4zO4bw/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/thinking-sphinx. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/groups/opt_out.
