Hi Garrett Generation can be slow - at the end of the day, it really comes down to how much data you're dealing with, and if you're using aggregation methods, how quick they are. It's all going through your Rails app (instead of just SQL queries), so optimising for that is different to adding db indices and such.
As for the error though... without having a copy of the database, it's a little hard to debug, but it sounds like there's a bug in TS with something in the data being passed through. Having a look at your app log may help identify the record in question... also, what does the index definition for that model look like? -- Pat On 28 Feb 2014, at 9:47 am, Garrett Dimon <[email protected]> wrote: > Howdy, Pat. We're in the process of upgrading from TS 2 with delayed deltas > to TS 3 with real time indexing. > > We've been able to get everything up and running, but we've run into a couple > of problems/questions around the indexing. These may ultimately be Sphinx > questions rather than Thinking Sphinx questions, but I thought I'd start here > since we're only changing Sphinx from 2.0 to 2.1. > > 1. Index Creation Performance > > Our production logs show about a 20 minute turnaround to do a complete > reindex of our production data with TS 2. Running some local tests, TS 3 > generate is taking at least an hour for that data. (The generate is crashing, > so it may ultimately take even longer.) Our indexing configuration is setup > so that a large portion of content is excluded from the index. (Inactive > accounts, archived projects, etc.) We've verified that searching is correctly > excluding the relevant records, but appears as if that's happening when the > query is run rather than when the indexing occurs. Our only theory so far is > that with TS 2 and traditional indexing, those weren't included in the index > at all, but that with real time indexing, they're included in the index and > filtered out when the query is run. Can you provide any insight about whether > this sounds like normal behavior or whether we've likely screwed something > up? :) > > 2. The Generate is crashing with "rake aborted! sphinxql: syntax error, > unexpected $undefined, expecting CONST_INT (or 4 other tokens) near ''..." > (where ... is content from our DB.) > > I've done some searching, but haven't had any luck. I've run the generate > rake task on two separate occasions, and both times it failed with the same > error message and content, so my gut is leading me to think that it's an > encoding or unescaped quotation mark problem. Does that problem ring any > bells? > > Thanks! > > > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/thinking-sphinx. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/groups/opt_out.
