>From reading through the source a bit more, it seems the `where` conditions
>for a real-time index can be set to either a symbol (representing an instance
>method name within the indexed model) or a proc (called with a single
>argument, the instance of the model), and if any condition exists, each much
>return true for the object to be added to the real-time index.
As for monitoring SphinxQL statements that are being sent to Sphinx, there's
been a recent commit to TS that adds real-time updates to the logs. Adding the
following to your Gemfile should sort that out:
gem 'thinking-sphinx', '~> 3.1.0',
:git => 'git://github.com/pat/thinking-sphinx.git',
:branch => 'develop',
:ref => '94ee176a7a'
Also: anything within the scope block (includes or otherwise) are only applied
for the full index generation. They're not used when updating the Sphinx
indices for a single record is created/edited, nor do they apply during
searches.
On 1 Mar 2014, at 2:46 am, Garrett Dimon <[email protected]> wrote:
> Thanks! We'll give that a shot. It seems the where *is* being applied at
> search time, but just not index-time. We're actually considering indexing
> everything with RT because previously when projects were unarchived or
> accounts unfrozen, they'd get picked backup with the full reindexes, but it
> seems with RT, that wouldn't happen because we don't run full re-indexes. Is
> there a way that we could tell TS to reindex in those situations if we use
> this approach and exclude them from the index?
>
> I'm trying to track down the specific content that's tripping up the
> generation, but I'm not having much luck. I found the record that's printed
> out when rake files, and I manually inspected the records in nearby proximity
> to it, but they all check out. Is there a way/location for me to see the SQL
> query that's being used at the moment the rake task fails?
>
>
> On Fri, Feb 28, 2014 at 9:21 AM, Pat Allan <[email protected]> wrote:
> The `where` method doesn't apply for real-time indices - but try this instead
> in the index definition block:
>
> scope { Issue.where "account_status_id IN (1,2) AND archived IS false" }
>
> That should ensure only the appropriate issues are indexed as part of the
> generate call.
>
> Beyond that, you may wish to add `includes` within that scope to cover all
> associations used within the index?
>
> Unrelated to any of this: the match_mode defaults to extended with TS v3
> (indeed, it can't be anything else).
>
> --
> Pat
>
> On 1 Mar 2014, at 2:09 am, Garrett Dimon <[email protected]> wrote:
>
> > Roger on the generation. Any high-level suggestions for optimization?
> >
> > I'll see if I can't figure out the exact record that's tripping up the
> > index generation. In the meantime, here's a gist of our index definition:
> > https://gist.github.com/garrettdimon/25f6c305541f30b3ce39
> >
> >
> > On Thu, Feb 27, 2014 at 9:21 PM, Pat Allan <[email protected]>
> > wrote:
> > Hi Garrett
> >
> > Generation can be slow - at the end of the day, it really comes down to how
> > much data you're dealing with, and if you're using aggregation methods, how
> > quick they are. It's all going through your Rails app (instead of just SQL
> > queries), so optimising for that is different to adding db indices and such.
> >
> > As for the error though... without having a copy of the database, it's a
> > little hard to debug, but it sounds like there's a bug in TS with something
> > in the data being passed through. Having a look at your app log may help
> > identify the record in question... also, what does the index definition for
> > that model look like?
> >
> > --
> > Pat
> >
> > On 28 Feb 2014, at 9:47 am, Garrett Dimon <[email protected]> wrote:
> >
> >> Howdy, Pat. We're in the process of upgrading from TS 2 with delayed
> >> deltas to TS 3 with real time indexing.
> >>
> >> We've been able to get everything up and running, but we've run into a
> >> couple of problems/questions around the indexing. These may ultimately be
> >> Sphinx questions rather than Thinking Sphinx questions, but I thought I'd
> >> start here since we're only changing Sphinx from 2.0 to 2.1.
> >>
> >> 1. Index Creation Performance
> >>
> >> Our production logs show about a 20 minute turnaround to do a complete
> >> reindex of our production data with TS 2. Running some local tests, TS 3
> >> generate is taking at least an hour for that data. (The generate is
> >> crashing, so it may ultimately take even longer.) Our indexing
> >> configuration is setup so that a large portion of content is excluded from
> >> the index. (Inactive accounts, archived projects, etc.) We've verified
> >> that searching is correctly excluding the relevant records, but appears as
> >> if that's happening when the query is run rather than when the indexing
> >> occurs. Our only theory so far is that with TS 2 and traditional indexing,
> >> those weren't included in the index at all, but that with real time
> >> indexing, they're included in the index and filtered out when the query is
> >> run. Can you provide any insight about whether this sounds like normal
> >> behavior or whether we've likely screwed something up? :)
> >>
> >> 2. The Generate is crashing with "rake aborted! sphinxql: syntax error,
> >> unexpected $undefined, expecting CONST_INT (or 4 other tokens) near ''..."
> >> (where ... is content from our DB.)
> >>
> >> I've done some searching, but haven't had any luck. I've run the generate
> >> rake task on two separate occasions, and both times it failed with the
> >> same error message and content, so my gut is leading me to think that it's
> >> an encoding or unescaped quotation mark problem. Does that problem ring
> >> any bells?
> >>
> >> Thanks!
> >>
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Thinking Sphinx" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to [email protected].
> >>
> >> To post to this group, send email to [email protected].
> >> Visit this group at http://groups.google.com/group/thinking-sphinx.
> >> For more options, visit https://groups.google.com/groups/opt_out.
> >
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "Thinking Sphinx" group.
> > To unsubscribe from this topic, visit
> > https://groups.google.com/d/topic/thinking-sphinx/7llAB4zO4bw/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to
> > [email protected].
> > To post to this group, send email to [email protected].
> > Visit this group at http://groups.google.com/group/thinking-sphinx.
> > For more options, visit https://groups.google.com/groups/opt_out.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Thinking Sphinx" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [email protected].
> > To post to this group, send email to [email protected].
> > Visit this group at http://groups.google.com/group/thinking-sphinx.
> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to a topic in the Google
> Groups "Thinking Sphinx" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/thinking-sphinx/7llAB4zO4bw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/thinking-sphinx.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Thinking Sphinx" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/thinking-sphinx.
> For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"Thinking Sphinx" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/thinking-sphinx.
For more options, visit https://groups.google.com/groups/opt_out.
