Hi Greg Sorry for the slow response...
This is definitely a complicated situation, but (if I'm understanding things correctly) this isn't the best way to go about it. Firstly, defining indexes on the fly won't work - because those indexes aren't in the generated configuration file (and shouldn't be, really). Delta indexing (and Sphinx indexes in general) aren't that flexible - Sphinx doesn't index single records (although that will change in the future), and so it indexes all changes since the last full index. I'm not exactly sure of the best implementation approach at the moment... but I would keep the standard define_index block and delta column there. Even if there's no pages, surely the indexing doesn't fail? Sorry, I know this is all vague - my brain's not quite focused at the moment, but I wanted to get *something* sent off, instead of letting this sit around with no reply even longer. Happy to talk things through further. -- Pat On 25/08/2010, at 2:20 AM, gjb wrote: > Hello again. > > I have a new question regarding what I'm trying to set up here. We > have a problem where our core indexing works properly but the delta > indexing does not. The following is a pretty long summation of what > I've done and found so far. I apologize for the length and hope it > doesn't deter anyone from reading it, but I felt it was needed to get > a gist of what the problem is. > > To reiterate from my post from a week ago, we have a documents table. > Each document has a one-to-many relationship with the pages table. We > want to be able to search both on each page's text individually, as > well as search on all the text within a document as a whole. > > The define_index blocks for each model were the following. > > For Page: > > define_index do > indexes text > has document_id > > set_property :delta => :delayed > end > > For Document: > > define_index do > indexes pages.text, :as => :doc_text > has app_id > where "status = 4" > > set_property :delta => :delayed > end > > One problem with this is that the pages are inserted by the convert > process which is a ruby process running outside of Rails. I won't get > into the nasty details, but suffice to say this was causing path > issues in our Document model. To solve this, I essentially turned off > the define_index block in the Document model when not in Rails: > > if !ENV['RAILS_ENV'].nil? > # Don't want code that needs the Document model outside of Rails > to blow up > # because of this indexing code. > define_index do > indexes pages.text, :as => :doc_text > has app_id > where "status = 4" > end > end > > Another problem with this is we have a somewhat tricky situation, one > that may not be conducive to how the delta indexer is usually > implemented. Documents that are added to our db and picked up by a > web service and converted in a queue. Eventually, upon conversion, > the pages are processed and added to our table. The issue (at least I > believe so) is that because a delayed job to index a document gets > created upon any kind of update to its record, there are several > attempts to index a document before its pages are even in the db. And > since the pages' text are what is supposed to be indexed, this won't > work properly. > > The solution we thought of was basically to turn off the automatic > delta indexing and trigger it manually after a point we know the pages > have been inserted. So we got rid of the "set_property :delta > => :delayed" line in the Document model and the "delta" field in the > documents table. We set up our own custom delayed job which is run > after the pages insertion is complete, which calls an "index_delta" > function in the Document model (I'm overriding the function in > thinking-sphinx\lib\thinking_sphinx\active_record.rb), which looks > like this: > > def self.index_delta(instance) > doc_index = define_index do > indexes pages.text, :as => :doc_text > has app_id > where "status = 4" > end > > doc_index.delta_object = ThinkingSphinx::Deltas::DelayedDelta.new > doc_index, doc_index.local_options > doc_index.delta_object.index(self, instance) > end > > Maybe not the most elegant code, as I'm repeating things, but upon > testing it did appear to insert a delayed_job for indexing the > document. The only problem is the indexing is obviously still not > working properly. I can search on the pages fine, but running a > search on the document as a whole on a query that should return a hit > does not return the document. Only when I run the core indexer does > the document appear to finally index correctly. Which makes me > believe that this problem is with the delta indexing completely. > > I have two possible theories, which may or may not have anything to do > with the issue. One is that because our code has a Document object > that this may somehow be colliding with a Document object that > thinking_sphinx defines. However, since the core indexing is working > properly, I don't think this is the case. > > The other theory I have I ferreted out from looking at the > delayed_jobs records, specifically the handler field. When I insert a > document with id of 1, that has a page with an id of 1, I see the > following in the handler fields. > > For the delayed job that handles the page: > > '--- !ruby/object:ThinkingSphinx::Deltas::FlagAsDeletedJob > \ndocument_id: 1\nindex: page_core\n' > > For the delayed job that handles the document: > > '--- !ruby/object:ThinkingSphinx::Deltas::FlagAsDeletedJob > \ndocument_id: 3\nindex: document_core\n' > > The fact that our Document's id was showing up as 3 surprised me, so I > tracked down where this value was coming from. It comes from the > sphinx_document_id function in thinking-sphinx\lib\thinking_sphinx > \active_record.rb. > > def sphinx_document_id > (self.id * ThinkingSphinx.indexed_models.size) + > > ThinkingSphinx.indexed_models.index(self.class.source_of_sphinx_index.name) > end > > Looking at this, the value of 3 made sense. "indexed_models" should > have a size of 2 for Page and Document, so the document value is 1* 2 > + 1. > > What's odd then is the Page's document_id value of 1. This indicates > the value coming from the function is 1*1+0, when I would think it > should be 1*2+0 = 2. In other words, indexed_models size seems to be > 1 when the Page gets indexed, and 2 when the Document does. Then I > realized that this makes sense, since we have indexing on the Document > model turned off at this point. This could also explain why the core > indexer works, since presumably both indexes would be created by it > and the code would be able to work out the relationship between the > two. > > Does the Document index need to be there when a Page is indexed? Is > either of my theories on the right track? Or is something else wrong > about the way I'm trying to set up delta indexing here? > > Any help is appreciated. I realize I'm getting quite esoteric here > and that some things I've talked about might be confusing. Any > questions you have I can certainly clarify. > > Greg > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/thinking-sphinx?hl=en. > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
