Hello again.
I have a new question regarding what I'm trying to set up here. We
have a problem where our core indexing works properly but the delta
indexing does not. The following is a pretty long summation of what
I've done and found so far. I apologize for the length and hope it
doesn't deter anyone from reading it, but I felt it was needed to get
a gist of what the problem is.
To reiterate from my post from a week ago, we have a documents table.
Each document has a one-to-many relationship with the pages table. We
want to be able to search both on each page's text individually, as
well as search on all the text within a document as a whole.
The define_index blocks for each model were the following.
For Page:
define_index do
indexes text
has document_id
set_property :delta => :delayed
end
For Document:
define_index do
indexes pages.text, :as => :doc_text
has app_id
where "status = 4"
set_property :delta => :delayed
end
One problem with this is that the pages are inserted by the convert
process which is a ruby process running outside of Rails. I won't get
into the nasty details, but suffice to say this was causing path
issues in our Document model. To solve this, I essentially turned off
the define_index block in the Document model when not in Rails:
if !ENV['RAILS_ENV'].nil?
# Don't want code that needs the Document model outside of Rails
to blow up
# because of this indexing code.
define_index do
indexes pages.text, :as => :doc_text
has app_id
where "status = 4"
end
end
Another problem with this is we have a somewhat tricky situation, one
that may not be conducive to how the delta indexer is usually
implemented. Documents that are added to our db and picked up by a
web service and converted in a queue. Eventually, upon conversion,
the pages are processed and added to our table. The issue (at least I
believe so) is that because a delayed job to index a document gets
created upon any kind of update to its record, there are several
attempts to index a document before its pages are even in the db. And
since the pages' text are what is supposed to be indexed, this won't
work properly.
The solution we thought of was basically to turn off the automatic
delta indexing and trigger it manually after a point we know the pages
have been inserted. So we got rid of the "set_property :delta
=> :delayed" line in the Document model and the "delta" field in the
documents table. We set up our own custom delayed job which is run
after the pages insertion is complete, which calls an "index_delta"
function in the Document model (I'm overriding the function in
thinking-sphinx\lib\thinking_sphinx\active_record.rb), which looks
like this:
def self.index_delta(instance)
doc_index = define_index do
indexes pages.text, :as => :doc_text
has app_id
where "status = 4"
end
doc_index.delta_object = ThinkingSphinx::Deltas::DelayedDelta.new
doc_index, doc_index.local_options
doc_index.delta_object.index(self, instance)
end
Maybe not the most elegant code, as I'm repeating things, but upon
testing it did appear to insert a delayed_job for indexing the
document. The only problem is the indexing is obviously still not
working properly. I can search on the pages fine, but running a
search on the document as a whole on a query that should return a hit
does not return the document. Only when I run the core indexer does
the document appear to finally index correctly. Which makes me
believe that this problem is with the delta indexing completely.
I have two possible theories, which may or may not have anything to do
with the issue. One is that because our code has a Document object
that this may somehow be colliding with a Document object that
thinking_sphinx defines. However, since the core indexing is working
properly, I don't think this is the case.
The other theory I have I ferreted out from looking at the
delayed_jobs records, specifically the handler field. When I insert a
document with id of 1, that has a page with an id of 1, I see the
following in the handler fields.
For the delayed job that handles the page:
'--- !ruby/object:ThinkingSphinx::Deltas::FlagAsDeletedJob
\ndocument_id: 1\nindex: page_core\n'
For the delayed job that handles the document:
'--- !ruby/object:ThinkingSphinx::Deltas::FlagAsDeletedJob
\ndocument_id: 3\nindex: document_core\n'
The fact that our Document's id was showing up as 3 surprised me, so I
tracked down where this value was coming from. It comes from the
sphinx_document_id function in thinking-sphinx\lib\thinking_sphinx
\active_record.rb.
def sphinx_document_id
(self.id * ThinkingSphinx.indexed_models.size) +
ThinkingSphinx.indexed_models.index(self.class.source_of_sphinx_index.name)
end
Looking at this, the value of 3 made sense. "indexed_models" should
have a size of 2 for Page and Document, so the document value is 1* 2
+ 1.
What's odd then is the Page's document_id value of 1. This indicates
the value coming from the function is 1*1+0, when I would think it
should be 1*2+0 = 2. In other words, indexed_models size seems to be
1 when the Page gets indexed, and 2 when the Document does. Then I
realized that this makes sense, since we have indexing on the Document
model turned off at this point. This could also explain why the core
indexer works, since presumably both indexes would be created by it
and the code would be able to work out the relationship between the
two.
Does the Document index need to be there when a Page is indexed? Is
either of my theories on the right track? Or is something else wrong
about the way I'm trying to set up delta indexing here?
Any help is appreciated. I realize I'm getting quite esoteric here
and that some things I've talked about might be confusing. Any
questions you have I can certainly clarify.
Greg
--
You received this message because you are subscribed to the Google Groups
"Thinking Sphinx" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/thinking-sphinx?hl=en.
