This is confusing - if the setting's in the configuration file, then it's up to Sphinx to use it - it's not managed by Thinking Sphinx at all. I'm not quite sure what the next step here is...
If you want to construct a demo rails app which can reproduce the issue, that'd be great, then I can have a look on my machine. -- Pat On 05/01/2010, at 5:29 AM, Canvas wrote: > Hi Pat, > > I just came back from a long vocation. Sorry for the long delay. > > sqlquery_killlist does appear in development.sphinx.conf after I run > "rake thinking_sphinx:configure". So the problem I have now is that > the configuration is there but has no effect. > > Best wishes, > > Canvas > > > On Dec 25 2009, 8:00 pm, Pat Allan <[email protected]> wrote: >> Hi Canvas >> >> Sorry for the delay in getting back to you... >> >> When you're using 1.3.12 (and I recommend switching to 1.3.14 anyway), does >> the kill list appear in the config file? Is it that the setting is there and >> not having any effect, or that it's not getting into the conf file? >> >> As for the speed changes, that's interesting to know... there's been a *lot* >> of changes since Ed's fork appeared, so I guess that means there's plenty of >> potential reasons why that's now the case. I'll try to investigate when I >> have some time. >> >> -- >> Pat >> >> On 18/12/2009, at 7:56 AM, Canvas wrote: >> >>> Hi Pat, >> >>> I just tried rails 2.3.4 + thinking-sphinx 0.9.9 (Ed's fork) + sphinx >>> 0.9.9 final release. Everything works fine. Indexing is fast, >>> searching is fast, and sql_query_killlist works.. It's really weird >>> that thinking-sphinx >>> 1.3.12 is slow in searching and sql_query_killlist does not work in my >>> case. >> >>> Best wishes, >> >>> Canvas >> >>> On Dec 17, 11:48 am, Canvas <[email protected]> wrote: >>>> Hi Pat, >> >>>> I still can not figure out why sql_query_killlist does not work in >>>> thinking-sphinx 1.3.12. Any advice is appreciated. I am now using >>>> rails 2.3.4, sphinx 0.9.9 final release, thinking-sphinx 1.3.12. >> >>>> It's quite interesting that sql_quwery_killlist works quite well with >>>> rails 2.0.2, sphinx 0.9.9 final release and thinking-sphinx 0.9.9 >>>> (Ed's fork). >> >>>> I am stuck here now. Any advice will be appreciated. >> >>>> Best wishes, >> >>>> Canvas >> >>>> On Dec 16, 5:29 pm, Canvas <[email protected]> wrote: >> >>>>> Hi Pat, >> >>>>> I just found out that sql_query_killlist works perfectly with sphinx >>>>> 0.9.9 + thinking-sphinx 0.9.9, but it does not work at all with sphinx >>>>> 0.9.9 + thinking-sphinx 1.3.12. This proves that sql_query_killlist >>>>> works as far as sphinx 0.9.9 is concerned, which means something might >>>>> be wrong with thinking sphinx 1.3.12. I couldn't figure it out yet. >>>>> Any help will be appreciated. >> >>>>> I also noticed that when doing full index, thinking-sphinx 1.3.12 is >>>>> much faster, but searching is much slower , than thinking-sphinx >>>>> 0.9.9. Is it possible to improve the searching performance for >>>>> thinking-sphinx 1.3.12? >> >>>>> Thank you very much. >> >>>>> Best wishes, >> >>>>> Canvas >> >>>>> On Dec 16, 2:31 pm, Canvas <[email protected]> wrote: >> >>>>>> Hi Pat, >> >>>>>> I customized thinking-sphinx. Threshold is not in use at all in my >>>>>> case. >> >>>>>> I created a table sphinx_delta_index_start_points. The table holds one >>>>>> and only one row. The db migrate is as following: >> >>>>>> class CreateTableSphinxDeltaIndexStartPoints < ActiveRecord::Migration >>>>>> def self.up >>>>>> create_table :sphinx_delta_index_start_points do |t| >>>>>> t.datetime :delta_index_start_at, :null => false >>>>>> end >>>>>> end >> >>>>>> def self.down >>>>>> drop_table :sphinx_delta_index_start_points >>>>>> end >>>>>> end >> >>>>>> Every time a full index or merge index is executed, the start-time >>>>>> will be updated in the only row in the table above. And the three key >>>>>> item in configuration file for delta index will be as following. # >>>>>> {[email protected]_table_name} is used to represent whatever table name >>>>>> your model represents in your application. >> >>>>>> sql_query = SELECT ... FROM "#[email protected]_table_name}" WHERE # >>>>>> {[email protected]_table_name}.id >= $start AND # >>>>>> {[email protected]_table_name}.id <= $end AND # >>>>>> {[email protected]_table_name}.`updated_at` >= ( SELECT MIN >>>>>> (delta_index_start_at) FROM sphinx_delta_index_start_points ) GROUP BY >>>>>> #[email protected]_table_name}.id ORDER BY NULL >> >>>>>> sql_query_range = SELECT IFNULL(MIN(`id`), 1), IFNULL(MAX(`id`), 1) >>>>>> FROM #[email protected]_table_name} WHERE updated_at >= ( SELECT MIN >>>>>> (delta_index_start_at) FROM sphinx_delta_index_start_points ). >> >>>>>> sql_query_killlist = "SELECT id FROM #[email protected]_table_name} >>>>>> WHERE updated_at >= (SELECT MIN(delta_index_start_at) FROM >>>>>> sphinx_delta_index_start_points)". >> >>>>>> Hope this helps. >> >>>>>> Best wishes, >> >>>>>> Canvas >> >>>>>> On Dec 9, 11:11 pm, Pat Allan <[email protected]> wrote: >> >>>>>>> Quick question: how did you write a select that returns deleted values? >>>>>>> :) >> >>>>>>> -- >>>>>>> Pat >> >>>>>>> On 01/12/2009, at 7:36 AM, Canvas wrote: >> >>>>>>>> Hi Pat, >> >>>>>>>> I just tried it. It works! >> >>>>>>>> Please refer to the following url for further information. >> >>>>>>>> http://www.sphinxsearch.com/docs/manual-0.9.9.html#conf-sql-query-kil... >> >>>>>>>> On Nov 27, 6:04 pm, Pat Allan <[email protected]> wrote: >>>>>>>>> Ah, I never knew that it only changed the values in memory... that's >>>>>>>>> good to know! And I really should get the kill list stuff into >>>>>>>>> Thinking Sphinx proper. >> >>>>>>>>> Thanks for that info, muchly appreciated. >> >>>>>>>>> Cheers >> >>>>>>>>> -- >>>>>>>>> Pat >> >>>>>>>>> On 28/11/2009, at 12:37 PM, Canvas wrote: >> >>>>>>>>>> Hi Pat, >> >>>>>>>>>> It turns out that "client.update('buy_sell_file_core', >>>>>>>>>> ['sphinx_deleted'], { 4013 => [1] }) " updates index in RAM only. And >>>>>>>>>> the original core index stays unchanged. I am now trying sphinx >>>>>>>>>> 0.9.9- >>>>>>>>>> rc2, which includes a new feature sql_query_killlist to deal with >>>>>>>>>> this >>>>>>>>>> issue. I'll let you know when I try it out. >> >>>>>>>>>> http://www.sphinxsearch.com/forum/view.html?id=2552 >> >>>>>>>>>> Thanks a lot. >> >>>>>>>>>> Canvas >> >>>>>>>>>> On Nov 26, 6:25 pm, Pat Allan <[email protected]> wrote: >>>>>>>>>>> Hmm, I wonder if Sphinx doesn't delete documents that appear in both >>>>>>>>>>> indexes... >> >>>>>>>>>>> Scenarios to try: >>>>>>>>>>> - delete a record, add a different record, merge and see if the >>>>>>>>>>> deleted record is kept around or not >>>>>>>>>>> - flag the core copy as deleted, merge with the empty delta, >>>>>>>>>>> re-index >>>>>>>>>>> delta, merge again, see if only the updates are kept. >> >>>>>>>>>>> -- >>>>>>>>>>> Pat >> >>>>>>>>>>> On 27/11/2009, at 12:48 PM, Canvas wrote: >> >>>>>>>>>>>> Hi Pat, >> >>>>>>>>>>>> Thanks for your timely reply. client.update() does work. But I >>>>>>>>>>>> still >>>>>>>>>>>> have a wierd problem. Following are the steps for me to reproduce >>>>>>>>>>>> the >>>>>>>>>>>> problem. >> >>>>>>>>>>>> Step 1: build full index >>>>>>>>>>>> # rake thinking_sphinx:index RAILS_ENV=development >> >>>>>>>>>>>> Step 2: make some changes to file 4013 >> >>>>>>>>>>>> Step 3: build delta index ==> the file 4013 is searchable with both >>>>>>>>>>>> old and new data >>>>>>>>>>>> # rake thinking_sphinx:index:delta RAILS_ENV=development >> >>>>>>>>>>>> Step 4: flag the file in core index as "sphinx_deleted" ==> the >>>>>>>>>>>> file >>>>>>>>>>>> is searchable only by new data, so far so good. >>>>>>>>>>>>>> client.update('buy_sell_file_core', ['sphinx_deleted'], { 4013 => >>>>>>>>>>>>>> [1] }) >> >>>>>>>>>>>> Step 5: merge delta to core >>>>>>>>>>>> # /usr/local/bin/indexer --config '/workspace/CA/BETA_2/ >>>>>>>>>>>> EconveyancePro/ >>>>>>>>>>>> config/development.sphinx.conf' --rotate --merge buy_sell_file_core >>>>>>>>>>>> buy_sell_file_delta --merge-dst-range sphinx_deleted 0 0 >> >>>>>>>>>>>> And now the problem occurs, I can search by both new and old data >>>>>>>>>>>> again. It doesn't make any sense to me. Why can I search by the old >>>>>>>>>>>> data again? Isn't the old index supposed to be deleted in Step 4? >>>>>>>>>>>> Did >>>>>>>>>>>> I do anything wrong in step 5? >> >>>>>>>>>>>> Thank you very much for your help. >> >>>>>>>>>>>> Best wishes, >> >>>>>>>>>>>> Canvas >> >>>>>>>>>>>> On Nov 25, 12:26 am, Pat Allan <[email protected]> wrote: >>>>>>>>>>>>> Hi Canvas >> >>>>>>>>>>>>> Are you using Thinking Sphinx? It adds an internal attribute >>>>>>>>>>>>> called >>>>>>>>>>>>> sphinx_deleted, and sets records' values to 1 when they are >>>>>>>>>>>>> deleted >>>>>>>>>>>>> in >>>>>>>>>>>>> Ruby code. Then, if you're using the datetime deltas, the merge >>>>>>>>>>>>> automatically uses the --merge-dst-range option to remove deleted >>>>>>>>>>>>> items from the index. >> >>>>>>>>>>>>> However, if you're using Riddle, then the equivalent call is >>>>>>>>>>>>> client.update, not UpdateAttributes. I recommend looking at the >>>>>>>>>>>>> source >>>>>>>>>>>>> code to get a good understanding of how it all works. Riddle >>>>>>>>>>>>> documentation is pretty thin on the ground, but I'd like to >>>>>>>>>>>>> improve >>>>>>>>>>>>> it >>>>>>>>>>>>> over time. >> >>>>>>>>>>>>> Hope this helps. >> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Pat >> >>>>>>>>>>>>> On 25/11/2009, at 8:00 AM, Canvas wrote: >> >>>>>>>>>>>>>> Hi there guys, >> >>>>>>>>>>>>>> I am currently using sphinx 9.8.1. The following is from sphinx >>>>>>>>>>>>>> 9.8.1 >>>>>>>>>>>>>> document: >> >>>>>>>>>>>>>> " The basic command syntax is as follows: >> >>>>>>>>>>>>>> indexer --merge DSTINDEX SRCINDEX [--rotate] >> >>>>>>>>>>>>>> Only the DSTINDEX index will be affected: the contents of >>>>>>>>>>>>>> SRCINDEX >>>>>>>>>>>>>> will be merged into it. --rotate switch will be required if >>>>>>>>>>>>>> DSTINDEX >>>>>>>>>>>>>> is already being served by searchd. The initially devised usage >>>>>>>>>>>>>> pattern is to merge a smaller update from SRCINDEX into DSTINDEX. >>>>>>>>>>>>>> Thus, when merging the attributes, values from SRCINDEX will win >>>>>>>>>>>>>> >>>>>>>>>>>>>> if >>>>>>>>>>>>>> duplicate document IDs are encountered. Note, however, that the >>>>>>>>>>>>>> "old" >>>>>>>>>>>>>> keywords will not be automatically removed in such cases. For >>>>>>>>>>>>>> example, >>>>>>>>>>>>>> if there's a keyword "old" associated with document 123 in >>>>>>>>>>>>>> DSTINDEX, >>>>>>>>>>>>>> and a keyword "new" associated with it in SRCINDEX, document 123 >>>>>>>>>>>>>> will >>>>>>>>>>>>>> be found by both keywords after the merge. You can supply an >>>>>>>>>>>>>> explicit >>>>>>>>>>>>>> condition to remove documents from DSTINDEX to mitigate that; the >>>>>>>>>>>>>> relevant switch is --merge-dst-range: >> >>>>>>>>>>>>>> indexer --merge main delta --merge-dst-range deleted 0 0 >> >>>>>>>>>>>>>> This switch lets you apply filters to the destination index along >>>>>>>>>>>>>> with >>>>>>>>>>>>>> merging. There can be >> >> ... >> >> read more ยป > > -- > > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/thinking-sphinx?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
