This is what I am doing (trying anyway) in production: 1) Full index every 24 hours 2) Delta index every 20 minutes with threshold set at 30 minutes (after an initial full index)
I end up with the most recent full index (up to 24 hours old) plus the most recent delta index regardless of previous delta indexes. So after 2 hours, I have everything up to 2 hours old, plus everything 30 minutes old, but nothing in between. I tested this by: 1) Running a full index 2) Waiting a few hours 3) Running a delta index with a threshold of 1 hour 4) Searching for a record 45 minutes old and finding it 5) Without doing anything else except reconfiguring, running a delta index with a threshold of 1 minute 6) Searching for a record 45 minutes old and NOT finding it This indicates to me that the delta index is not being merged in. I'm also wondering how the index handles duplicates under this scenario since it will index records multiple times and there doesn't seem to be any id based de-duping code. On Jun 15, 7:34 pm, Pat Allan <[email protected]> wrote: > I don't think I'm quite understanding the situation, so forgive me if > I sound a little condescending... > > Are you saying you've not run a main index for over ten hours, and > then run a delta index (with threshold set to 1 hour), and you don't > get the changes that have happened since the main index but before the > last hour? That's the expected outcome - the delta index task needs to > be run at the same regularity as the threshold. > > Or have I got it completely wrong? > > -- > Pat > > On 15/06/2009, at 10:24 PM, Alex wrote: > > > > > Possibly. But basically if I have records spanning the last 10 hours, > > and I set the threshold at 12 hours, they all get indexed with the > > main index. If I change the threshold to 1 hour, then only the last > > hour plus the main index shows up. > > > -Alex > > > On Jun 15, 6:28 pm, Pat Allan <[email protected]> wrote: > >> Hi Alex > > >> The output looks right... is it possible for you to wrap up the > >> repeatable test in some way so I can run it on my machine? > > >> -- > >> Pat > > >> On 15/06/2009, at 8:45 PM, Alex wrote: > > >>> Here is the output from rake ts:in:delta : > > >>> (in /mnt/app/releases/20090610140740) > >>> Sphinx 0.9.8.1-release (r1533) > >>> Copyright (c) 2001-2008, Andrew Aksyonoff > > >>> using config file '/mnt/app/releases/20090610140740/config/ > >>> production.sphinx.conf'... > >>> indexing index 'source_listing_delta'... > >>> collected 41730 docs, 180.2 MB > >>> collected 0 attr values > >>> sorted 0.0 Mvalues, 100.0% done > >>> sorted 26.6 Mhits, 100.0% done > >>> total 41730 docs, 180158829 bytes > >>> total 123.160 sec, 1462798.97 bytes/sec, 338.83 docs/sec > >>> rotating indices: succesfully sent SIGHUP to searchd (pid=5673). > >>> Sphinx 0.9.8.1-release (r1533) > >>> Copyright (c) 2001-2008, Andrew Aksyonoff > > >>> using config file '/mnt/app/releases/20090610140740/config/ > >>> production.sphinx.conf'... > >>> merged 3294.0 Kwords > >>> merged in 167.3 sec > >>> rotating indices: succesfully sent SIGHUP to searchd (pid=5673). > > >>> On Jun 15, 4:59 pm, Alex <[email protected]> wrote: > >>>> I'm using sphinx-0.9.8.1 and it is definitely not working - I have > >>>> done a number of repeatable tests. > > >>>> Sorry for the long response time - I've been traveling. > > >>>> On Jun 10, 5:16 pm, Pat Allan <[email protected]> wrote: > > >>>>> Hi Alex > > >>>>> Yes, it should be merging, so something's not quite right. What > >>>>> version of Sphinx are you running? > > >>>>> -- > >>>>> Pat > > >>>>> On 10/06/2009, at 9:52 AM, Alex wrote: > > >>>>>> When I set a a threshold of 10.minutes, I get the last full re- > >>>>>> index > >>>>>> plus the last 10 minutes of updates. Re-running rake:in:delta > >>>>>> replaces the previous delta instead of adding to it. That > >>>>>> means my > >>>>>> threshold needs to be set according to full reindexing periods, > >>>>>> not > >>>>>> delta periods, which throws everything off. > > >>>>>> Isn't this supposed to merge the delta into the main index? That > >>>>>> doesn't seem to be working as expected. > > >>>>>> -Alex > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en -~----------~----~----~----~------~----~------~--~---
