Hi Pat No... I've tried all of those methods and nothing is doing.
Thanks for that link for the workling indexer. I might have a look at that although at this stage I'm heading towards running local delta indexes on the web servers and re-indexing every few minutes to get the site launched. The load-balancer directs the same IP to the same box for the most part so as long as I re-index at a shorter interval than the timeout for that I should be good to go and no user will notice. Changes need to be immediate in the app but only really for the user making them. A couple of minutes delay for everyone else (or everyone else that is directed to another server) shouldn't be a problem. The biggest table is only relatively small at 50 - 100k rows so the index runs quickly at this stage. We'll upgrade to rails 3 in the next couple of months and I'll have another go with delayed_job then. Thanks for all your help. Graham On Feb 1, 12:48 am, Pat Allan <[email protected]> wrote: > Hi Graham > > It's a shame that it's still not working. I guess my one last question is - > is there any difference between running script/delayed_job, rake jobs:work > and Delayed::Worker.work_off? (Granted, I'm used to the Rails 3 version, so > maybe these are slightly different). > > It shouldn't be too hard to create a new gem for a different queue system - > one exists for Workling: > > https://github.com/dpickett/workling_delta_indexer > > And I know others have been created (but maybe not released publicly). > > Good luck! > > -- > Pat > > On 01/02/2011, at 9:24 AM, Graham Pengelly wrote: > > > > > > > > > Hi Pat > > > I've spent another age on this and still haven't got a solution. The > > issue is definitely down to delayed_job or one of its dependencies. I > > can happily run everything from your code outside of delayed_job but > > once I start to queue it. > > > There are lots of issues on the delayed_job issues page on github > > (https://github.com/collectiveidea/delayed_job/issues) which don't > > seem to be getting addressed in the 2.0.* branch. I've had a look but > > can't see anything I can contribute myself at this stage and the gem > > has moved on now to the rails 3 version. > > > In terms of progress, I've actually got it to run now but it hangs on > > the second indexing job for some reason and the DJ daemon has to be > > restarted, at which point it will run another indexing job and then > > hang again. I got to this stage with DJ 2.0.5 and daemons 1.0.10 (1.1 > > has known issues with DJ it appears and won't let the daemon start). > > I've also tried swapping out the use of daemons with daemon-spawn and > > got the same result. First index is fine, subsequent one hangs. > > > I'm going to have to give up on this approach and try something else, > > probably another queue and write my own custom delta worker. If > > there's anything I can contribute back to TS I will do... > > > Thanks for your help > > > Graham > > > On Jan 30, 2:30 pm, Graham Pengelly <[email protected]> wrote: > >> Hi Pat > > >> Yes... I tried system(... but just got the same problem. It did at one > >> stage go through but output a "memory can't be allocated" message but > >> when I looked into it I'd just run the rake task so many times that > >> I'd filled up the RAM on the box. Once I cleared it we were back to > >> square one with it hanging. I thought I was on to something with that > >> as well but apparently not. > > >> I'm logged in as the user that rails runs under so if I execute the > >> index command directly it should be the same user as when I do it > >> through rake ts:dd etc. I did think that it may be a permissions issue > >> so I changed the permissions on the indexer to 777 but it didn't make > >> any difference. The other delayed jobs that ts queues go through fine > >> when I re-run the rake task. It is just the indexer one that has the > >> problem so it seems that delayed_job is working fine except with the > >> sphinx indexer. > > >> I'm going to have another crack at it this evening so I'll post back > >> if I make any progress... > > >> Thanks > > >> Graham > > >> On Jan 30, 1:21 pm, Pat Allan <[email protected]> wrote: > > >>> Hi Graham > > >>> What happens if you switch `#{config.bin_path}...` to > >>> system("#{config.bin_path}...")? > > >>> No idea why it would be any different, mind you. Are you running the > >>> commands manually with the same user that runs delayed job? > > >>> -- > >>> Pat > > >>> On 30/01/2011, at 9:04 AM, Graham Pengelly wrote: > > >>>> Hi Pay > > >>>> I still haven't fixed it but it seems that it is hanging in the > >>>> perform method on DeltaJob when the indexer gets called. The command > >>>> is fine - > > >>>> /usr/bin/indexer --config /home/rails/secure.mapslegal.co.uk/releases/ > >>>> 20110128140157/config/production.sphinx.conf --rotate policy_delta > >>>> policy_delta > > >>>> ...and if I execute 'manually' it works as expected. Curiously, if I > >>>> alter that line and change it to exec "#{config.bin_path}... The > >>>> indexer runs successfully without hanging but obviously it never > >>>> returns and the rake task exits without updating the delayed_jobs > >>>> table. > > >>>> I'll have another go at it tomorrow but thought I'd report back and > >>>> see if my 'findings' so far shed any light on the problem. > > >>>> Thanks > > >>>> Graham > > >>>> On Jan 27, 8:22 am, Graham Pengelly <[email protected]> wrote: > >>>>> Hi Pat > > >>>>> Yes... thinking_sphinx, ts-delayed-delta and delayed-job are all > >>>>> installed as gems. I'm away today so can't work on it but I'll have > >>>>> another crack at it tomorrow and post back on here with any progress. > > >>>>> Thanks again for your advice. > > >>>>> Graham > > >>>>> On Jan 27, 1:21 am, Pat Allan <[email protected]> wrote: > > >>>>>> Hi Graham > > >>>>>> Just to confirm - are you using TS as a gem? From memory, I think if > >>>>>> ts-delayed-delta is being used, TS needs to be installed as a gem > >>>>>> instead of a plugin. > > >>>>>> Beyond that, I'm at a loss as to why this could be happening. I guess, > >>>>>> I'd start debugging by editing the delayed job and ts-delayed-delta > >>>>>> gems to figure out where it's hanging. Can understand if you don't > >>>>>> want to do this, mind you... if you're willing to give me access to > >>>>>> the project so I can try reproducing the issue, contact me off-list. > > >>>>>> Cheers > > >>>>>> -- > >>>>>> Pat > > >>>>>> On 26/01/2011, at 10:47 PM, Graham Pengelly wrote: > > >>>>>>> Hi Pat > > >>>>>>> Thanks for responding... I'm still stuck with it and was about to > >>>>>>> resort to trying to fix it with a hammer :) > > >>>>>>> I thought there might be some version issues myself as the latest > >>>>>>> delayed_job looked to be only compatible with rails 3 and the ts- > >>>>>>> delayed-job gem seemed to pull in the latest version of delayed_job > >>>>>>> when dj wasn't installed. So... I installed the earlier, 2.3.* > >>>>>>> compatible version of dj. > > >>>>>>> My app is currently on: > > >>>>>>> Rails 2.3.8 > >>>>>>> thinking-sphinx 1.4.1 > >>>>>>> ts-delayed-delta 1.1.1 > >>>>>>> delayed_job 2.0.4 > > >>>>>>> Thanks > > >>>>>>> Graham > > >>>>>>> On Jan 26, 11:20 am, Pat Allan <[email protected]> wrote: > >>>>>>>> Hi Graham > > >>>>>>>> This is definitely something odd. What versions of Rails and Delayed > >>>>>>>> Job are you using? > > >>>>>>>> -- > >>>>>>>> Pat > > >>>>>>>> On 24/01/2011, at 10:02 PM, Graham Pengelly wrote: > > >>>>>>>>> Hi > > >>>>>>>>> I'm attempting to set up delta indexing using delayed_job via the > >>>>>>>>> the > >>>>>>>>> ts-delayed-delta (1.1.1) gem. The app is happily queuing jobs to the > >>>>>>>>> delayed_job table in the db but when I run the ts:dd rake task (or > >>>>>>>>> any > >>>>>>>>> other method of getting the delayed_job worker running) it hangs > >>>>>>>>> after > >>>>>>>>> locking the first delta job with the log - > > >>>>>>>>> Starting job worker > >>>>>>>>> acquired lock on ThinkingSphinx::Deltas::DeltaJob for policy_delta, > >>>>>>>>> policy_delta > > >>>>>>>>> I can create a ThinkingSphinx::Deltas::DeltaJob and then 'perform' > >>>>>>>>> it > >>>>>>>>> successfully from the console so that works on the machine. The db > >>>>>>>>> table is also getting written to as the 'locked_at' and 'locked_by' > >>>>>>>>> field are completed. It isn't specifically the policy model causing > >>>>>>>>> the problem as I use delta indexing on a couple of other models and > >>>>>>>>> they hang after the lock too. > > >>>>>>>>> Any ideas would be gratefully received. > > >>>>>>>>> Thanks in advance > > >>>>>>>>> Graham > > >>>>>>>>> -- > >>>>>>>>> You received this message because you are subscribed to the Google > >>>>>>>>> Groups "Thinking Sphinx" group. > >>>>>>>>> To post to this group, send email to > >>>>>>>>> [email protected]. > >>>>>>>>> To unsubscribe from this group, send email to > >>>>>>>>> [email protected]. > >>>>>>>>> For more options, visit this group > >>>>>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google > >>>>>>> Groups "Thinking Sphinx" group. > >>>>>>> To post to this group, send email to [email protected]. > >>>>>>> To unsubscribe from this group, send email to > >>>>>>> [email protected]. > >>>>>>> For more options, visit this group > >>>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "Thinking Sphinx" group. > >>>> To post to this group, send email to [email protected]. > >>>> To unsubscribe from this group, send email to > >>>> [email protected]. > >>>> For more options, visit this group > >>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Thinking Sphinx" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/thinking-sphinx?hl=en. -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
