Well, BackgroundRB does have a problem because workers can't overlap,
so a worker is put in the queue when it's started, which is a bit of a
pain because some HTTP-requests take longer than others. But look at
this:
def schedule_queries
i =
[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,35]
i.each do |j|
Thread.new do
logger.info "Hello #{j}"
sleep 2
logger.info "Hello Again #{j}"
end
sleep 1
end
end
This is executed every 10 seconds and my log shows this:
Hello 1
Hello 2
Hello Again 1
Hello 3
Hello Again 2
Hello 4
Hello Again 3
Hello 5
Hello Again 4
Hello 6
Hello Again 5
(...)
Hello Again 23
Hello 35
Hello Again 24
Hello Again 35
Hello 1
Hello 2
Hello Again 1
Hello 3
The expected behaviour is for the second "Hello 1" to appear after the
first "Hello 10", but it doesn't. However, that's not a real problem,
because I can easily work with bigger collections. However, when I
implement the script you presented earlier, it works! I don't know
why, I certainly don't know how, but it does.
Could this have anything to do with ActiveRecord? Every time I call
set_active, I do a host.save. Could that be the problem? I'll work on
this some more in the morning.
On 18 nov, 21:28, jhaagmans <[email protected]> wrote:
> > > The reason why we didn't choose jRuby was because it uses too much
> > > memory to be able to run this on a VPS. Is there any documentation
> > > available on using jRuby on a low-memory (<256MB) system? I've looked
> > > for it, but couldn't find it. Maybe there's an alternative workaround
> > > for the GIL? Our application uses up quite alot of memory, so when
> > > presented with the jRuby vs. Ruby (EE) question, I thought it was a
> > > choice between thread safety and memory usage, so I chose the latter.
> > > I didn't know there was more to think about.
>
> > > > Regardless, I can't seem to reproduce the OPs behviour:
>
> > > Which Ruby implementation are you using? I'm very sure every thread in
> > > my piece of code is waiting for the other thread to finish, because I
> > > log the time at which the data is saved. Most of the time there's 10 -
> > > 40 seconds between them, even though the backgroundrb process should
> > > save at least one object every second.
>
> > x...@clover:~/projects$ ruby -v
> > ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]
>
> > Threading's not the issue. Try looking into Lionel's suggestion;
> > perhaps things will work as expected if you switch to delayed_job
> > instead of BackgroundRB, or posssssibly even just a different HTTP
> > client like curb. Or perhaps the problem's not related to the HTTP
> > fetching at all and we don't have the whole story.
>
> If the problem is in BackgroundRB I'm screwed. This whole system
> depends on BackgroundRB. It's not just a long-running task, most tasks
> will run daily for years and years to come. I'll check whether this is
> related to BackgroundRB and report back.
--
You received this message because you are subscribed to the Google Groups "Ruby
on Rails: Talk" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.
