Re: [Mongrel] problem in downloading large file with apache+mongrel

[James Tucker]

On 23 Sep 2010, at 02:42, Amit Tomar wrote:

> James Tucker wrote:
>> On 22 Sep 2010, at 12:00, Amit Tomar wrote:
>>> luis i change my code as you said
>>> 
>>> filename =  "#...@stream.location}"
>>> response.headers['Content-Type'] = "application/force-download"
>>> response.headers['Content-Disposition'] = "attachment; 
>>> filename=\"#{File.basename(filename)}\""
>> 
>> The problem is here 
>> 
>>> response.headers["X-Sendfile"] = filename
>>> response.headers['Content-length'] = File.size(filename)
>> 
>> File.size should fail, as rails will be in Dir.pwd == Rails.root, and as 
>> such, File.size(filename) for filename with no path should fail. This 
>> should be raising Errno::ENOENT.
> 
> but tucker, i am getting failed to allocate memory and that is also of 
> second time

You confuse me for someone that doesn't read. You also clearly don't understand 
"should", or what I've just told you.
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] problem in downloading large file with apache+mongrel

[James Tucker]

On 22 Sep 2010, at 12:00, Amit Tomar wrote:
> luis i change my code as you said
> 
> filename =  "#...@stream.location}"
> response.headers['Content-Type'] = "application/force-download"
> response.headers['Content-Disposition'] = "attachment; 
> filename=\"#{File.basename(filename)}\""

The problem is here 

> response.headers["X-Sendfile"] = filename
> response.headers['Content-length'] = File.size(filename)

File.size should fail, as rails will be in Dir.pwd == Rails.root, and as such, 
File.size(filename) for filename with no path should fail. This should be 
raising Errno::ENOENT.

___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] [Ruby Forum] Message from user amit-singh

[James Tucker]

My guess is middleware is trying to load the file to determine the etag or such.
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] scaling unicorn

[James Tucker]

What was the request rate and total bandwidth flowing at your peak?

How far is that from your theoretical potential on the box?

On 21 Jun 2010, at 16:58, snacktime wrote:

> Interested in some feeback on this (does it sound right?), or maybe
> this might be of interest to others.
> 
> We are launching a new facebook app in a couple weeks and we did some
> load testing over the weekend on our unicorn web cluster.  The servers
> are 8 way xeon's with 24gb ram.  Our app ended up being primarily cpu
> bound.  So far the sweet spot for the number of unicorns seems to be
> around 40.  This seemed to yield the most requests per second without
> overloading the server or hitting memory bandwidth issues.  The
> backlog is at the somaxconn default of 128, I'm still not sure if we
> will bump that up or not.  Increasing the number of unicorns beyond a
> certain point resulted in a noticable drop in the requests per second
> the server could handle.   I'm pretty sure the cause is the box
> running out of memory bandwidth.  The load average and resource usage
> in general (except for memory) would keep going down but so did the
> requests per second.  At 80 unicorns the requests per second dropped
> by more then half.  I'm going to disable hyperthreading and rerun some
> of the tests to see what impact that has.
> 
> Chris
> ___
> Mongrel-users mailing list
> Mongrel-users@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] VirtualHost like in the Apache-PHP world

[James Tucker]

On 25 Mar 2010, at 14:52, Jonathan Rochkind wrote:

> Do you want a seperate app per domain, or do you want one app handling 
> multiple domains?
> 
> If you want a seperate app per domain, I think you'd need a mongrel per 
> domain too, I don't see any other way to do it.

map "http://domain1.com/"; do
  run App1.new
end

map "http://domain2.com/"; do
  run App2.new
end

I see a way to do it. 

> 
> If you want one app that handles multiple domains, than it's up to you to 
> write an app that can do that, and it's not a mongrel issue.
> 
> Jonathan
> 
> Nagy Tamás wrote:
>> Thank you Hassan, James!
>> 
>> So i will need one mongrel per domain. I think I can't carry out this 
>> because I have a thousands of domains.
>> 
>> I think i will try the rack-thing James mentioned. I assume i can't run 
>> individual ruby script files but whole apps, one app for one domain.
>> 
>> ooohh so hard.. :)
>> 
>> 
>> 
>> Újdonság! Megérkezett a RIGHT GUARD férfi testápolási termékcsalád! Most 25% 
>> kedvezménnyel! Akár 48 órás hatásos védelem a testszag ellen! 
>> 
>> 
>>  
> ___
> Mongrel-users mailing list
> Mongrel-users@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] VirtualHost like in the Apache-PHP world

[James Tucker]

On 25 Mar 2010, at 14:34, Nagy Tamás wrote:

> Thank you Hassan, James!
> 
> So i will need one mongrel per domain. I think I can't carry out this because 
> I have a thousands of domains.

That sounds like a recipe for out of memory errors and cross-application errors 
taking down the whole set at the same time. It also sounds like it's better 
suited to passenger.

> I think i will try the rack-thing James mentioned. I assume i can't run 
> individual ruby script files but whole apps, one app for one domain.

You can load whatever you like, and run anything that responds to #call, in the 
standard rack manner. What you won't be able to do, is load different versions 
of dependencies into the same process and expect to be bug free.

> ooohh so hard.. :)
> 
> 
> 
> Újdonság! Megérkezett a RIGHT GUARD férfi testápolási termékcsalád! Most 25% 
> kedvezménnyel! Akár 48 órás hatásos védelem a testszag ellen! 
> ___
> Mongrel-users mailing list
> Mongrel-users@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] VirtualHost like in the Apache-PHP world

[James Tucker]

On 24 Mar 2010, at 22:13, Hassan Schroeder wrote:

> 2010/3/24 Nagy Tamás :
> 
>> but i'd like to try out Mongrel..
>> so, I set the apache with mod_proxy.. i think it should forward the request
>> to mongrel, localhost:3000.. but how will the mongrel figure out where are
>> the ruby apps? it will get a request for example.tld or example6.tld and
>> example4.tld at the same time.. .. ??
> 
> You need to have at least one mongrel instance per domain, and set
> up your httpd proxy accordingly.

Actually, you can use rack to do it:

map "http://domain1/"; do
  run App1
end

map "http://domain2/"; do
  run App2
end

Then specify multiple vhost aliases in the apache proxy configuration

> 
> -- 
> Hassan Schroeder  hassan.schroe...@gmail.com
> twitter: @hassan
> ___
> Mongrel-users mailing list
> Mongrel-users@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users



smime.p7s
Description: S/MIME cryptographic signature
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] Patch to fix mongrel_service bugs with Windows Vista/2008 support and log file location

[James Tucker]

On 11 Sep 2009, at 18:20, Luis Lavena wrote:

On Fri, Sep 11, 2009 at 7:03 PM, Daniel Gies  
 wrote:

Hello Mongrel users,

I¹m a software developer at BigFix ( http://www.bigfix.com ) and we  
use
Mongrel in one of our products.  We have made changes to the  
mongrel_service

component to address problems outlined in Mongrel tickets 44 and 54:
http://mongrel.rubyforge.org/ticket/44
http://mongrel.rubyforge.org/ticket/54

The modifications are as follows:
Change the way mongrel_service detects its runtime environment to  
use a
method compliant with the Windows Vista security model.  This  
modification

consists primarily of replacing elevated privilege code for process
inspection with a command-line argument.
Check for the ³-l² command line option and use that for the log  
file instead

of using a hard-coded location.  The ³-P² option was not implemented.

In accordance with term 2.a of the Ruby License (
http://www.ruby-lang.org/en/LICENSE.txt ), we are making our  
changes freely

available to the open source community.
The patch is based off of root/branches/stable_1-2/projects/ 
mongrel_service

as of 2009-09-10, also known as mongrel_service 0.35.
If you would prefer to receive the patch in an email attachment  
please let

me know.


Hello Daniel,

Thank you for your patches!

I'll have to invest on mongrel_service the upcoming week to fix this
issue and integrate other pull request at GitHub repository.

While I can do manually, would you mind review your patches against
mongrel_service repository at GitHub?

http://github.com/fauna/mongrel_service

Either way, I'm going to take care of this.

Thank you once again for your contributions.


Hey Luis,

Just thought I'd mention, I've got some (very simple) service wrappers  
using win32-service in http://libraggi.rubyforge.org/ these days, and  
the codes are up on my github. I know those libs used to be a problem  
for us, but combined with the rubyw_helper they are very stable these  
days (in production in a couple of environments). Of course, there's a  
side note that win32-service depends on libs that compile with windows  
system error handling extensions, which won't build on mingw (gcc  
specifically), so some of the dependencies have to be gem installed  
with --platform x86-mswin32 for the prebuilds.


You may not be willing to switch over, given the above, and that the  
freebasic wrappers have been working well for such a long time, but I  
thought I'd mention it.


Regards,

James



--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to  
add,

but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users


___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] Mongrel_rails memory usage ballooning rapidly

[James Tucker]

200-300MB is not unusual for a rails application.

Is it growing past that?

Maybe use 3 or 4 mongrels, rather than 8.

On 22 Jul 2009, at 09:18, Navneet Aron wrote:


Hi Folks,
 I've a rails application in production environment.The rails app  
has both a website as well as REST apis that are called by mobile  
devices. It has a apache front end and  8 mongrel_rails running when  
I start the cluster. Each of them stabalize around 60 MB initially.  
After that pretty rapidly one or two of the mongrel_rails process's  
memory usage climbs upto 300 MB (within 30 min). If I leave the  
system for 2 hours, pretty much all the processes would have reached  
upwards of 300 MB .  (Also there are times when I can leave the  
system running pretty much the whole day and memory usage will NOT  
go upto 300MB).


The entire site becomes really slow and I've to restart the server.  
We wanted to profile the app, but we couldn't find a good profiling  
tool for Ruby 1.8.7


Can someone please suggest how should we go about troubleshooting  
this?


Thanks,
Navneet
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users


___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] Mongrel as a Windows Service (not running Rails)

[James Tucker]

You have a few options for your situation:

1) check mongrel_service FreeBASIC code (which is the actual service)
and implement your own variation.
(willl take you some time)

2) look into options like svrany or firedaemon, which let you run
console applications as services.


3) http://github.com/raggi/ruby_service_helper/, and I recommend you  
also use http://github.com/raggi/rubyw_helper/, these work with win32- 
service-0.6.1. If you're on a mingw build of ruby, you may need to gem  
install --platform=x86-mswin32-60 win32-service first, as gcc can't  
deal with SEH.


The above provides something that's relatively easy to prototype, but  
also still mostly debuggable:


http://pastie.textmate.org/private/rsnvpy5eogjhicawttjgra (untested)

HTH
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] HTTP parse error due to an extra percent sign

[James Tucker]

On 8 Jan 2009, at 17:51, Jonathan Rochkind wrote:


We have contracts with literally hundreds of such providers.


Most of which do the right thing.

This one provider in particular that sends the bad URL is a  
particular large company (EBSCO), with billions of dollars in  
revenue, and thousands of customers of which we are just one.


It should generally take one, or a handful of lines of code to  
properly escape urls. It sounds like they can definitely afford to do  
this.


Most of their other customers are not using mongrel-fronted (or  
Rails at all) solutions for OpenURL link resolving; the solutions  
they are using manage to deal with the faulty URLs.


There are plenty of stories of how mongrels conformance to the RFC has  
raised similar concerns. Indeed this happened with Google too. Some of  
the stories can be found on Zeds blog. AFAIK, Google fixed their stuff.


 Now ours does too, with some ugly apache/perl hacking in front of  
mongrel.


As David said, your solution is a valid one. Changing mongrel is not.

I complained both privately and publically about this. But the world  
is not always as we would like it, and sometimes our software needs  
to deal with incoming data that is not standards compliant, just the  
way it is.


Understood, nonetheless, you should keep trying, as we all should, to  
improve this world.


___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] HTTP parse error due to an extra percent sign

[James Tucker]

On 7 Jan 2009, at 21:31, Jonathan Rochkind wrote:

Yes, the third party sending me this information in a query string  
was doing it in a way that was illegal and violated standards, but  
they are more powerful than I, and I can not make them change their  
behavior, and I need to handle those URLs anyway.


This is a sad day, they won.

Next time, scream from the rooftops, unless you already signed your  
free speech away, that is.


Printing an RFC and highlighting the relevant clauses and handing that  
to a level higher than disgruntled developers is often relatively  
effective too.

___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] speed curiosity

[James Tucker]

On 30 Aug 2008, at 21:57, Roger Pack wrote:


As a note--using the mongrel example from

http://mongrel.rubyforge.org/web/mongrel/files/README.html and Mongrel
1.1.5

It yielded (for me) ~800 req/s [running ab -n 1000 -c 1
http://localhost:3000/test]

and if I changed
   out.write("hello!\n")

to
   out.write("hello!\n"*10_000)


AFAIK that's not the fastest of operations.




it yielded ~300 req/s.

I was unable to get evented mongrel to run so wasn't able to compare  
the

two.


Doing a little bit of investigating, kcachegrind+ruby-prof points the
latency to http_response.rb line 137

   @socket.write(data)

Experimenting by changing this line haphazardly to

 while data and data.length > 0
   wrote = @socket.write_nonblock(data)
   data = data[wrote..-1]
 end

yielded ~938 req/s [AFAICT]


Thin and ebb both write more like this.





Thoughts?

-=R
--
Posted via http://www.ruby-forum.com/.
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users


___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] Some architecture questions for my mongrelian friends

[James Tucker]

On 16 May 2008, at 21:34, [EMAIL PROTECTED] wrote:


Hey,

I'm working on a project, and mongrel may be part of the stack, but  
I've got some more general questions and ideas I'm hoping to run by  
this list. The people on this list have a broader knowledgebase and  
more experience than any place else I know - plus a general  
friendliness and willingness to help!


I'm working with a company who has a really antique application  
stack. Literally from 1998. IIS + ASP + MS SQL server. They want me  
to help "modernize" things. In the abstract I'd say, "get a really  
good .NET team and go that route." But they want me to help. All I  
work in these days is Ruby. And that's all I want to work in. :)


So my questions are like this:

1) Can I in good conscience start migrating this company from IIS/ 
ASP to Mongrel/Ruby/Merb/ORM (or something like that)? They have  
roughly 2-3M page views per month.


It can be done. Others can better tell you on the hosting costs, but  
there are sites doing this and apparently making money still :)


1.a) No matter how good they think I am, wouldn't it be smarter to  
move forward with M$ since that's what they've got already? I don't  
want to be the guy who screws them deeper into the hole by really  
confusing their stack.


The biggest problems you'll hit with packing Ruby into a production  
windows stack are:

 * No decent web proxy, at least, not like nginx and friends.
 * Apache I've benched up to 8000/s on a quad core on Windows with a  
hello world app and the one click installer. If I remember correctly  
the app was a camping one, so not too bad.

 * The mingw builds of ruby are about 30% faster than the OCI build.
 * You'll have to wait for the release of a decent service manager  
for windows, or roll your own monitor for your services.
 * Logging integration is faster to do unintegrated - analogger has  
been working great for us, but you'll need the trunk build from the  
mercurial repo that has my patches.
 * Windows ruby socket implementations (and ruby herself) have low  
limits for the number of file descriptors. JRuby can help I think, we  
proxy out our services where appropriate.


I hate it when new dudes come in with their "stack" and bias  
development based on their preferences withou considering what's  
already there. I'd rather walk away from this if Microsoft is really  
their odds-on smart choice (i.e. I don't need the money - I have  
some personal relations that led me here). All I want is the company  
to be successful.


JRuby could be a good option for you due to the double-stack style of  
it. You could switch to a Java stack underneath if you need speed /  
less magic / java programmers / whatever.


2) Their MS SQL setup is relatively fine. Lots of wacky stored procs  
which bug me but mostly it's fine. Am I crazy to try to run MS SQL  
against Ruby/ORM? Seems like there are some people doing it?


Personally I'd go PostgreSQL, if they really like their stored  
procedures, that's not going to play nice with most ruby ORMs. We have  
a product that seals the entire DB behind stored procedures for  
security reasons (extreme case). The front end for that app is rails,  
and we have a single point of control that basically replaces  
activerecord (acts as our virtual-ORM) with about 200 lines of ruby,  
and circa 10k lines of (largely generated) plpgsql.


3) If I do this, I'd plan to segment this site into two separate  
boxes and run the Ruby on a Linux box (and maybe outsource that  
management to a group like EngineYard). Then have the LB's split  
traffic between the boxes based on url patterns. Again: crazy?  
unwise? Currently they're at rackspace which knows poodle about Ruby/ 
Mongrel afaict.


If you want managed services, then yeah, go for a ruby company. There  
are a lot of differences to deploying ruby fast than some places are  
used to or can envisage. As far as splitting traffic goes, eugh, well  
just think about it carefully.


Context: The front-end site is not impossibly complex. But there is  
"deep" integration with some backend admin processes which run a  
large part of the business: some crm, PPC, finance/accounting, email  
and billing: all partially implemented and built in hand coded ASP.  
It's a real tangle and it breaks all the time right now. I want to  
get most of these processes out into third party systems with much  
narrower points of contact between the production DB's and the  
specific admin services. This can only happen incrementally over  
time. This is in addition to the front-end websystem migration.


Do one piece at a time. It sounds like the kind of scenario where  
you'll be needing to refactor the ASP in order to keep things stable  
whilst migrating pieces of a large system.


Budgets for this work are not tiny but not enormous. Ditto  
timeframe. Maybe $250k over 6-8 months.


That's not a lot of money to train up people on ruby. Sure ruby is  
simple from the outset, but some 

Re: [Mongrel] Mongrel performance study

[James Tucker]

On 8 Apr 2008, at 08:20, Zed A. Shaw wrote:
>
> I don't think it's unix sockets vs. ip sockets.  These days those
> aren't really that much faster than a localhost connection thanks to
> advancements in performance for internal IPC.
>
> Still, 4% increase for all that work done in Thin is kind of lame.
> These kinds of results keep bringing me back to the real problem:
> Ruby.  It's not what IO loop you use, or threading model, or socket
> type, but more just that Ruby is slow and it's processing sucks.

I believe this is all correct, however there are other advantages to  
unix sockets for certain setups. They're easier to secure (this is not  
always relevant, but it can be). In some scenarios they can be easier  
to manage too (you can glob them, it's harder to 'glob' using  
netstat). I know that was a discussion about performance, this is just  
my $0.02 :-)

___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] mongrel garbage collection

[James Tucker]

On 25 Mar 2008, at 17:05, Steve Midgley wrote:
> At 03:41 AM 3/25/2008, [EMAIL PROTECTED] wrote:
>> Date: Tue, 25 Mar 2008 10:40:50 +
>> From: James Tucker <[EMAIL PROTECTED]>
>> Subject: Re: [Mongrel] mongrel garbage collection
>> To: mongrel-users@rubyforge.org
>> Message-ID: <[EMAIL PROTECTED]>
>> Content-Type: text/plain; charset="us-ascii"
>> [snip]
>> Also, implicit OOMEs or GC runs quite often DO NOT affect the
>> extensions correctly. I don't know what rmagick is doing under the
>> hood in this area, but having been generating large portions of
>> country maps with it (and moving away from it very rapidly), I know
>> the GC doesn't do "The Right Thing".
>> [snip]
>
> Hi James,
>
> My understanding with RMagick is that it is hooking the Imagemagick
> libs directly in process. As a result, memory is not always freed when
> you'd expect it to be. I haven't read up on the details, having chosen
> to just use out of process image management, but you might find this
> link interesting - in it, there is a claim that the latest releases of
> RMagick do *not* in fact leak any memory and that running a full GC
> manually will reclaim all memory it uses after the references are out
> of scope.

Thank you for kindly ensuring that I got this. We actually moved  
completely away from anything ImageMagick based. There really wasn't  
any sensible way to 'fix' it.

Whilst destroy! looks ok, even when doing what we were (high res  
tiling, covering around 250 square miles), we found performance was  
fine and we could avoid all allocation issues by using the crazy  
thread solution (Thread.new { loop { sleep some_time; GC.start } }).

This is all good in most scenarios but then there are times when  
running a framework like eventmachine, where threads (yes, even green  
ones) can be total performance killers too. Mind you, under rails,  
there's always a linear reaction run, so I'm not going to speculate  
more on that detail. It's also OT for here, mostly...

> http://rubyforge.org/forum/forum.php?thread_id=1374&forum_id=1618
>
> Steve

Thanks again,

James.

P.S. Personally, if I was coming up against this problem today, I'd  
drop out to a separate process, driven by something like background  
job if under rails. If under pure ruby, I'd use drb or eventmachine +  
a marshalling protocol, depending on specific requirements. The  
biggest issue for our old project was hitting swap / page file. Image  
rendering, when you're already working on the per-pixel layer, is  
really easy to divide up, though, so optimizing for speed is pretty  
easy really.

When it comes to background concurrent scheduling, staying away from  
ACID can really help, too, but that really is another topic for  
another time. Lets just say, allow slack, and life will be easier if  
you ever hit a silly scale. I've seen people trying to recover broken  
ACID implementations by trawling logs, and my god, tearful.
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] mongrel garbage collection

[James Tucker]

On 25 Mar 2008, at 15:26, Kirk Haines wrote:
> On Tue, Mar 25, 2008 at 4:40 AM, James Tucker <[EMAIL PROTECTED]>  
> wrote:
>> Forgive me for not having read the whole thread, however, there is  
>> one thing
>> that seems to be really important, and that is, ruby hardly ever  
>> runs the
>> damned GC. It certainly doesn't do full runs nearly often enough  
>> (IMO).
>
> There's only one kind of garbage collection sweep.  And yeah,
> depending on what's happening, GC may not run very often.  That's not
> generally a problem.

Sure, inside ruby there's only one kind of run, but

>> Also, implicit OOMEs or GC runs quite often DO NOT affect the  
>> extensions
>> correctly. I don't know what rmagick is doing under the hood in  
>> this area,
>> but having been generating large portions of country maps with it  
>> (and
>> moving away from it very rapidly), I know the GC doesn't do "The  
>> Right
>> Thing".
>
> There should be no difference between a GC run that is initiated by
> the interpreter and one that is initiated by one's code.  It ends up
> calling the same thing in gc.c.  Extensions can easily mismanage
> memory, though, and I have a hunch about what's happening with
> rmagick.

I just realised the obvious truth, that ruby isn't actually running  
the GC under those OOME conditions.

>> First call of address is GC_MALLOC_LIMIT and friends. For any small  
>> script
>> that doesn't breach that value, the GC simply doesn't run. More  
>> than this,
>> RMagick, in it's apparent 'wisdom' never frees memory if the GC  
>> never runs.
>> Seriously, check it out. Make a tiny script, and make a huge image  
>> with it.
>> Hell, make 20, get an OOME, and watch for a run of the GC. The OOME  
>> will
>> reach your code before the GC calls on RMagick to free.
>>
>> Now, add a call to GC.start, and no OOME. Despite the limitations  
>> of it
>> (ruby performance only IMO), most of the above experience was built  
>> up on
>> windows, and last usage was about 6 months ago, FYI.
>
> My hunch is that rmagick is allocating large amounts of RAM ouside of
> Ruby.  It registers its objects with the interpreter, but the RAM
> usage in rmagick itself doesn't count against GC_MALLOC_LIMIT because
> Ruby didn't allocate it, so doesn't know about it.

Yup, it's ImageMagick, un-patched and they don't provide afaik a  
callback to replace malloc, or maybe that's an rmagick issue.

> So, it uses huge amounts of RAM, but doesn't use huge numbers of
> objects.  Thus you never trigger a GC cycle by exceeding the
> GC_MALLOC_LIMIT nor by running our of object slots in the heap.  I'd
> have to go look at the code to be sure, but the theory fits the
> behavior that is described very well.

Right, in fact, I think the OOME actually comes from outside of ruby  
(unverified), and ruby can't or won't run the GC before going down. As  
the free() calls inside RMagick / ImageMagick aren't happening without  
calling GC.start. The GC.start call, somewhere/how is being used to  
trigger frees in the framework. Personally, this is bad design, and  
the really common complaints may also suggest so, however, I don't  
know what their domain specific issues and limitations are. Maybe it's  
an ImageMagick thing.

Creating an OOME inside ruby, the interpreter calls on GC.start prior  
to going down. I started talking to zenspider about this stuff, and  
eventually he just pointed me at gc.c, fair enough. I still hold the  
opinion that an OOME hitting the interpreter (from whatever source)  
should attempt to invoke the GC. Of course, a hell of a lot of  
software doesn't check the result of a call to malloc(), tut tut.

Tool: http://ideas.water-powered.com/projects/libgreat

> I don't think this is a case for building GC.foo memory management
> into Mongrel, though.  As I think you are suggesting, just call
> GC.start yourself in your code when necessary.  In a typical Rails app
> doing big things with rmagick, the extra time to do GC.start at the
> end of the image manipulation, in the request handling, isn't going to
> be noticable.

Absolutely right, and yes, this is my opinion.

>> But that's not really the overall point.  My overall point is how to
>> properly handle a rails app that uses a great deal of memory during  
>> each
>> request.  I'm pretty sure this happens in other rails applications  
>> that
>> don't happen to use 'RMagick'.
>>
>> Personally, I'll simply say call the GC more often. Seriously. I  
>

Re: [Mongrel] mongrel garbage collection

[James Tucker]

Forgive me for not having read the whole thread, however, there is one  
thing that seems to be really important, and that is, ruby hardly ever  
runs the damned GC. It certainly doesn't do full runs nearly often  
enough (IMO).


Also, implicit OOMEs or GC runs quite often DO NOT affect the  
extensions correctly. I don't know what rmagick is doing under the  
hood in this area, but having been generating large portions of  
country maps with it (and moving away from it very rapidly), I know  
the GC doesn't do "The Right Thing".


First call of address is GC_MALLOC_LIMIT and friends. For any small  
script that doesn't breach that value, the GC simply doesn't run. More  
than this, RMagick, in it's apparent 'wisdom' never frees memory if  
the GC never runs. Seriously, check it out. Make a tiny script, and  
make a huge image with it. Hell, make 20, get an OOME, and watch for a  
run of the GC. The OOME will reach your code before the GC calls on  
RMagick to free.


Now, add a call to GC.start, and no OOME. Despite the limitations of  
it (ruby performance only IMO), most of the above experience was built  
up on windows, and last usage was about 6 months ago, FYI.


On 24 Mar 2008, at 20:37, Luis Lavena wrote:
On Mon, Mar 24, 2008 at 4:59 PM, Scott Windsor <[EMAIL PROTECTED]>  
wrote:
On Mon, Mar 24, 2008 at 12:18 PM, Luis Lavena  
<[EMAIL PROTECTED]> wrote:





On Mon, Mar 24, 2008 at 3:58 PM, Scott Windsor  
<[EMAIL PROTECTED]> wrote:




You're using *RMagick*, not ImageMagick directly. If you used the
later (via system calls) there will no be memory leakage you can  
worry

about.


You're correct - I'm using 'RMagick' - and it uses a large amount  
of memory.

But that's not really the overall point.  My overall point is how to
properly handle a rails app that uses a great deal of memory during  
each
request.  I'm pretty sure this happens in other rails applications  
that

don't happen to use 'RMagick'.


Personally, I'll simply say call the GC more often. Seriously. I mean  
it. It's not *that* slow, not at all. In fact, I call GC.start  
explicitly inside of by ubygems.rb due to stuff I have observed before:


http://blog.ra66i.org/archives/informatics/2007/10/05/calling-on-the-gc-after-rubygems/ 
   - N.B. This isn't "FIXED" it's still a good idea (gem 1.0.1).

http://zdavatz.wordpress.com/2007/07/18/heap-fragmentation-in-a-long-running-ruby-process/

Now, by my reckoning (and a few production apps seem to be showing  
emperically (purely emperical, sorry)) we should be calling on the GC  
whilst loading up the apps. I mean come on, when are a really serious  
number of temporary objects being created. Actually, it's when  
rubygems loads, and that's the first thing that happens in, hmm,  
probably over 90% of ruby processes out there.




Yes, I faced huge memory usage issues with other things non related to
image processing and found that a good thing was move them out of the
request-response cycle and into a out-of-bound background job.



So far, running the GC under fastcgi has given me pretty good  
results.  The
zombing issue with fast cgi is a known issue with mod_fastcgi and  
I'm pretty

sure unrelated to RMagick or garbage collection.



Yes, but even you "reclaim" the memory with GC, there will be pieces
that wouldn't be GC'ed ever, since the leaked in the C side, outside
GC control (some of the RMagick and ImageMagick mysteries).


Sure, but leaks are odd things. Some processes that appear to be  
leaking are really just fragmenting (allocating more ram due to lack  
of 'usable' space on 'the heap'. Call the GC more often, take a 0.01%  
performance hit, and monitor. I bet it'll get better. In fact, you can  
drop fragmentation the first allocated segment significantly just by  
calling GC.start after a rubygems load, if you have more than a few  
gems.




Can you tell me how you addressed the "schedule" of the garbage
collection execution on your previous scenario? AFAIK most of the
frameworks or servers don't impose to the user how often GC should  
be

performed.


In fact there are many rubyists who hate the idea of splatting  
GC.start into processes. Given what I've seen, I'm willing to reject  
that notion completely. Test yourself, YMMV.


FYI, even on windows under the OCI, where performance for the  
interpreter sucks, really really hard, I couldn't reliably measure the  
runtime of a call to GC.start after loading rubygems. I don't know  
what kind of 'performance' people are after, but I can't see the point  
in not running the GC more often, especially for 'more common' daemon  
load. Furthermore, hitting the kernel for more allocations more often,  
is actually pretty slow too, so this may actually even result in  
faster processes under *certain* conditions.


Running a lib like RMagick, I would say you *should* be doing this,  
straight up, no arguments.




In the previous scenario I was using fast_cgi with rails.  In my  
previous

reply I provided a link to t

Re: [Mongrel] consolidated log file?

[James Tucker]

On 13 Feb 2008, at 21:56, Ed Hickey wrote:

> At least in my case, the issue is definitely related to the size of
> the log.  All the sapped memory gets freed as soon as I truncate the
> production.log.
>
> I tried SyslogLogger today
> (http://rails-analyzer.rubyforge.org/hacks/classes/SyslogLogger.html)
> but couldn't get the rails log to only show up in one place when
> following the README's directions  It was logging to
> /var/log/production.log as well as /var/log/messages.  Server load
> jumped up to 2, I assume because of all the disk IO.
> Anyone who's deployed SyslogLogger and has insight, comments welcome.
>
> Also, anyone tried Analogger?  http://analogger.swiftcore.org/

I'm not running it in production yet, but we're using it in a  
development environment for our application. I sent Kirk Haines a  
patch which implements a Logger style api which you can hand to the  
rails log config options. I believe this will be coming with the  
release that is coming up soon.

It's been working just fine under development for our group, and we  
will be going production with it shortly.

>
>
> ed
>
>
> On Feb 13, 2008 3:13 PM, Joey Geiger <[EMAIL PROTECTED]> wrote:
>> So, it's the size of the LOG that is causing the issue?
>>
>> I'm rotating every night anyway using logrotate, but I was thinking
>> about moving to syslogger if I could find a nice how-to.
>>
>>
>> On Feb 13, 2008 1:11 PM, John Joseph Bachir <[EMAIL PROTECTED] 
>> > wrote:
>>> On Feb 13, 2008 10:50 AM, Sean Brown <[EMAIL PROTECTED]>  
>>> wrote:
>>>
 If anyone would
 like to see our logrotate configuration, let me know.
>>>
>>> I would be interested in seeing your config, thanks for offering.
>>> Perhaps it's worthy of a new thread?
>>>
>>> --
>>> John Joseph Bachir
>>> http://blog.johnjosephbachir.org
>>> http://lyceum.ibiblio.org
>>> http://dissent.cc
>>> http://jjb.cc
>>> ___
>>>
>>> Mongrel-users mailing list
>>> Mongrel-users@rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>>
>> ___
>> Mongrel-users mailing list
>> Mongrel-users@rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>
> ___
> Mongrel-users mailing list
> Mongrel-users@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] deployment survey

[James Tucker]

On 3 Jan 2008, at 13:45, Evan Weaver wrote:

> Hello Mongrels,
>
> Building on the last messages about Fastthread, can we get a detailed
> survey of the different ways people are deploying their applications?
> It will help with near-future Mongrel development.
>
> Please include the following things:
>
> * Framework, if any (Camping, Merb, Rails, Nitro, Ramaze, IOWA,  
> Rack...)

Rails mostly, some camping, some ramaze in the works.

> * Mongrel version

Latest, after security release.

> * Mongrel handlers used (rails, dirhandler, camping, cgiwrapper...)

Standard framework handlers, and a couple of nice Racks.

> * How many mongrel routes and handlers per route registered (if you
> don't know, it's probably <= 2)

Indeed.

> * Any Mongrel plugins used (mongrel_upload_progress, mongrel_gzip,
> mongrel_cow_cluster, mongrel_experimental...)

None to date.

> * Mongrel runners used (mongrel_rails, mongrel::cluster,
> mongrel_service, RV, others... please be *very specific* about which
> options of the runner you use. For example, some people use
> mongrel::cluster but only for the --clean functionality, not for the
> clustering).

Standard development runners (script/server) are used by most of our  
devs. One server uses mongrel_rails + the Swiftiply patches for  
evented mongrel.

> * Number of mongrels per server per app

Windows 2003 Server we setup via Apache on a quad core, with 10  
mongrels for the best performance. I had numbers somewhere, but IIRC  
it reached 4-8k req/s.
On FreeBSD and Debian, we're running nginx (and it's great), I don't  
have performance numbers for these configurations, as we're more  
focused on static file handling direct out of nginx there.

> * Monitoring system (runit, monit, god...)

Windows Services + a custom service wrapper.
djbs daemontools.

God is no good for us as it's not cross platform, same goes for monit.  
daemontools falls in the same category, but it manages far more than  
just webservers for us, and has been around for quite some time.

> * Proxy or software loadbalancer, if any (apache mod_proxy_balancer,
> nginx, pen...)

mod_proxy and nginx for the respective platforms, as noted above.

> * HW loadbalancer, if any (Netscaler...)

None required to date.

> * Caching strategy (memcached fragments, memcached object, squid,
> rails page cache, rails page fragments, ESI)

Memcached, large disk caches.

> * Whether you serve media assets via mongrel itself, as opposed to
> through a webserver

We've been doing very large numbers of small assets (think tiny  
images), and nginx performs wonderfully at that.

> * Operating system including distribution or version (OS X 10.4.10,
> Ubuntu/Linux 7.10, WinXP SP2, OpenBSD 4.1...)

Production: Windows 2003 Server, FreeBSD, Debian, Solaris very briefly  
(and no more).
Development: Debian, OS X, Windows XP SP2.

> * Architecture, via 'uname -a' preferably (x86, x86_64, Sparc, PPC,
> Arm (ha), JRuby)

To date, x86 across the board, although many of the machines are x64/ 
IA64 capable.

> * CPU count

In production, we have 7 CPUs out there. Largest is a quad core.

> * Ruby version including custom distribution patches,
> (1.8.6p110+threadhooks, 1.8.5, JRuby 1.1b1, Rubinius trunk... also
> note where you got it, in case it isn't clear, for example, OS X 10.5
> built-in, Ubuntu apt, Instant Rails, direct compile from source)

Mostly 1.8.6p110 as far as I know. Windows devs run the OCI.

> * Rubygems (yes/no, version)

yes, 0.9.4+

> Please mention anything else about your system that's kind of weird,
> and anything that's been particularly troublesome regarding mongrel
> deployment.

Honestly, Mongrel has been *the* easy bit, and you can quote me on  
that should you so desire.

>
>
> Evan
>
> PS. You can get some of the Ruby information via the 'tattle' gem:
>
> $ gem install tattle --ignore-dependencies
> $ tattle report
>
>
> -- 
> Evan Weaver
> Cloudburst, LLC
> ___
> Mongrel-users mailing list
> Mongrel-users@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] Mongrel Timeout?

[James Tucker]

I would have expected you to be getting the following on load:
c:/ruby/lib/ruby/gems/1.8/gems/daemons-1.0.9/lib/daemons/daemonize.rb:103:in 
`fork': fork() function is unimplemented on this machine (NotImplementedError)

The "-d" is not actually supported on Windows, thus the warning:
> ** WARNING: Win32 does not support daemon mode.

Have you patched something to get this far?

Oliver Schneider wrote:
> Hi,
> 
> I am using a railsapp to call an external app and print some barcode-labels.
> My problem is, after a while there seems to be a timeout. Then I need to 
> stop the Mongrel-Server and restart it.
> Below you can see the error message, when i stop the server.
> 
> Has somebody seen this before?
> 
> Greetings,
> Oli
> 
> C:\etikett>mongrel_rails start -d -e production
> ** WARNING: Win32 does not support daemon mode.
> ** Daemonized, any open files are closed.  Look at log/mongrel.pid and 
> log/mongrel.log for info.
> ** Starting Mongrel listening at 0.0.0.0:3000 
> ** Starting Rails with production environment...
> C:0:Warning: require_gem is obsolete.  Use gem instead.
> ** Rails loaded.
> ** Loading any Rails specific GemPlugins
> ** Signals ready.  INT => stop (no restart).
> ** Mongrel 1.1.1 available at 0.0.0.0:3000 
> ** INT signal received.
> Wed Nov 21 11:21:20 +0100 2007: Reaping 17 threads for slow workers 
> because of 'shutdown'
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.1-mswin32/bin/../lib/mongrel.rb:216:in
>  
> `reap_dead_workers': undefined local variable or method `w' for 
> #(NameError)
> from c:/ruby/lib/ruby/gems/1.8/gems/mongrel- 
> 1.1.1-mswin32/bin/../lib/mongrel/configurator.rb:303:in `join'
> from 
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.1-mswin32/bin/../lib/mongrel/configurator.rb:303:in
>  
> `join'
> from c:/ruby/lib/ruby/gems/1.8/gems/mongrel- 
> 1.1.1-mswin32/bin/../lib/mongrel/configurator.rb:303:in `each'
> from 
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.1-mswin32/bin/../lib/mongrel/configurator.rb:303:in
>  
> `join'
> from c:/ruby/lib/ruby/gems/1.8/gems/mongrel- 
> 1.1.1-mswin32/bin/mongrel_rails:137:in `run'
> from 
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.1-mswin32/bin/../lib/mongrel/command.rb:212:in
>  
> `run'
> from c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.1-mswin32 
> /bin/mongrel_rails:281
> from c:/ruby/bin/mongrel_rails:16:in `load'
> from c:/ruby/bin/mongrel_rails:16
> Batchvorgang abbrechen (J/N)? j
> 
> 
> 
> 
> ___
> Mongrel-users mailing list
> Mongrel-users@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] how to configure mongrel_cluster in windows

[James Tucker]

Luis Lavena wrote:
> On Nov 21, 2007 3:18 AM, James Tucker <[EMAIL PROTECTED]> wrote:
>> Yeah this is an unfortunate bug with the way that 'bin' files are setup on 
>> windows. In particular, you can't call exec on a text file containing 
>> nothing but ruby. The solution is to add ".bat" or ".cmd" to the binary name 
>> in the offending call to system / exec, lead it with a call to ruby.
>>
> 
> this is the way system() works. a shame :-P

It is actually possible to make it work:

 * Implement a program which scans for, and runs any shebang on the given 
argument (roughly emulating execve).
 * set PATHEXT+=.
 * Associate extensionless files with the program you wrote for step 1.

Although that's well beyond the scope of mongrel herself.

> 
>> It's probably worth noting at this point that the "-d" option doesn't work 
>> either, so I am not sure the current cluster commands will work on windows.
> 
> Yep, I keep telling this over and over, no mongrel_cluster for windows.
> 

With regard to fork, IIRC win32-process adds this functionality.
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users

Re: [Mongrel] how to configure mongrel_cluster in windows

[James Tucker]

Shiva Kumaran wrote:
> Rafael García wrote:
>> Shiva Kumaran escribió:
>>> hi,
>>> how to configure mongrel_cluster in windows.
>>> mongrel_rails cluster::configure -e development -p 3000 -N 2 i have used
>>> this one its configured correctly then if i start the server it is
>>> throwing error.if any one knows how to configure help me
>>>
>>> with regards
>>> shiva
>>>   
>> what error?
> 
> iam getting this error
> 
> 
> D:\project\sutra>mongrel_rails cluster::start
> starting port 3000
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/lib/mongrel_cluster/init.rb
> :98:in ``': Exec format error - mongrel_rails start -d -e development -p 
> 3000 -P
>  tmp/pids/mongrel.3000.pid -l log/mongrel.3000.log (Errno::ENOEXEC)
> from 
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/lib/mongrel_cl
> uster/init.rb:98:in `start'
> from 
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/lib/mongrel_cl
> uster/init.rb:80:in `each'
> from 
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/lib/mongrel_cl
> uster/init.rb:80:in `start'
> from 
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/lib/mongrel_cl
> uster/init.rb:239:in `run'
> from 
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.1-mswin32/bin/../lib/mon
> grel/command.rb:212:in `run'
> from 
> c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.1-mswin32/bin/mongrel_ra
> ils:281
> from c:/ruby/bin/mongrel_rails:16:in `load'
> from c:/ruby/bin/mongrel_rails:16

Yeah this is an unfortunate bug with the way that 'bin' files are setup on 
windows. In particular, you can't call exec on a text file containing nothing 
but ruby. The solution is to add ".bat" or ".cmd" to the binary name in the 
offending call to system / exec, lead it with a call to ruby.

It's probably worth noting at this point that the "-d" option doesn't work 
either, so I am not sure the current cluster commands will work on windows.
___
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users