David,
Every time you create a thread, regardless of whether there is any body
in the thread, memory is required. There is copying of variables done
under the covers to make them available to thread to start with.
When you have this many jobs to process, throwing up a thread for every
job is not the way to go. Even if you have the memory requirements, the
process will probably spend more time context switching between threads
than actually doing the work.
I'd recommend putting the jobs on some sort of queue (e.g in
memory/JMS/Amazon SQS) and create a pool of threads of a reasonable size
that will pull jobs from the queue and process them.
Hope that helps.
Andy
On 03/10/2012 17:24, David G Ortega wrote:
Hi Nitai,
why putting threads inside a loop is a bad idea? Thats esencially the
same than doing a simple cfm that uses threads requested 1000000 times
in a row, would be even worse if requested several at a time.
If you see Im spawning just only 5 threads and wait them to be
disposed so at next loop everything should be fine. We are not even
talking about creating vars inside the threads, or creating 5000000
threads. Just only 5 threads at a time with no process inside the
thread!!! with that simple code the mem is eaten...
Unfortunately I cant do it other way, I have done an api thats working
for web inputs but also a Hadoop cluster is running the API to inject
recolected data from other sources.
I think there is a memory leak here.
On Wednesday, October 3, 2012 6:04:41 PM UTC+2, Nitai @ Razuna wrote:
David,
Putting thread into a loop is a very bad idea as it will completely
eat up your memory (as you have experienced). If those jobs inside
the
thread need even more memory you are in some serious trouble.
Isn't there any other way you can fire off the jobs or segment this
further? It just seams a bit crazy to fire off 1 Mil. jobs (at any
web
server that is).
Kind Regards,
Nitai
On Wed, Oct 3, 2012 at 5:43 PM, David G Ortega
<[email protected] <javascript:>> wrote:
> Hi!!
>
> This week I realized that my app can no use threads, when I do
that all the
> memory allocated to jetty is completely eaten...
> I need to run near 1000000 jobs (1 million) that runs a
multithreaded
> function... the whole process takes like 10 hours, with threads,
without
> would be 50 hours more or less. Its crucial to make it within a
day. I
> thought that maybe somehow I was creating references that could
not be
> recollected so I tried a much more simple test and I got same
results.
> In my machine takes 3 min to eat all the 4GB allocated. The cpu
is fine,
> under 20%.
> Following several useful posts I tried to manually set
ouput="false" in the
> cfc or functions but it didn't solve the leak.
>
http://blog.maestropublishing.com/fixing-a-mysterious-memory-leak-on-coldfusion
<http://blog.maestropublishing.com/fixing-a-mysterious-memory-leak-on-coldfusion>
>
> Use the cfc below as a rest service at
> http://myopenbdserver/Test.cfc?method=test
<http://myopenbdserver/Test.cfc?method=test>
>
>
> Server : Windows 2008
> Memory allocated to openbd : 4GB
> CPU : Intel Core i3 2130
2x2(HT)x3.4+ GHz
> openbd version : 2.0.2 Nightly Build
>
> Test.cfc
>
> <cfcomponent>
> <cffunction name = 'test' access='remote' returnformat='json'>
> <cfscript>
> var ii = 0;
> while(ii < 500000)
> {
> for(var i=1; i<=5; i++)
> {
> thread action="run" name="thread#ii##i#" myvar=i
> {
> console("running thread number #myvar#"); //if you use desktop
version ;)
> you will see it running
> }
> }
>
> for(var i=1; i<=5; i++)
> {
> thread action="join" name="thread#ii##i#";
> thread action="terminate" name="thread#ii##i#";
> }
>
> ii++;
> }
>
> return "done ;)";
> </cfscript>
> </cffunction>
> </cfcomponent>
>
> --
> online documentation: http://openbd.org/manual/
> http://groups.google.com/group/openbd?hl=en
<http://groups.google.com/group/openbd?hl=en>
--
See for yourself how easy it is to manage files today. Join the
revolution!
Razuna - Hosted Digital Asset Management Solution
http://www.razuna.com/
Razuna - Open Source Digital Asset Management
http://www.razuna.org/
Twitter - http://twitter.com/razunahq
Facebook - http://www.facebook.com/razunahq
<http://www.facebook.com/razunahq>
Support Platform - http://getsatisfaction.com/razuna
<http://getsatisfaction.com/razuna>
--
online documentation: http://openbd.org/manual/
http://groups.google.com/group/openbd?hl=en
--
online documentation: http://openbd.org/manual/
http://groups.google.com/group/openbd?hl=en
