[
https://issues.apache.org/jira/browse/DROIDS-46?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700040#action_12700040
]
Mingfai Ma commented on DROIDS-46:
----------------------------------
After studied the MT TaskMaster, I've implemented my own TaskMaster to get rid
of the memory-leak problem. The MultiThreadedTaskMaster is a bit complex and it
would be good to refactor it and add docs for readability, and also good to add
some test cases. Besides, the use of ThreadPoolExecutor actually introduce
another Queue, and the tasks are transferred from one queue to another. I think
the 2-queue design should be reviewed. (my own TaskMaster also uses 2 queues,
so i don't have suggestion yet)
for the Task Master design as a whole:
- the sequential task master is conceptually the same as a single thread
multi-thread task master. i suggest there should be one SimpleTaskMaster that
could sprawn multiple thread to do the job, and people who need
single-thread/sequential usage could specify 1 as maxThreads (which could be
the default if no parameter is provided) I think most ppl won't use single
thread anyway.
- The pausable implementation seems to make things more complex, i suggest to
make it extend a SimpleTaskMaster (if possible) and to implement the additional
interface.
> MultiThreadedTaskMaster (WorkRunner) memory leak
> ------------------------------------------------
>
> Key: DROIDS-46
> URL: https://issues.apache.org/jira/browse/DROIDS-46
> Project: Droids
> Issue Type: Bug
> Components: core
> Affects Versions: 0.01
> Reporter: Mingfai Ma
> Priority: Blocker
>
> In a Droids job that has been run for around 6 hours, it eats a lot of memory
> that cannot be free by the GC. with "jmap -histo", the report shows there are
> 1.5m instance of MultiThreadedTaskMaster$WorkerRunner and
> LinkedBlockingQueue$Node that consumes 594+475m memory. The instance cannot
> be free by any full GC (and they reside in tenured generation of the heap)
> num #instances #bytes class name
> ----------------------------------------------
> 1: 957740 1007648216 [C
> 2: 14874175 594967000
> org.apache.droids.impl.MultiThreadedTaskMaster$WorkerRunner
> 3: 14873977 475967264
> java.util.concurrent.LinkedBlockingQueue$Node
> For #1, I'm not use what's that but that may be a problem with my own
> program. But #2 and #3 shall come from Droids. I haven't checked the source
> of MultiThreadedTaskMaster yet and it would be great if the original
> developer could take a quick look to see if there are any chance that the
> WorkRunner may be referenced.
> Besides, there is another issue related to MultiThreadedTaskMaster DROIDS-43
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
