I like this proposal. I was thinking along the same lines. Olga
-----Original Message----- From: Utkarsh Srivastava [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 12, 2007 4:23 PM To: [email protected] Subject: Re: [jira] Commented: (PIG-40) Memory management in BigDataBag is probably wrong The spillable objects are data bags right? We don't want bags themselves to be garbage collected. Its the tuples inside them that are the problem. I was thinking more along the lines of > class MemoryManager { List<Spillable> spillables; public static MemoryManager getMemoryManager(); > public void register(Spillable spillable){ spillables.add(spillable); } public boolean isMemoryLow(){ //Sam's stuff } private reclaimMemory(){ //traverse the list and somehow ask the biggest one to spill spillable.spill(); //garbage collect? } > } > On Dec 12, 2007, at 3:31 PM, Benjamin Reed (JIRA) wrote: > > [ https://issues.apache.org/jira/browse/PIG-40? > page=com.atlassian.jira.plugin.system.issuetabpanels:comment- > tabpanel#action_12551159 ] > > Benjamin Reed commented on PIG-40: > ---------------------------------- > > How about > > class MemoryManager { > static MemoryManager getMemoryManager(); > void add(Spillable spillable); > } > > add() will put Spillable on a WeakList so that the Spillable object > will get GCed when strong references go away. > >> Memory management in BigDataBag is probably wrong >> ------------------------------------------------- >> >> Key: PIG-40 >> URL: https://issues.apache.org/jira/browse/PIG-40 >> Project: Pig >> Issue Type: Bug >> Components: impl >> Reporter: Sam Pullara >> Assignee: Benjamin Reed >> Attachments: BigDataBag.java, MemoryUsage.java >> >> >> src/org/apache/pig/data/BigDataBag.java >> 1) You should not use finalizers for things other than external >> resources -- using them here is very dangerous and could >> inadvertantly lead to deadlocks and object resurrection and just >> decreases performance without any advantage. >> 2) Using .freeMemory() the way it is used in this class is broken. >> freeMemory() is going to return a mostly random number between 0 and >> the real amount. Adding gc() in here is a terrible performance >> burden. If you really want to do something like this you should >> using softreferences and finalization queues. > > -- > This message is automatically generated by JIRA. > - > You can reply to this email to add a comment to the issue online. >
