Many thanks for your replies Triggering gc doesn't seem to help (though the object count goes down) but reducing how much is processed at a time does. I actually didn't have to go down to a molecule-at-a-time level but made due with inputs half of the previous size. The RAM still fills up after a few files but then stays that way without overflowing the swap and crashing things.
Cheers, Adam On 27-Jun-15 16:54, Dmitri Maziuk wrote: > On 6/27/2015 5:45 AM, Greg Landrum wrote: > ... >> That's assuming that the futures code (which I don't know) isn't doing >> anything odd behind the scenes to hold onto references. > Or every mol in supplier holds a pointer to c++ dll that python vm > doesn't quite know how to garbage-collect, which keeps a > still-referenced object inside the job, which means jobs=[] creates a > new array without deleting the old jobs. Who knows. > > The first answer was the right one: process one molecule at a time. Even > better, split the input file into one-per-molecule, then use ec2 or > condor or osg to run your one-at-a-time script on all of them at once. > > Dima > > > ------------------------------------------------------------------------------ > Monitor 25 network devices or servers for free with OpManager! > OpManager is web-based network management software that monitors > network devices and physical & virtual servers, alerts via email & sms > for fault. Monitor 25 devices for free with no restriction. Download now > http://ad.doubleclick.net/ddm/clk/292181274;119417398;o > _______________________________________________ > Rdkit-discuss mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/rdkit-discuss ------------------------------------------------------------------------------ Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o _______________________________________________ Rdkit-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rdkit-discuss
