Anthony Foglia wrote:
[...]
Now in trying to abstract the disk storage, I'm having trouble
figuring out how to keep track of the open files, and knowing when to
close them. I'm keeping a list of weakrefs to nodes in the file, and
when they all go away, I try to close the file, but I get an exception
"exceptions.AttributeError: AttributeError("'NoneType' object has no
attribute '_f_close'",) in <function remove at ...> ignored". (remove
doesn't call _f_close, but file.close, so it must be coming from in
there. (I've attached this implementation.)
I've tracked down my problem to within
tables.group.RootGroup._g_closeDescendents. I'm using a file with only
one node. When trying to close the list of self._v_file._aliveNodes,
the list contains a dead weakref to the node. So when in closeNodes,
getNode returns None, and tries to call node._f_close() where node is None.
I see File._aliveNodes is a dictionary, containing weakrefs if
File._aliveNodes.hassoftlinks is True. It also appears to contain dead
nodes if File._aliveNodes.hasdeadnodes is also True. Both are True in
this case. But I fail to see how anything ever leaves the cache,
especially if they are weakrefs. Also, if the contents can be dead
nodes, does that mean something different than the weakref for them is dead?
I've attached the code for my cache object. I've been testing it like
(with names of files and nodes removed)...
In [1]: import cache
In [2]: c = cache.H5FileCache()
In [3]: n = c.get_node(<filename>,<node_path>)
In [4]: del n
Is File._aliveNodes part of the public API? If so, I may be able to
use it for my purposes, but from the name, I doubt it.
--
Anthony Foglia
Princeton Consultants
(609) 987-8787 x233
import sys
import weakref
import tables
class H5FileCache(object) :
class CacheItem(object) :
__slots__ = ("file","open_nodes")
def __init__(self) :
self.file = None
# Should this be a dictionary from node name to node object?
# Would make searching for pre-existing nodes easier, but the
# keyed ref would need to store both the key in this
# dictionary and the enclosing one...
self.open_nodes = set()
def __init__(self) :
def remove(wr, selfref=weakref.ref(self)) :
self = selfref()
print "Removing weakref:",wr
print "weakref has key:",wr.key
sys.stdout.flush()
if self is not None :
print self.data[wr.key].open_nodes
sys.stdout.flush()
self.data[wr.key].open_nodes.remove(wr)
print self.data[wr.key].open_nodes
sys.stdout.flush()
print "not self.data[wr.key].open_nodes", \
(not self.data[wr.key].open_nodes)
sys.stdout.flush()
if not self.data[wr.key].open_nodes :
print "Testing..."
sys.stdout.flush()
print "file.isopen =",self.data[wr.key].file.isopen
print "Closing file %s..." % str(self.data[wr.key].file.filename)
sys.stdout.flush()
self.data[wr.key].file.close()
del self.data[wr.key]
self._remove = remove
self.data = {}
def get_node(self, filename, nodename, mode='r') :
# Need to handle root group better. That will never be freed.
# Maybe whenever a dataset is freed go through the remaining and
# remove the root node...
try :
cache_item = self.data[filename, mode]
except KeyError :
# open file
cache_item = self.CacheItem()
cache_item.file = tables.openFile(filename, mode=mode)
self.data[filename, mode] = cache_item
for n_ref in cache_item.open_nodes :
n = n_ref()
if n._v_pathname == nodename :
return n
else :
n = cache_item.file.getNode(nodename)
cache_item.open_nodes.add(
weakref.KeyedRef(n, self._remove, (filename, mode)))
return n
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
