Hello,
I observe a quite strange bug that involves python's multiprocessing
library. I try to use (read only) one graph instance with several
*Multithreading.Process*. The graph is unserialized in the parent process.
Each child receives a reference to the graph. Then each child does simple
repetitive calls to *graph_tool.topology.shortest_distance*. Everything
great each child process works as fast as it can. However when the main
process executes the *hash_graphs* function presented below, each child
process hangs infinitely. The *hash_graphs* is executed prior to the
children start.
def hash_graphs(*args):
"""
Provides an edge based graph digest that can be used to invalidate old
cache.
:type args: tuple of :class:`graph_tool.GraphView`
:param args: the graphs to be hashed.
:rtype: str
:return: a hash digest of the input graph.
"""
graph_hash = hashlib.md5()
for graph in args:
graph_hash.update(gt.edge_endpoint_property(graph, graph.vp['id'],
"source").a.tobytes())
graph_hash.update(gt.edge_endpoint_property(graph, graph.vp['id'],
"target").a.tobytes())
return graph_hash.hexdigest()
I package a MWE, it is available here :
https://drive.google.com/file/d/0B5GhhBKHOKOxVnpfYTBwNDZxODA/view?usp=sharing.
To run it simply do :
tar xzf mwe.tar.gz
# run the buggy version
python3 -m mwe DO_HASH
# run as expected
python3 -m mwe
The buggy output looks like :
<MyProcess(MyProcess-1, started)> does shortest_distance from 452946 to
391015
<MyProcess(MyProcess-2, started)> does shortest_distance from 266188 to
207342
<MyProcess(MyProcess-3, started)> does shortest_distance from 514127 to
290838
<MyProcess(MyProcess-4, started)> does shortest_distance from 439705 to
87897
<MyProcess(MyProcess-5, started)> does shortest_distance from 223098 to
440593
<MyProcess(MyProcess-6, started)> does shortest_distance from 279880 to
368550
<MyProcess(MyProcess-7, started)> does shortest_distance from 108357 to
199593
<MyProcess(MyProcess-8, started)> does shortest_distance from 273888 to
275937
The expected output looks like :
<MyProcess(MyProcess-1, started)> does shortest_distance from 452946 to
391015
<MyProcess(MyProcess-2, started)> does shortest_distance from 266188 to
207342
<MyProcess(MyProcess-3, started)> does shortest_distance from 514127 to
290838
<MyProcess(MyProcess-5, started)> does shortest_distance from 223098 to
440593
<MyProcess(MyProcess-6, started)> does shortest_distance from 279880 to
368550
<MyProcess(MyProcess-1, started)> done.
<MyProcess(MyProcess-1, started)> does shortest_distance from 452946 to
391015
<MyProcess(MyProcess-2, started)> done.
<MyProcess(MyProcess-2, started)> does shortest_distance from 266188 to
207342
<MyProcess(MyProcess-4, started)> does shortest_distance from 439705 to
87897
<MyProcess(MyProcess-7, started)> does shortest_distance from 108357 to
199593
<MyProcess(MyProcess-3, started)> done.
<MyProcess(MyProcess-1, started)> done.
<MyProcess(MyProcess-3, started)> does shortest_distance from 514127 to
290838
<MyProcess(MyProcess-1, started)> does shortest_distance from 452946 to
391015
<MyProcess(MyProcess-8, started)> does shortest_distance from 273888 to
275937
<MyProcess(MyProcess-2, started)> done.
<MyProcess(MyProcess-2, started)> does shortest_distance from 266188 to
207342
<MyProcess(MyProcess-3, started)> done.
<MyProcess(MyProcess-3, started)> does shortest_distance from 514127 to
290838
<MyProcess(MyProcess-1, started)> done.
<MyProcess(MyProcess-1, started)> does shortest_distance from 452946 to
391015
<MyProcess(MyProcess-6, started)> done.
<MyProcess(MyProcess-6, started)> does shortest_distance from 279880 to
368550
<MyProcess(MyProcess-4, started)> done.
<MyProcess(MyProcess-4, started)> does shortest_distance from 439705 to
87897
<MyProcess(MyProcess-8, started)> done.
<MyProcess(MyProcess-8, started)> does shortest_distance from 273888 to
275937
<MyProcess(MyProcess-1, started)> done.
<MyProcess(MyProcess-1, started)> does shortest_distance from 452946 to
391015
<MyProcess(MyProcess-2, started)> done.
<MyProcess(MyProcess-2, started)> does shortest_distance from 266188 to
207342
<MyProcess(MyProcess-3, started)> done.
<MyProcess(MyProcess-3, started)> does shortest_distance from 514127 to
290838
<MyProcess(MyProcess-5, started)> done.
<MyProcess(MyProcess-5, started)> does shortest_distance from 223098 to
440593
<MyProcess(MyProcess-1, started)> done.
<MyProcess(MyProcess-1, started)> does shortest_distance from 452946 to
391015
<MyProcess(MyProcess-8, started)> done.
<MyProcess(MyProcess-8, started)> does shortest_distance from 273888 to
275937
<MyProcess(MyProcess-7, started)> done.
<MyProcess(MyProcess-7, started)> does shortest_distance from 108357 to
199593
<MyProcess(MyProcess-3, started)> done.
<MyProcess(MyProcess-3, started)> does shortest_distance from 514127 to
290838
...
How could I explain this behavior ?
Bests,
François.
_______________________________________________
graph-tool mailing list
[email protected]
https://lists.skewed.de/mailman/listinfo/graph-tool
