New submission from Andres Riancho: In pool.py, the worker function reads as follows:
http://svn.python.org/view/python/trunk/Lib/multiprocessing/pool.py?view=markup """ 68 job, i, func, args, kwds = task 69 try: 70 result = (True, func(*args, **kwds)) 71 except Exception, e: 72 result = (False, e) ... 488 if self._success: 489 return self._value 490 else: 491 raise self._value """ If an exception is raised in the function you sent to the pool, the exception you get has "raise self._value" as the last line; which is correct but useless for debugging. mp_exception_bug.py reproduces this error. This is the output: """ Traceback (most recent call last): File "mp_exception_bug.py", line 8, in <module> print p.map(f, [1,2,3]) File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map return self.map_async(func, iterable, chunksize).get() File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get raise self._value NameError: global name 'y' is not defined """ As you can imagine, "NameError: global name 'y' is not defined" is not enough in complex projects. If we apply some changes to the pool.py we could get something similar to this: """ Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/pool.py", line 98, in worker result = (True, func(*args, **kwds)) File "/usr/lib/python2.7/multiprocessing/pool.py", line 67, in mapstar return map(*args) File "mp_exception_bug.py", line 4, in f return x*y NameError: global name 'y' is not defined Traceback (most recent call last): File "mp_exception_bug.py", line 8, in <module> print p.map(f, [1,2,3]) File "/usr/lib/python2.7/multiprocessing/pool.py", line 231, in map return self.map_async(func, iterable, chunksize).get() File "/usr/lib/python2.7/multiprocessing/pool.py", line 535, in get raise self._value[0] NameError: global name 'y' is not defined """ The patch is simple but ugly: """ > import sys > import traceback 72c100,102 < result = (False, e) --- > exc_info = sys.exc_info() > tb_string = traceback.format_exc(exc_info[2]) > result = (False, (e, tb_string)) 491c532,535 < raise self._value --- > # Do something with the exception here, the simplest (but ugliest) > # thing to do is to simply print it to the console > print self._value[1] > raise self._value[0] """ Note that I tried to replace the "raise self._value[0]" with a raise with three parameters, being the last one the traceback we get using "exc_info = sys.exc_info()" but sadly it is not possible to pickle tracebacks. I understand that printing is not the best thing to do here, but I wanted to get this discussion started and come to a real solution. Thanks ---------- components: Library (Lib) files: mp_exception_bug.py messages: 187751 nosy: Andres.Riancho priority: normal severity: normal status: open title: multiprocessing exceptions with useful traceback versions: Python 3.5 Added file: http://bugs.python.org/file30009/mp_exception_bug.py _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue17836> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
