Richard, I had not realized that the file descriptor was inherited by the child. It looks like PEP443 <http://www.python.org/dev/peps/pep-0433/> seeks to overcome this with an optional flag in Python 3.3. I wonder if something similar does not exist for those of us 'stuck' in 2.x for a bit longer.
If anyone else has any ideas on the specifics of what is occurring, I would love to understand the lower level details of what the concurrent seeks/tells are doing to cause an overflow. Thanks, Jay P.S. Many thanks for the traceback code. This will likely become standard in a good bit of my test code as debugging can sometimes be a pain. On Wed, Jun 5, 2013 at 9:07 AM, Richard Oudkerk <[email protected]> wrote: > On 05/06/2013 3:30pm, Jay L. wrote: > >> I am working with the multiprocessing module attempting to get parallel >> reads working. Currently, if I open the file in each child process and >> read from an offset I achieve concurrency. If I attempt to open the >> file in the parent (via a class) and pass the class object to the >> children I get an overflow error. >> > > If you inherit the parent process's fd, then (if I remember correctly) all > the processes will share the same file offset: seeking in one process will > affect the file offset in *all* processes. It's not surprising that this > causes problems, although I can't give a precise explanation of what is > happening. > > BTW, the exception object returned to the main process will not have a > traceback attached to it. To see a full traceback you could catch, print > and reraise the exception in the worker process: > > def get_object(row): > try: > return file.get(row) > except: > # print exception with traceback then reraise > sys.excepthook(**sys.exc_info(**)) > raise > ______________________________**_________________ > concurrency-sig mailing list > [email protected] > http://mail.python.org/**mailman/listinfo/concurrency-**sig<http://mail.python.org/mailman/listinfo/concurrency-sig> >
_______________________________________________ concurrency-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/concurrency-sig
