Lynton Grice wrote: > Hi there, > > I am writing a little persistant Queue class/s using sqlalchemy and > sqlite. All is going fine with the development but now I have reached > the stage where I would like to have each request spawned as a new > process (from Python's muliprocessing module).
The result of "sessionmaker()" is a class. Its defined at the module level and should be globally present in memory, and automatically available to any child processes that are spawned thereafter. It's not something that's really appropriate to be passing around even within a single process in most cases - its part of your application's globally available structure. Additionally, when you use multiprocessing, any arguments passed across the boundary of a process are pickled, sent over a pipe, and then unpickled on the other side. For this reason, it is critical that the arguments passed between parent/child are minimized in terms of their size. This is in the multiprocessing docs: http://docs.python.org/library/multiprocessing.html#programming-guidelines . So I would also argue that you really shouldn't be passing *any* significant state across process boundaries. I would pass only small easily-packageable bits of state that are just enough for the child process to reconstruct what it needs from the application's global state as well as the database. Things like classes, sessions, engines, connections and transactions are not candidates for sending over a pipe. > > The error I am getting is: "Can't pickle <class > 'sqlalchemy.orm.session.Session'>" it's not the same object as > sqlalchemy.orm.session.Session > > pickle.PicklingError: > > In my one class that is the closest to SQLite I have the following in > the INIT method: > > def __init__(self, queueConnection): > self._queue = queueConnection.queue > self._queueName = queueConnection.queue.name > self._engine = queueConnection.engine > self._sessionmaker = sessionmaker(bind=self._engine) > self._session = self._sessionmaker() > mapper(_Message, self._queue) > > And if the calling client called the "SEND" method to send a message > to the queue...the code is as follows: > > def send(self, message, timeout=10, callback=None): > start_pipe, end_pipe = Pipe() > start_pipe.send(message) > p = Process(target=self._queue_proxy.send, args= > (end_pipe,)).start() > result = p.join(timeout = timeout) > if result == False: > return "Timeout" > else: > return start_pipe.recv() > > The exception is raised when the following line of code runs above: > > p = Process(target=self._queue_proxy.send, args=(end_pipe,)).start > () > > NOTE: The "self._queue_proxy" object above is essentailly the object > that holds the reference to the "SessionMaker" above. > > Does anyone have any idea how I can get around this Pickling error? > > Any help will be greatly appreciated ;-) > > Lynton > > > -- > > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/sqlalchemy?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
