Re: [protobuf] Re: Python Client: pickling error
The multiprocessing queue uses pickle to serialize/deserialize the objects before/after sending them between processes. Since the proto message contains a C extension class, it can't be serialized or deserialized with pickle. And it doesn't need to, just use the protobuf serialization/deserialization routines. So serialize and do Q.put(serizlied), then retrieve using Q.get() and deserialize. On Tue, Apr 26, 2011 at 7:37 PM, vish vishi...@gmail.com wrote: So far what I've found is python protocol buffer objects are not serializable using Pickle. On Apr 25, 3:55 pm, vish vishi...@gmail.com wrote: Environment: CentOS 64bit Python 2.6 Protocol Buffers 2.4.0.a Python client with Native C++ support python protocol buffers installed to ~/protobuf LD_LIBRARY_PATH=~/protobuf/lib:~/protobuf/google/protobuf/src/.libs PYTHONPATH=:~/protobuf/:.:~/protobuf/google/protobuf/src/.libs:~/ protobuf/lib Hi, In a multiprocessing environment, my code is trying to add decoded messages to a shared Queue, and then pop them off at a later stage. Inserting messages in the Queue is working fine. But while fetching those messages, am getting this error: Traceback (most recent call last): File /usr/lib64/python2.6/multiprocessing/queues.py, line 242, in _feed send(obj) PicklingError: Can't pickle type 'google3.net.google.protobuf.python.internal._net_proto2___python.CMessage' : import of module google3.net.google.protobuf.python.internal._net_proto2___python failed Verified LD_LIBRARY_PATH, LD_RUN_PATH, PYTHONPATH and they all seem to be valid. Anyway help in solving this problem is appreciated. Inserting messages to Queue: while not self.stopped(): el = self.rcvr.next_message() if el is not None: print %s is the event%el Q.put(el) Popping messages off Queue: def next_message(self): Returns next event (if available) from Queue return Q.get() where Q is multiprocessing.Queue -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Python Client: pickling error
Attaching python issue: http://bugs.python.org/issue8323 On Apr 28, 11:41 am, vish vishi...@gmail.com wrote: What I did so far is to store byte content retrieved from wire onto Queue itself, and later the consumer of that message is going to construct an object out of it. On Apr 28, 11:12 am, Petar Petrov pesho.pet...@gmail.com wrote: The multiprocessing queue uses pickle to serialize/deserialize the objects before/after sending them between processes. Since the proto message contains a C extension class, it can't be serialized or deserialized with pickle. And it doesn't need to, just use the protobuf serialization/deserialization routines. So serialize and do Q.put(serizlied), then retrieve using Q.get() and deserialize. On Tue, Apr 26, 2011 at 7:37 PM, vish vishi...@gmail.com wrote: So far what I've found is python protocol buffer objects are not serializable using Pickle. On Apr 25, 3:55 pm, vish vishi...@gmail.com wrote: Environment: CentOS 64bit Python 2.6 Protocol Buffers 2.4.0.a Python client with Native C++ support python protocol buffers installed to ~/protobuf LD_LIBRARY_PATH=~/protobuf/lib:~/protobuf/google/protobuf/src/.libs PYTHONPATH=:~/protobuf/:.:~/protobuf/google/protobuf/src/.libs:~/ protobuf/lib Hi, In a multiprocessing environment, my code is trying to add decoded messages to a shared Queue, and then pop them off at a later stage. Inserting messages in the Queue is working fine. But while fetching those messages, am getting this error: Traceback (most recent call last): File /usr/lib64/python2.6/multiprocessing/queues.py, line 242, in _feed send(obj) PicklingError: Can't pickle type 'google3.net.google.protobuf.python.internal._net_proto2___python.CMessage' : import of module google3.net.google.protobuf.python.internal._net_proto2___python failed Verified LD_LIBRARY_PATH, LD_RUN_PATH, PYTHONPATH and they all seem to be valid. Anyway help in solving this problem is appreciated. Inserting messages to Queue: while not self.stopped(): el = self.rcvr.next_message() if el is not None: print %s is the event%el Q.put(el) Popping messages off Queue: def next_message(self): Returns next event (if available) from Queue return Q.get() where Q is multiprocessing.Queue -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Python Client: pickling error
So far what I've found is python protocol buffer objects are not serializable using Pickle. On Apr 25, 3:55 pm, vish vishi...@gmail.com wrote: Environment: CentOS 64bit Python 2.6 Protocol Buffers 2.4.0.a Python client with Native C++ support python protocol buffers installed to ~/protobuf LD_LIBRARY_PATH=~/protobuf/lib:~/protobuf/google/protobuf/src/.libs PYTHONPATH=:~/protobuf/:.:~/protobuf/google/protobuf/src/.libs:~/ protobuf/lib Hi, In a multiprocessing environment, my code is trying to add decoded messages to a shared Queue, and then pop them off at a later stage. Inserting messages in the Queue is working fine. But while fetching those messages, am getting this error: Traceback (most recent call last): File /usr/lib64/python2.6/multiprocessing/queues.py, line 242, in _feed send(obj) PicklingError: Can't pickle type 'google3.net.google.protobuf.python.internal._net_proto2___python.CMessage' : import of module google3.net.google.protobuf.python.internal._net_proto2___python failed Verified LD_LIBRARY_PATH, LD_RUN_PATH, PYTHONPATH and they all seem to be valid. Anyway help in solving this problem is appreciated. Inserting messages to Queue: while not self.stopped(): el = self.rcvr.next_message() if el is not None: print %s is the event%el Q.put(el) Popping messages off Queue: def next_message(self): Returns next event (if available) from Queue return Q.get() where Q is multiprocessing.Queue -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
