[protobuf] Protocol Buffers for Python 3.1

[Kentamanos]

I followed earlier advice for porting protobuf to Python 3.1 and I'm
pretty close to having the SVN trunk version working. Of the 174 tests
in "setup.py test", I have 4 failures and 10 errors that I'm still
working on.

I made the changes to protoc to add a Python3 output as was earlier
recommended. It fixes the metaclass issue and removes the calls to
unicode as previously recommended.

I installed "distribute" to get the "setuptools" stuff to work
properly ("distribute" is a Python3 compatible "clone" of "setuptools"
as previously mentioned).

I'm also attempting to make it specifically use "str" and "bytes"
types as opposed to just "str" for everything. My motivation is to
avoid the whole question of "is this string a string or is it really a
byte array?". I think I've got most of that sorted out (byte arrays in
the Python API have to specifically be set with something like
b'value' for instance). Is this a good idea or a bad idea? Internally
anything that's bytes I want to store as bytes, but perhaps I should
still allow setting it via a str assignment? This makes some of the
tests a bit weird to sort out if I do (there's one that makes sure you
can't set a "bytes" value as unicode(...) for instance).

My last area of concern is "text_format.py". I assume that I really
don't want to break the text representation here. Currently I have
protoc writing out the serialized proto in the Python classes as a
bytes value (b'...'), but I'm guessing that's a bad idea, since the
serialized ".proto" should always remain "text" even if it's stored in
a Python class? I know I have to handle text for all of the functions
for merging in .proto and text representations of proto. I think all
of the above 14 problems (4 failures and 10 errors) can be traced to
either _CEscape and _CUnescape so they should be my last two methods
to figure out (the whole lack of decode('string_escape') is making it
fun to figure out).

I'm pretty new to Python (more of a C#/Java/C++ in that order kind of
guy), so I can't promise it's optimal in content or performance, but I
need a Python3 version of protobuf for use in Blender 2.5 (Python3
API).

-Kent

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: Protocol Buffers for Python 3.1

[Ivo Benner]

> AIUI, setuptools (and therefore pkg_resources) have not yet been ported
> to python3 yet. I believe there are discussions about including
> pkg_resources directly in python3 (perhaps via distutils), but the PEP
> seems withdrawn.

There is a python 3 compatible fork of setuptools with a new
pkg_resources.py i guess.
http://bitbucket.org/tarek/distribute/wiki/Home

By the way, I would like to see Python 3 support in google protocol
buffers too.
At the moment iam trying to build a python3 network protocol with the
python struct module. But thats not fun. Especially if you used
protocol buffers before :)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Protocol Buffers for Python 3.1

[Monty Taylor]

Kenton Varda wrote:
> Thanks for doing this.  Unfortunately, I guess this requires forking the
> code?  There is a lot of work that needs to be done on the Python
> version and having to do them all twice will be pretty annoying.  In
> fact there is a rather large change coming which improves performance
> quite a bit which presumably we'll also want in the Python 3 version,
> but it would probably be easier to re-do the port than to try to
> integrate the change into what you have.

Agree. I can wait until that change hits and then re-do it.

> So I'm not sure what we should do with this...

I'm not sure what the best way is to maintain it moving forward. The
change to protoc was pretty small and self contained, but the python3
dir is a full-on fork of the python dir.

> On Mon, Sep 14, 2009 at 10:46 PM, Monty Taylor  > wrote:
> 
> Monty Taylor wrote:
> > Kenton Varda wrote:
> >> Does Python 3.x implement metaclasses the same way?  It looks to
> me like
> >> the metaclass may not be getting applied.
> >
> > AIUI, setuptools (and therefore pkg_resources) have not yet been
> ported
> > to python3 yet. I believe there are discussions about including
> > pkg_resources directly in python3 (perhaps via distutils), but the PEP
> > seems withdrawn.
> 
> > I've also got a diff (attaching) to the python dir that works with
> > python3. I have not patched protoc, nor have I done significant
> testing,
> > but this works:
> 
> I obviously have _way_ too much time on my hands...
> 
> Here is a _large_ patch which adds python3 support in addition to
> python2 support. It adds what I've found so far are the changes needed
> in protoc output. (the b"".decode("utf-8") vs. unicode("","utf-8") and
> metaclass = "foo" in the class specifier). It also enables the test
> suite and gets it running.
> 
> The test suite does _not_ pass. Much of the test suite (and protobuf
> itself) is dependent on transferring bytes, and with the switch from
> string()=="" and unicode() in python2 to unicode()=="" and bytes() in
> python3, there are serveral bugs in the straight-forward porting. I have
> not dug fully in to these bugs, but the test suite does now at least
> run. (albeit via python3 test.py not via python3 setup.py test)
> 
> Monty
> 
> 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Protocol Buffers for Python 3.1

[Kenton Varda]

Thanks for doing this.  Unfortunately, I guess this requires forking the
code?  There is a lot of work that needs to be done on the Python version
and having to do them all twice will be pretty annoying.  In fact there is a
rather large change coming which improves performance quite a bit which
presumably we'll also want in the Python 3 version, but it would probably be
easier to re-do the port than to try to integrate the change into what you
have.
So I'm not sure what we should do with this...

On Mon, Sep 14, 2009 at 10:46 PM, Monty Taylor  wrote:

> Monty Taylor wrote:
> > Kenton Varda wrote:
> >> Does Python 3.x implement metaclasses the same way?  It looks to me like
> >> the metaclass may not be getting applied.
> >
> > AIUI, setuptools (and therefore pkg_resources) have not yet been ported
> > to python3 yet. I believe there are discussions about including
> > pkg_resources directly in python3 (perhaps via distutils), but the PEP
> > seems withdrawn.
>
> > I've also got a diff (attaching) to the python dir that works with
> > python3. I have not patched protoc, nor have I done significant testing,
> > but this works:
>
> I obviously have _way_ too much time on my hands...
>
> Here is a _large_ patch which adds python3 support in addition to
> python2 support. It adds what I've found so far are the changes needed
> in protoc output. (the b"".decode("utf-8") vs. unicode("","utf-8") and
> metaclass = "foo" in the class specifier). It also enables the test
> suite and gets it running.
>
> The test suite does _not_ pass. Much of the test suite (and protobuf
> itself) is dependent on transferring bytes, and with the switch from
> string()=="" and unicode() in python2 to unicode()=="" and bytes() in
> python3, there are serveral bugs in the straight-forward porting. I have
> not dug fully in to these bugs, but the test suite does now at least
> run. (albeit via python3 test.py not via python3 setup.py test)
>
> Monty
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Protocol Buffers for Python 3.1

[Monty Taylor]

Kenton Varda wrote:
> Does Python 3.x implement metaclasses the same way?  It looks to me like
> the metaclass may not be getting applied.

AIUI, setuptools (and therefore pkg_resources) have not yet been ported
to python3 yet. I believe there are discussions about including
pkg_resources directly in python3 (perhaps via distutils), but the PEP
seems withdrawn.


Python 2.6.2 (release26-maint, Apr 19 2009, 01:58:18)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> __import__('pkg_resources').declare_namespace(__name__)
>>>
mord...@camelot:~/src/protobuf/protobuf/python/google$ python3
Python 3.0.1+ (r301:69556, Apr 15 2009, 17:25:52)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> __import__('pkg_resources').declare_namespace(__name__)
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named pkg_resources

However, there is this:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

That can be used to do namespace packages in python3.

You're going to run in to more problems than just that, of course. I'm
supposing you're already looking at 2to3? buffer() needs to turn in to
memoryview() - for one.

...

Ok. I got it to work.

There will need to be changes to the python emitted by protoc.
(specifically the metaclass stuff Kenton was mentioning) That should
look like this:

class Schema(message.Message, metaclass =
reflection.GeneratedProtocolMessageType):

Also, default_value="" instead of default_value=unicode("", "utf-8")

I'm attaching a schema.proto and the edited/generated code.

I've also got a diff (attaching) to the python dir that works with
python3. I have not patched protoc, nor have I done significant testing,
but this works:

mord...@camelot:~/src/drizzle/mordred/drizzled/message$ python3
Python 3.0.1+ (r301:69556, Apr 15 2009, 17:25:52)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import schema_pb2
>>> s=schema_pb2.Schema()
>>> s.name="foo"
>>> print(s.SerializeToString())
b'\n\x03foo'

Hope this helps. (Sorry for the rambling brain-dump)

Monty


> On Mon, Sep 14, 2009 at 5:21 PM, cjimison  > wrote:
> 
> 
> Hi all,
> 
> Our application has an embedded python 3.1 interpreter and we would
> really like to use protocol buffers.  I have started to make the
> protocol buffers python 3 compatible but I am running into issues with
> starting up the module.  I create a very simple proto definition in a
> file Example.proto:
> 
> package tutorial;
> 
> message MyTest {
>required string name = 1;
> }
> 
> and generated the Example_pb2.proto file with no problems at all.
> When I run the following script:
> 
> import Example_pb2
> 
> test = Example_pb2.MyTest()
> test.name  = "Bob"
> test.SerializeToString()# ERROR raise NotImplementedError
> NotImplementedError
> 
> It looks to me like it is trying to call the SerializeToString in the
> message.py message class which should be abstract.  I had to remove
> the __import__('pkg_resources').declare_namespace(__name__) call in
> python\google\__init__.py  so I think something is not getting
> "initialized" correctly.
> 
> Any ideas?
> 
> Thanks for the help!
> 
> 
> 
> > 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

# Generated by the protocol buffer compiler.  DO NOT EDIT!

from google.protobuf import descriptor
from google.protobuf import message
from google.protobuf import reflection
from google.protobuf import service
from google.protobuf import service_reflection
from google.protobuf import descriptor_pb2



_SCHEMA = descriptor.Descriptor(
  name='Schema',
  full_name='drizzled.message.Schema',
  filename='schema.proto',
  containing_type=None,
  fields=[
descriptor.FieldDescriptor(
  name='name', full_name='drizzled.message.Schema.name', index=0,
  number=1, type=9, cpp_type=9, label=2,
  default_value="",
  message_type=None, enum_type=None, containing_type=None,
  is_extension=False, extension_scope=None,
  options=None),
descriptor.FieldDescriptor(
  name='collation', full_name='drizzled.message.Schema.collation', index=1,
  number=2, type=9, cpp_type=9, label=1,
  default_value="",
  message_type=None, enum_type=None, containing_type=None,
  is_extension=False, extension_scope=None,
  options=None),
  ],
  extensions=[
  ],
  nested_types=[],  # TODO(ro

Re: Protocol Buffers for Python 3.1

[Kenton Varda]

Does Python 3.x implement metaclasses the same way?  It looks to me like the
metaclass may not be getting applied.

On Mon, Sep 14, 2009 at 5:21 PM, cjimison  wrote:

>
> Hi all,
>
> Our application has an embedded python 3.1 interpreter and we would
> really like to use protocol buffers.  I have started to make the
> protocol buffers python 3 compatible but I am running into issues with
> starting up the module.  I create a very simple proto definition in a
> file Example.proto:
>
> package tutorial;
>
> message MyTest {
>required string name = 1;
> }
>
> and generated the Example_pb2.proto file with no problems at all.
> When I run the following script:
>
> import Example_pb2
>
> test = Example_pb2.MyTest()
> test.name = "Bob"
> test.SerializeToString()# ERROR raise NotImplementedError
> NotImplementedError
>
> It looks to me like it is trying to call the SerializeToString in the
> message.py message class which should be abstract.  I had to remove
> the __import__('pkg_resources').declare_namespace(__name__) call in
> python\google\__init__.py  so I think something is not getting
> "initialized" correctly.
>
> Any ideas?
>
> Thanks for the help!
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Protocol Buffers for Python 3.1

[cjimison]

Hi all,

Our application has an embedded python 3.1 interpreter and we would
really like to use protocol buffers.  I have started to make the
protocol buffers python 3 compatible but I am running into issues with
starting up the module.  I create a very simple proto definition in a
file Example.proto:

package tutorial;

message MyTest {
required string name = 1;
}

and generated the Example_pb2.proto file with no problems at all.
When I run the following script:

import Example_pb2

test = Example_pb2.MyTest()
test.name = "Bob"
test.SerializeToString()# ERROR raise NotImplementedError
NotImplementedError

It looks to me like it is trying to call the SerializeToString in the
message.py message class which should be abstract.  I had to remove
the __import__('pkg_resources').declare_namespace(__name__) call in
python\google\__init__.py  so I think something is not getting
"initialized" correctly.

Any ideas?

Thanks for the help!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---