Re: [openstack-dev] [all][py3][swift][devstack] USE_PYTHON3 works! (well somewhat)

2017-01-09 Thread John Villalovos 
On Mon, Jan 2, 2017 at 1:06 PM, Davanum Srinivas  wrote:

> Folks,
>
> Other teams:
> Please consider adding DSVM jobs with USE_PYTHON3=True for your
> projects. This will hopefully help us get to our Pike goal for Python
> 3.x [10]
>
>
As a note the variable name for DSVM jobs has been changed to:
DEVSTACK_GATE_USE_PYTHON3=True
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][py3][swift][devstack] USE_PYTHON3 works! (well somewhat)

2017-01-04 Thread Davanum Srinivas 
Sorry for top post: thanks to Tim Burke, we are past this and a few
other problems. Here's the latest work:

Swift - Review from Tim : https://review.openstack.org/#/c/401397/
Swift - My review (WIP) : https://review.openstack.org/#/c/416084/
DevStack - My review : https://review.openstack.org/#/c/416064/

With all these, we get to the point where s-proxy gets "stuck" and the
tls-proxy times out, logs are here:
http://logs.openstack.org/00/412500/9/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d453c3f/logs/

Happy to turn this over to the swift team from here on.

One more good news, Thanks to the Keystone team for their very first
DevStack+python3.5 based functional test
(gate-keystone-dsvm-py35-functional-v3-only-ubuntu-xenial-nv) with
just a couple of failures left to go:
https://review.openstack.org/#/c/412500/

Thanks,
Dims

On Wed, Jan 4, 2017 at 4:49 PM, Doug Hellmann  wrote:
> Excerpts from John Dickinson's message of 2017-01-03 12:00:07 -0800:
>>
>> On 3 Jan 2017, at 10:38, Doug Hellmann wrote:
>>
>> > Excerpts from John Dickinson's message of 2017-01-03 09:02:19 -0800:
>> >>
>> >> On 2 Jan 2017, at 13:06, Davanum Srinivas wrote:
>> >>
>> >>> Folks,
>> >>>
>> >>> Short Story :
>> >>> [1] has merged in devstack, it adds support for a python 3.5 based
>> >>> up/down devstack test that just starts services and brings them down.
>> >>> see [2] for a test run.
>> >>>
>> >>> Need help from swift folks:
>> >>> Swift still needs work i have gotten as far as [3] UnpicklingError on
>> >>> ring data using [4][5][6][7]. Can someone from the swift team pick
>> >>> this up?
>> >>> Once you get this working, please add "swift" to the white list in [8]
>> >>> and remove the disable_service for swift services in [9]
>> >>
>> >> IIRC the issue is the differences between pickle, json, and arrays in py2 
>> >> vs py3 (short summary: you can't deserialize in py3 stuff that was 
>> >> serialized in py2 without first changing the py2 code).
>> >
>> > Is that right? It seems like it would be the other way around. There
>> > are newer pickle protocols in 3 that aren't available at all for 2.
>> >
>> > There are also some new options to the load() function in 3 to do
>> > things like fix imports for standard library modules that were
>> > renamed and set the right default string encoding to make it possible
>> > to change the *3* code to be able to more easily load a pickle
>> > created by 2 [1].  Is that what you meant?
>>
>> Nah, it's not the pickle protocol. It's the different way (some versions of) 
>> py27 [de]serializes arrays vs how py3 does it. The following breaks for 
>> py2.7.10 and works for py2.7.12 (.11 is untested).
>>
>> python3 -c 'import array, pickle, os, sys; pickle.dump(array.array("I", [0, 
>> 0, 0]), os.fdopen(1, "wb"), protocol=2)' | python -c 'import pickle, os, 
>> sys; print(pickle.load(os.fdopen(0, "rb")))'
>
> o_O
>
> Sigh.
>
>>
>> So maybe there are ways to always ensure doing the right thing through some 
>> combination of try/excepts, "if six.PY3" blocks, plus lots of docs for ops 
>> to make sure upgrades go smoothly, but the real solution is to not use 
>> pickle. This is doubly true when considering that the data structure will be 
>> used by non-python code, too.
>
> "Friends don't let their friends use pickle for portable persistence."
>
>>
>> This is one of the things to put on the list for py3, and it certainly is 
>> not the last.
>
> It would be good to start building up the list in the "Common Pitfalls"
> section of https://wiki.openstack.org/wiki/Python3
>
> Doug
>
>>
>> --John
>>
>> >
>> > Doug
>> >
>> > [1] https://docs.python.org/3.5/library/pickle.html#pickle.load
>> >
>> >>
>> >> We're tracking this at https://bugs.launchpad.net/swift/+bug/1644387 and 
>> >> have a patch at https://review.openstack.org/#/c/401397/, but I can't 
>> >> guarantee that services will start as soon as that patch lands. (i.e. 
>> >> it's necessary, but might not be sufficient)
>> >>
>> >> --John
>> >>
>> >>>
>> >>> Other teams:
>> >>> Please consider adding DSVM jobs with USE_PYTHON3=True for your
>> >>> projects. This will hopefully help us get to our Pike goal for Python
>> >>> 3.x [10]
>> >>>
>> >>> Please stop by #openstack-python3 channel to chat.
>> >>>
>> >>> Thanks,
>> >>> Dims
>> >>>
>> >>> [1] https://review.openstack.org/#/c/414176/
>> >>> [2] 
>> >>> http://logs.openstack.org/76/414176/11/check/gate-devstack-dsvm-py35-updown-ubuntu-xenial-nv/
>> >>> [3] 
>> >>> http://logs.openstack.org/00/412500/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f5a7fe7/logs/screen-s-proxy.txt.gz
>> >>> [4] https://review.openstack.org/#/c/412500/
>> >>> [5] https://review.openstack.org/#/c/414727/
>> >>> [6] https://review.openstack.org/#/c/416064/
>> >>> [7] https://review.openstack.org/#/c/416084/
>> >>> [8] https://review.openstack.org/#/c/414176/11/inc/python
>> >>> [9] 
>> >>> https://review.openstack.org/#/c/413775/5/jenkins/jobs/devstack-gate.yaml
>> >>> [10]

Re: [openstack-dev] [all][py3][swift][devstack] USE_PYTHON3 works! (well somewhat)

2017-01-04 Thread Doug Hellmann 
Excerpts from John Dickinson's message of 2017-01-03 12:00:07 -0800:
> 
> On 3 Jan 2017, at 10:38, Doug Hellmann wrote:
> 
> > Excerpts from John Dickinson's message of 2017-01-03 09:02:19 -0800:
> >>
> >> On 2 Jan 2017, at 13:06, Davanum Srinivas wrote:
> >>
> >>> Folks,
> >>>
> >>> Short Story :
> >>> [1] has merged in devstack, it adds support for a python 3.5 based
> >>> up/down devstack test that just starts services and brings them down.
> >>> see [2] for a test run.
> >>>
> >>> Need help from swift folks:
> >>> Swift still needs work i have gotten as far as [3] UnpicklingError on
> >>> ring data using [4][5][6][7]. Can someone from the swift team pick
> >>> this up?
> >>> Once you get this working, please add "swift" to the white list in [8]
> >>> and remove the disable_service for swift services in [9]
> >>
> >> IIRC the issue is the differences between pickle, json, and arrays in py2 
> >> vs py3 (short summary: you can't deserialize in py3 stuff that was 
> >> serialized in py2 without first changing the py2 code).
> >
> > Is that right? It seems like it would be the other way around. There
> > are newer pickle protocols in 3 that aren't available at all for 2.
> >
> > There are also some new options to the load() function in 3 to do
> > things like fix imports for standard library modules that were
> > renamed and set the right default string encoding to make it possible
> > to change the *3* code to be able to more easily load a pickle
> > created by 2 [1].  Is that what you meant?
> 
> Nah, it's not the pickle protocol. It's the different way (some versions of) 
> py27 [de]serializes arrays vs how py3 does it. The following breaks for 
> py2.7.10 and works for py2.7.12 (.11 is untested).
> 
> python3 -c 'import array, pickle, os, sys; pickle.dump(array.array("I", [0, 
> 0, 0]), os.fdopen(1, "wb"), protocol=2)' | python -c 'import pickle, os, sys; 
> print(pickle.load(os.fdopen(0, "rb")))'

o_O

Sigh.

> 
> So maybe there are ways to always ensure doing the right thing through some 
> combination of try/excepts, "if six.PY3" blocks, plus lots of docs for ops to 
> make sure upgrades go smoothly, but the real solution is to not use pickle. 
> This is doubly true when considering that the data structure will be used by 
> non-python code, too.

"Friends don't let their friends use pickle for portable persistence."

> 
> This is one of the things to put on the list for py3, and it certainly is not 
> the last.

It would be good to start building up the list in the "Common Pitfalls"
section of https://wiki.openstack.org/wiki/Python3

Doug

> 
> --John
> 
> >
> > Doug
> >
> > [1] https://docs.python.org/3.5/library/pickle.html#pickle.load
> >
> >>
> >> We're tracking this at https://bugs.launchpad.net/swift/+bug/1644387 and 
> >> have a patch at https://review.openstack.org/#/c/401397/, but I can't 
> >> guarantee that services will start as soon as that patch lands. (i.e. it's 
> >> necessary, but might not be sufficient)
> >>
> >> --John
> >>
> >>>
> >>> Other teams:
> >>> Please consider adding DSVM jobs with USE_PYTHON3=True for your
> >>> projects. This will hopefully help us get to our Pike goal for Python
> >>> 3.x [10]
> >>>
> >>> Please stop by #openstack-python3 channel to chat.
> >>>
> >>> Thanks,
> >>> Dims
> >>>
> >>> [1] https://review.openstack.org/#/c/414176/
> >>> [2] 
> >>> http://logs.openstack.org/76/414176/11/check/gate-devstack-dsvm-py35-updown-ubuntu-xenial-nv/
> >>> [3] 
> >>> http://logs.openstack.org/00/412500/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f5a7fe7/logs/screen-s-proxy.txt.gz
> >>> [4] https://review.openstack.org/#/c/412500/
> >>> [5] https://review.openstack.org/#/c/414727/
> >>> [6] https://review.openstack.org/#/c/416064/
> >>> [7] https://review.openstack.org/#/c/416084/
> >>> [8] https://review.openstack.org/#/c/414176/11/inc/python
> >>> [9] 
> >>> https://review.openstack.org/#/c/413775/5/jenkins/jobs/devstack-gate.yaml
> >>> [10] https://review.openstack.org/#/c/349069/
> >>>
> >>> -- 
> >>> Davanum Srinivas :: https://twitter.com/dims
> >>>
> >>> __
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][py3][swift][devstack] USE_PYTHON3 works! (well somewhat)

2017-01-03 Thread John Dickinson 


On 3 Jan 2017, at 10:38, Doug Hellmann wrote:

> Excerpts from John Dickinson's message of 2017-01-03 09:02:19 -0800:
>>
>> On 2 Jan 2017, at 13:06, Davanum Srinivas wrote:
>>
>>> Folks,
>>>
>>> Short Story :
>>> [1] has merged in devstack, it adds support for a python 3.5 based
>>> up/down devstack test that just starts services and brings them down.
>>> see [2] for a test run.
>>>
>>> Need help from swift folks:
>>> Swift still needs work i have gotten as far as [3] UnpicklingError on
>>> ring data using [4][5][6][7]. Can someone from the swift team pick
>>> this up?
>>> Once you get this working, please add "swift" to the white list in [8]
>>> and remove the disable_service for swift services in [9]
>>
>> IIRC the issue is the differences between pickle, json, and arrays in py2 vs 
>> py3 (short summary: you can't deserialize in py3 stuff that was serialized 
>> in py2 without first changing the py2 code).
>
> Is that right? It seems like it would be the other way around. There
> are newer pickle protocols in 3 that aren't available at all for 2.
>
> There are also some new options to the load() function in 3 to do
> things like fix imports for standard library modules that were
> renamed and set the right default string encoding to make it possible
> to change the *3* code to be able to more easily load a pickle
> created by 2 [1].  Is that what you meant?

Nah, it's not the pickle protocol. It's the different way (some versions of) 
py27 [de]serializes arrays vs how py3 does it. The following breaks for 
py2.7.10 and works for py2.7.12 (.11 is untested).

python3 -c 'import array, pickle, os, sys; pickle.dump(array.array("I", [0, 0, 
0]), os.fdopen(1, "wb"), protocol=2)' | python -c 'import pickle, os, sys; 
print(pickle.load(os.fdopen(0, "rb")))'

So maybe there are ways to always ensure doing the right thing through some 
combination of try/excepts, "if six.PY3" blocks, plus lots of docs for ops to 
make sure upgrades go smoothly, but the real solution is to not use pickle. 
This is doubly true when considering that the data structure will be used by 
non-python code, too.

This is one of the things to put on the list for py3, and it certainly is not 
the last.

--John




>
> Doug
>
> [1] https://docs.python.org/3.5/library/pickle.html#pickle.load
>
>>
>> We're tracking this at https://bugs.launchpad.net/swift/+bug/1644387 and 
>> have a patch at https://review.openstack.org/#/c/401397/, but I can't 
>> guarantee that services will start as soon as that patch lands. (i.e. it's 
>> necessary, but might not be sufficient)
>>
>> --John
>>
>>>
>>> Other teams:
>>> Please consider adding DSVM jobs with USE_PYTHON3=True for your
>>> projects. This will hopefully help us get to our Pike goal for Python
>>> 3.x [10]
>>>
>>> Please stop by #openstack-python3 channel to chat.
>>>
>>> Thanks,
>>> Dims
>>>
>>> [1] https://review.openstack.org/#/c/414176/
>>> [2] 
>>> http://logs.openstack.org/76/414176/11/check/gate-devstack-dsvm-py35-updown-ubuntu-xenial-nv/
>>> [3] 
>>> http://logs.openstack.org/00/412500/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f5a7fe7/logs/screen-s-proxy.txt.gz
>>> [4] https://review.openstack.org/#/c/412500/
>>> [5] https://review.openstack.org/#/c/414727/
>>> [6] https://review.openstack.org/#/c/416064/
>>> [7] https://review.openstack.org/#/c/416084/
>>> [8] https://review.openstack.org/#/c/414176/11/inc/python
>>> [9] 
>>> https://review.openstack.org/#/c/413775/5/jenkins/jobs/devstack-gate.yaml
>>> [10] https://review.openstack.org/#/c/349069/
>>>
>>> -- 
>>> Davanum Srinivas :: https://twitter.com/dims
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][py3][swift][devstack] USE_PYTHON3 works! (well somewhat)

2017-01-03 Thread Doug Hellmann 
Excerpts from John Dickinson's message of 2017-01-03 09:02:19 -0800:
> 
> On 2 Jan 2017, at 13:06, Davanum Srinivas wrote:
> 
> > Folks,
> >
> > Short Story :
> > [1] has merged in devstack, it adds support for a python 3.5 based
> > up/down devstack test that just starts services and brings them down.
> > see [2] for a test run.
> >
> > Need help from swift folks:
> > Swift still needs work i have gotten as far as [3] UnpicklingError on
> > ring data using [4][5][6][7]. Can someone from the swift team pick
> > this up?
> > Once you get this working, please add "swift" to the white list in [8]
> > and remove the disable_service for swift services in [9]
> 
> IIRC the issue is the differences between pickle, json, and arrays in py2 vs 
> py3 (short summary: you can't deserialize in py3 stuff that was serialized in 
> py2 without first changing the py2 code).

Is that right? It seems like it would be the other way around. There
are newer pickle protocols in 3 that aren't available at all for 2.

There are also some new options to the load() function in 3 to do
things like fix imports for standard library modules that were
renamed and set the right default string encoding to make it possible
to change the *3* code to be able to more easily load a pickle
created by 2 [1].  Is that what you meant?

Doug

[1] https://docs.python.org/3.5/library/pickle.html#pickle.load

> 
> We're tracking this at https://bugs.launchpad.net/swift/+bug/1644387 and have 
> a patch at https://review.openstack.org/#/c/401397/, but I can't guarantee 
> that services will start as soon as that patch lands. (i.e. it's necessary, 
> but might not be sufficient)
> 
> --John
> 
> >
> > Other teams:
> > Please consider adding DSVM jobs with USE_PYTHON3=True for your
> > projects. This will hopefully help us get to our Pike goal for Python
> > 3.x [10]
> >
> > Please stop by #openstack-python3 channel to chat.
> >
> > Thanks,
> > Dims
> >
> > [1] https://review.openstack.org/#/c/414176/
> > [2] 
> > http://logs.openstack.org/76/414176/11/check/gate-devstack-dsvm-py35-updown-ubuntu-xenial-nv/
> > [3] 
> > http://logs.openstack.org/00/412500/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f5a7fe7/logs/screen-s-proxy.txt.gz
> > [4] https://review.openstack.org/#/c/412500/
> > [5] https://review.openstack.org/#/c/414727/
> > [6] https://review.openstack.org/#/c/416064/
> > [7] https://review.openstack.org/#/c/416084/
> > [8] https://review.openstack.org/#/c/414176/11/inc/python
> > [9] 
> > https://review.openstack.org/#/c/413775/5/jenkins/jobs/devstack-gate.yaml
> > [10] https://review.openstack.org/#/c/349069/
> >
> > -- 
> > Davanum Srinivas :: https://twitter.com/dims
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][py3][swift][devstack] USE_PYTHON3 works! (well somewhat)

2017-01-03 Thread John Dickinson 


On 2 Jan 2017, at 13:06, Davanum Srinivas wrote:

> Folks,
>
> Short Story :
> [1] has merged in devstack, it adds support for a python 3.5 based
> up/down devstack test that just starts services and brings them down.
> see [2] for a test run.
>
> Need help from swift folks:
> Swift still needs work i have gotten as far as [3] UnpicklingError on
> ring data using [4][5][6][7]. Can someone from the swift team pick
> this up?
> Once you get this working, please add "swift" to the white list in [8]
> and remove the disable_service for swift services in [9]

IIRC the issue is the differences between pickle, json, and arrays in py2 vs 
py3 (short summary: you can't deserialize in py3 stuff that was serialized in 
py2 without first changing the py2 code).

We're tracking this at https://bugs.launchpad.net/swift/+bug/1644387 and have a 
patch at https://review.openstack.org/#/c/401397/, but I can't guarantee that 
services will start as soon as that patch lands. (i.e. it's necessary, but 
might not be sufficient)

--John




>
> Other teams:
> Please consider adding DSVM jobs with USE_PYTHON3=True for your
> projects. This will hopefully help us get to our Pike goal for Python
> 3.x [10]
>
> Please stop by #openstack-python3 channel to chat.
>
> Thanks,
> Dims
>
> [1] https://review.openstack.org/#/c/414176/
> [2] 
> http://logs.openstack.org/76/414176/11/check/gate-devstack-dsvm-py35-updown-ubuntu-xenial-nv/
> [3] 
> http://logs.openstack.org/00/412500/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f5a7fe7/logs/screen-s-proxy.txt.gz
> [4] https://review.openstack.org/#/c/412500/
> [5] https://review.openstack.org/#/c/414727/
> [6] https://review.openstack.org/#/c/416064/
> [7] https://review.openstack.org/#/c/416084/
> [8] https://review.openstack.org/#/c/414176/11/inc/python
> [9] https://review.openstack.org/#/c/413775/5/jenkins/jobs/devstack-gate.yaml
> [10] https://review.openstack.org/#/c/349069/
>
> -- 
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][py3][swift][devstack] USE_PYTHON3 works! (well somewhat)

2017-01-02 Thread Davanum Srinivas 
Folks,

Short Story :
[1] has merged in devstack, it adds support for a python 3.5 based
up/down devstack test that just starts services and brings them down.
see [2] for a test run.

Need help from swift folks:
Swift still needs work i have gotten as far as [3] UnpicklingError on
ring data using [4][5][6][7]. Can someone from the swift team pick
this up?
Once you get this working, please add "swift" to the white list in [8]
and remove the disable_service for swift services in [9]

Other teams:
Please consider adding DSVM jobs with USE_PYTHON3=True for your
projects. This will hopefully help us get to our Pike goal for Python
3.x [10]

Please stop by #openstack-python3 channel to chat.

Thanks,
Dims

[1] https://review.openstack.org/#/c/414176/
[2] 
http://logs.openstack.org/76/414176/11/check/gate-devstack-dsvm-py35-updown-ubuntu-xenial-nv/
[3] 
http://logs.openstack.org/00/412500/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f5a7fe7/logs/screen-s-proxy.txt.gz
[4] https://review.openstack.org/#/c/412500/
[5] https://review.openstack.org/#/c/414727/
[6] https://review.openstack.org/#/c/416064/
[7] https://review.openstack.org/#/c/416084/
[8] https://review.openstack.org/#/c/414176/11/inc/python
[9] https://review.openstack.org/#/c/413775/5/jenkins/jobs/devstack-gate.yaml
[10] https://review.openstack.org/#/c/349069/

-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev