Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-04 Thread Amrith Kumar 
Absolutely, the default would be disabled and those who want it can enable
it. There will be a cost; hashing and signing aren't cheap and we shouldn't
have to pay the cost when we don't have to.

-amrith

> -Original Message-
> From: gordon chung [mailto:g...@live.ca]
> Sent: Friday, November 4, 2016 8:22 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [all][dev][python] constructing a
deterministic
> representation of a python data structure
> 
> 
> 
> On 04/11/16 08:02 AM, Amrith Kumar wrote:
> > [Amrith Kumar] I wasn't sure I'd be able to do this before (the only
> > place where this model will work is in oslo.messaging) but thanks to
> > the links that Dims forwarded, there may be a chance to make this part
of
> oslo!
> 
> if you do this in oslo.messaging, can i suggest you disable it by default.
i
> haven't tried this in quite a while but i do remember it made message
> sending a few times slower when signing messages with hmac. i imagine it
> also added the same overhead on receiving end.
> 
> cheers,
> --
> gord
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-04 Thread gordon chung 


On 04/11/16 08:02 AM, Amrith Kumar wrote:
> [Amrith Kumar] I wasn't sure I'd be able to do this before (the only place
> where this model will work is in oslo.messaging) but thanks to the links
> that Dims forwarded, there may be a chance to make this part of oslo!

if you do this in oslo.messaging, can i suggest you disable it by 
default. i haven't tried this in quite a while but i do remember it made 
message sending a few times slower when signing messages with hmac. i 
imagine it also added the same overhead on receiving end.

cheers,
-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-04 Thread Amrith Kumar 


> -Original Message-
> From: Clint Byrum [mailto:cl...@fewbar.com]
> Sent: Thursday, November 3, 2016 6:04 PM
> To: openstack-dev <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [all][dev][python] constructing a
deterministic
> representation of a python data structure
> 
> Excerpts from Amrith Kumar's message of 2016-11-03 20:50:01 +:
> > Josh,
> >
> > I have the key management part figured out and in actuality I will be
> > signing the messages.
> >
> > But step 1 is getting a deterministic representation and step 2 is
hashing.
> > Step 3 would be signing.
> >
> > So, steps 2 and 3 are all set; just need step 1 :) And I'm marveling
> > at the link that Morgan provided, it may have what I need.
> >
> 
> Please please please do not invent your own home rolled cryptographic
> envelope!!

[Amrith Kumar] Of course, did it once, have the badge and don't need another
:)

> 
> sender.py:
> 
> 
> to_send = {
>   'fact': 'red is the best color'
> }
> payload = json.dumps(to_send)
> message = gpg_sign(payload, key)
> send_message(message)
> 
> 
> receiver.py:
> 
> message = recv_message()
> (payload, key) = gpg_verify_message(message) if key not in trusted_keys:
>   raise Exception('Untrusted sender!')
> operate_on_payload(payload)
> 
> With all due respect, any of us are almost guaranteed to screw it up
> otherwise. Just use a thing known to work. There are plenty already.

[Amrith Kumar] I wasn't sure I'd be able to do this before (the only place
where this model will work is in oslo.messaging) but thanks to the links
that Dims forwarded, there may be a chance to make this part of oslo!


> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread Clint Byrum 
Excerpts from Amrith Kumar's message of 2016-11-03 20:50:01 +:
> Josh,
> 
> I have the key management part figured out and in actuality I will be
> signing the messages. 
> 
> But step 1 is getting a deterministic representation and step 2 is hashing.
> Step 3 would be signing.
> 
> So, steps 2 and 3 are all set; just need step 1 :) And I'm marveling at the
> link that Morgan provided, it may have what I need.
> 

Please please please do not invent your own home rolled cryptographic
envelope!!

sender.py:


to_send = {
  'fact': 'red is the best color'
}
payload = json.dumps(to_send)
message = gpg_sign(payload, key)
send_message(message)


receiver.py:

message = recv_message()
(payload, key) = gpg_verify_message(message)
if key not in trusted_keys:
  raise Exception('Untrusted sender!')
operate_on_payload(payload)

With all due respect, any of us are almost guaranteed to screw it up
otherwise. Just use a thing known to work. There are plenty already.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread Davanum Srinivas 
Amrith,

Please see the older work from 2013:
https://blueprints.launchpad.net/oslo.messaging/+spec/trusted-messaging
https://etherpad.openstack.org/p/HavanaOsloMessaging
https://blog-nkinder.rhcloud.com/?p=62
https://adam.younglogic.com/2014/04/pki-for-oslo-messaging/

-- Dims


On Thu, Nov 3, 2016 at 3:35 PM, Amrith Kumar <amr...@tesora.com> wrote:
> Dims, I don't think ovo addresses this issue. It isn't one of versioning that 
> I'm attempting to address but rather as Gordon points out, of signing and 
> verifying authenticity. Please see also my longer response to him on this.
>
> -amrith
>
> -Original Message-
> From: Davanum Srinivas [mailto:dava...@gmail.com]
> Sent: Thursday, November 3, 2016 3:03 PM
> To: OpenStack Development Mailing List (not for usage questions) 
> <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [all][dev][python] constructing a deterministic 
> representation of a python data structure
>
> Does oslo.versionedobjects solve some of your needs?
>
> http://www.slideshare.net/davanum/ovo-deep-dive
> https://gorka.eguileor.com/learning-something-new-about-oslo-versioned-objects/
> http://www.danplanet.com/blog/2015/10/06/upgrades-in-nova-objects/
>
> -- Dims
>
> On Thu, Nov 3, 2016 at 2:24 PM, Amrith Kumar <amr...@tesora.com> wrote:
>> TL;DR
>>
>>
>>
>> I want to take a python data structure (see later for details) and
>> represent it in a format that will be stable across python versions,
>> and platforms so that I can construct a stable hash. I’m looking for
>> pointers to some best practices on how to do this.
>>
>>
>>
>> The longer version
>>
>>
>>
>> Assume that there’s some function in python that is:
>>
>>
>>
>> def fn(*args, **kwargs):
>>
>> …
>>
>>
>>
>> I’d like to take (args, kwargs) and construct a hash of some
>> representation that is deterministic. Specifically, assume that fn()
>> is a method in the API (oslo.messaging transport …). I’d like to
>> construct the hash on the sender side and on the RPC server side and
>> make sure that the parameters are the same.
>>
>>
>>
>> So, just before calling call() or cast(), I could compute the hash and
>> stuff it into the dictionary that is being sent over, and I can do the
>> same on the receiving side. But since I cannot guarantee that the
>> representation on the receiving side is necessarily identical to the
>> representation on the sending side, I have issues computing the hash.
>>
>>
>>
>> In IRC, jroll suggested json; can one safely assume that json.dumps()
>> is a deterministic representation?
>>
>>
>>
>> Thanks for any pointers and suggestions.
>>
>>
>>
>> -amrith
>>
>>
>> __
>>  OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread Amrith Kumar 
 

 

From: Morgan Fainberg [mailto:morgan.fainb...@gmail.com] 
Sent: Thursday, November 3, 2016 4:31 PM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [all][dev][python] constructing a deterministic 
representation of a python data structure

 

 

On Thu, Nov 3, 2016 at 1:04 PM, Amrith Kumar <amr...@tesora.com 
<mailto:amr...@tesora.com> > wrote:

Gordon,

You can see a very quick-and-dirty prototype of the kind of thing I'm
looking to do in Trove at
https://gist.github.com/amrith/6a89ff478f81c2910e84325923eddebe

Uncommenting line 51 would simulate a bad hash.

I'd be happy to propose something similar in oslo.messaging if you think
that would pass muster there.

-amrith

-Original Message-
From: gordon chung [mailto:g...@live.ca <mailto:g...@live.ca> ]
Sent: Thursday, November 3, 2016 3:09 PM
To: openstack-dev@lists.openstack.org 
<mailto:openstack-dev@lists.openstack.org> 
Subject: Re: [openstack-dev] [all][dev][python] constructing a deterministic
representation of a python data structure




On 03/11/16 02:24 PM, Amrith Kumar wrote:

>
> So, just before calling call() or cast(), I could compute the hash and
> stuff it into the dictionary that is being sent over, and I can do the
> same on the receiving side. But since I cannot guarantee that the
> representation on the receiving side is necessarily identical to the
> representation on the sending side, I have issues computing the hash.
>
>

based on description, you're trying to sign the messages? there was some
effort done in oslo.messaging[1]

we do something similar in Ceilometer to sign IPC messages[2]. it does add
overhead though.

[1] https://review.openstack.org/#/c/205330/
[2]
https://github.com/openstack/ceilometer/blob/ffc9ee99c10ede988769907fdb0594a 
<https://github.com/openstack/ceilometer/blob/ffc9ee99c10ede988769907fdb0594a512c890cd/ceilometer/publisher/utils.py#L43-L58>
 
512c890cd/ceilometer/publisher/utils.py#L43-L58

cheers,
--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 

I had to solve a similar issue for deterministic key generation in dogpile (key 
for memcache/etc) when memoizing methods/functions with kwargs. There are a 
couple issues you run into, default args are not represented in **kwargs, and 
non-positional args can come in any order. 

 

[Amrith Kumar] This, as it turns out, is a good thing. In the specific use case 
that I have, the sender of the message may be on one version and the recipient 
may be on a later version which defaults some parameters. Therefore 
intercepting and handing the arguments in a decorator is perfect for me because 
I don’t get to see the impact of the default arguments that are added by the 
new version of the receiver. Also, since this is used in the context of cast() 
and call() in oslo.messaging, all parameters are passed as kwargs and there are 
no positional arguments.

 

Here, for example, is a call to _cast() for the resize_flavor() RPC call in the 
Trove Taskmanager.

 

def resize_flavor(self, instance_id, old_flavor, new_flavor):

…

 

self._cast("resize_flavor", self.version_cap,

   instance_id=instance_id,

   old_flavor=self._transform_obj(old_flavor),

   new_flavor=self._transform_obj(new_flavor))

 

If you want an example of what we did to generate the cache-key programatically 
you can look here:

 

https://bitbucket.org/zzzeek/dogpile.cache/src/669582c2e5bf12b1303f50c4b7ba3dad308eb1cc/dogpile/cache/util.py?at=master
 
<https://bitbucket.org/zzzeek/dogpile.cache/src/669582c2e5bf12b1303f50c4b7ba3dad308eb1cc/dogpile/cache/util.py?at=master=file-view-default#util.py-67:118>
 =file-view-default#util.py-67:118

 

[Amrith Kumar] Thanks, this is awesome! But, maybe a little more than I need. 
I’ll snag the guts of generate_key() and that should give me what I need (I 
think).

 

You don't need all the namespace and probably not fn/module info, but this can 
look at the call and handle / ensure defaults also match (or be used to extract 
default kwargs if needed) for passing down to RPC.

 

--Morgan



smime.p7s
Description: S/MIME cryptographic signature

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread Amrith Kumar 
Josh,

I have the key management part figured out and in actuality I will be
signing the messages. 

But step 1 is getting a deterministic representation and step 2 is hashing.
Step 3 would be signing.

So, steps 2 and 3 are all set; just need step 1 :) And I'm marveling at the
link that Morgan provided, it may have what I need.

-amrith

-Original Message-
From: Joshua Harlow [mailto:harlo...@fastmail.com] 
Sent: Thursday, November 3, 2016 4:31 PM
To: OpenStack Development Mailing List (not for usage questions)
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [all][dev][python] constructing a deterministic
representation of a python data structure

I wouldn't recommend this (the basic hash) if you actually want to do any
kind of validation that the contents weren't altered. Is that the purpose?
Or are you trying to ensure bits aren't flipped?

If u want some level of validation that the message wasn't tampered with u
probably at least want https://docs.python.org/2/library/hmac.html and then
you need to start figuring out what to do about key distribution &
management and rotation :-/

Amrith Kumar wrote:
> Gordon,
>
> You can see a very quick-and-dirty prototype of the kind of thing I'm 
> looking to do in Trove at 
> https://gist.github.com/amrith/6a89ff478f81c2910e84325923eddebe
>
> Uncommenting line 51 would simulate a bad hash.
>
> I'd be happy to propose something similar in oslo.messaging if you 
> think that would pass muster there.
>
> -amrith
>
> -Original Message-
> From: gordon chung [mailto:g...@live.ca]
> Sent: Thursday, November 3, 2016 3:09 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [all][dev][python] constructing a 
> deterministic representation of a python data structure
>
>
>
> On 03/11/16 02:24 PM, Amrith Kumar wrote:
>
>> So, just before calling call() or cast(), I could compute the hash 
>> and stuff it into the dictionary that is being sent over, and I can 
>> do the same on the receiving side. But since I cannot guarantee that 
>> the representation on the receiving side is necessarily identical to 
>> the representation on the sending side, I have issues computing the hash.
>>
>>
>
> based on description, you're trying to sign the messages? there was 
> some effort done in oslo.messaging[1]
>
> we do something similar in Ceilometer to sign IPC messages[2]. it does 
> add overhead though.
>
> [1] https://review.openstack.org/#/c/205330/
> [2]
> https://github.com/openstack/ceilometer/blob/ffc9ee99c10ede988769907fd
> b0594a
> 512c890cd/ceilometer/publisher/utils.py#L43-L58
>
> cheers,
> --
> gord
>
> __
>  OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
>  OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread Morgan Fainberg 
On Thu, Nov 3, 2016 at 1:04 PM, Amrith Kumar <amr...@tesora.com> wrote:

> Gordon,
>
> You can see a very quick-and-dirty prototype of the kind of thing I'm
> looking to do in Trove at
> https://gist.github.com/amrith/6a89ff478f81c2910e84325923eddebe
>
> Uncommenting line 51 would simulate a bad hash.
>
> I'd be happy to propose something similar in oslo.messaging if you think
> that would pass muster there.
>
> -amrith
>
> -Original Message-
> From: gordon chung [mailto:g...@live.ca]
> Sent: Thursday, November 3, 2016 3:09 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [all][dev][python] constructing a
> deterministic
> representation of a python data structure
>
>
>
> On 03/11/16 02:24 PM, Amrith Kumar wrote:
>
> >
> > So, just before calling call() or cast(), I could compute the hash and
> > stuff it into the dictionary that is being sent over, and I can do the
> > same on the receiving side. But since I cannot guarantee that the
> > representation on the receiving side is necessarily identical to the
> > representation on the sending side, I have issues computing the hash.
> >
> >
>
> based on description, you're trying to sign the messages? there was some
> effort done in oslo.messaging[1]
>
> we do something similar in Ceilometer to sign IPC messages[2]. it does add
> overhead though.
>
> [1] https://review.openstack.org/#/c/205330/
> [2]
> https://github.com/openstack/ceilometer/blob/
> ffc9ee99c10ede988769907fdb0594a
> 512c890cd/ceilometer/publisher/utils.py#L43-L58
>
> cheers,
> --
> gord
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
I had to solve a similar issue for deterministic key generation in dogpile
(key for memcache/etc) when memoizing methods/functions with kwargs. There
are a couple issues you run into, default args are not represented in
**kwargs, and non-positional args can come in any order.

If you want an example of what we did to generate the cache-key
programatically you can look here:

https://bitbucket.org/zzzeek/dogpile.cache/src/669582c2e5bf12b1303f50c4b7ba3dad308eb1cc/dogpile/cache/util.py?at=master=file-view-default#util.py-67:118

You don't need all the namespace and probably not fn/module info, but this
can look at the call and handle / ensure defaults also match (or be used to
extract default kwargs if needed) for passing down to RPC.

--Morgan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread Joshua Harlow 
I wouldn't recommend this (the basic hash) if you actually want to do 
any kind of validation that the contents weren't altered. Is that the 
purpose? Or are you trying to ensure bits aren't flipped?


If u want some level of validation that the message wasn't tampered with 
u probably at least want https://docs.python.org/2/library/hmac.html and 
then you need to start figuring out what to do about key distribution & 
management and rotation :-/


Amrith Kumar wrote:

Gordon,

You can see a very quick-and-dirty prototype of the kind of thing I'm
looking to do in Trove at
https://gist.github.com/amrith/6a89ff478f81c2910e84325923eddebe

Uncommenting line 51 would simulate a bad hash.

I'd be happy to propose something similar in oslo.messaging if you think
that would pass muster there.

-amrith

-Original Message-
From: gordon chung [mailto:g...@live.ca]
Sent: Thursday, November 3, 2016 3:09 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][dev][python] constructing a deterministic
representation of a python data structure



On 03/11/16 02:24 PM, Amrith Kumar wrote:


So, just before calling call() or cast(), I could compute the hash and
stuff it into the dictionary that is being sent over, and I can do the
same on the receiving side. But since I cannot guarantee that the
representation on the receiving side is necessarily identical to the
representation on the sending side, I have issues computing the hash.




based on description, you're trying to sign the messages? there was some
effort done in oslo.messaging[1]

we do something similar in Ceilometer to sign IPC messages[2]. it does add
overhead though.

[1] https://review.openstack.org/#/c/205330/
[2]
https://github.com/openstack/ceilometer/blob/ffc9ee99c10ede988769907fdb0594a
512c890cd/ceilometer/publisher/utils.py#L43-L58

cheers,
--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread Amrith Kumar 
Gordon,

You can see a very quick-and-dirty prototype of the kind of thing I'm
looking to do in Trove at
https://gist.github.com/amrith/6a89ff478f81c2910e84325923eddebe

Uncommenting line 51 would simulate a bad hash.

I'd be happy to propose something similar in oslo.messaging if you think
that would pass muster there.

-amrith

-Original Message-
From: gordon chung [mailto:g...@live.ca] 
Sent: Thursday, November 3, 2016 3:09 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][dev][python] constructing a deterministic
representation of a python data structure



On 03/11/16 02:24 PM, Amrith Kumar wrote:

>
> So, just before calling call() or cast(), I could compute the hash and 
> stuff it into the dictionary that is being sent over, and I can do the 
> same on the receiving side. But since I cannot guarantee that the 
> representation on the receiving side is necessarily identical to the 
> representation on the sending side, I have issues computing the hash.
>
>

based on description, you're trying to sign the messages? there was some
effort done in oslo.messaging[1]

we do something similar in Ceilometer to sign IPC messages[2]. it does add
overhead though.

[1] https://review.openstack.org/#/c/205330/
[2]
https://github.com/openstack/ceilometer/blob/ffc9ee99c10ede988769907fdb0594a
512c890cd/ceilometer/publisher/utils.py#L43-L58

cheers,
--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread Amrith Kumar 
Dims, I don't think ovo addresses this issue. It isn't one of versioning that 
I'm attempting to address but rather as Gordon points out, of signing and 
verifying authenticity. Please see also my longer response to him on this.

-amrith

-Original Message-
From: Davanum Srinivas [mailto:dava...@gmail.com] 
Sent: Thursday, November 3, 2016 3:03 PM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [all][dev][python] constructing a deterministic 
representation of a python data structure

Does oslo.versionedobjects solve some of your needs?

http://www.slideshare.net/davanum/ovo-deep-dive
https://gorka.eguileor.com/learning-something-new-about-oslo-versioned-objects/
http://www.danplanet.com/blog/2015/10/06/upgrades-in-nova-objects/

-- Dims

On Thu, Nov 3, 2016 at 2:24 PM, Amrith Kumar <amr...@tesora.com> wrote:
> TL;DR
>
>
>
> I want to take a python data structure (see later for details) and 
> represent it in a format that will be stable across python versions, 
> and platforms so that I can construct a stable hash. I’m looking for 
> pointers to some best practices on how to do this.
>
>
>
> The longer version
>
>
>
> Assume that there’s some function in python that is:
>
>
>
> def fn(*args, **kwargs):
>
> …
>
>
>
> I’d like to take (args, kwargs) and construct a hash of some 
> representation that is deterministic. Specifically, assume that fn() 
> is a method in the API (oslo.messaging transport …). I’d like to 
> construct the hash on the sender side and on the RPC server side and 
> make sure that the parameters are the same.
>
>
>
> So, just before calling call() or cast(), I could compute the hash and 
> stuff it into the dictionary that is being sent over, and I can do the 
> same on the receiving side. But since I cannot guarantee that the 
> representation on the receiving side is necessarily identical to the 
> representation on the sending side, I have issues computing the hash.
>
>
>
> In IRC, jroll suggested json; can one safely assume that json.dumps() 
> is a deterministic representation?
>
>
>
> Thanks for any pointers and suggestions.
>
>
>
> -amrith
>
>
> __
>  OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



--
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread gordon chung 


On 03/11/16 02:24 PM, Amrith Kumar wrote:

>
> So, just before calling call() or cast(), I could compute the hash and
> stuff it into the dictionary that is being sent over, and I can do the
> same on the receiving side. But since I cannot guarantee that the
> representation on the receiving side is necessarily identical to the
> representation on the sending side, I have issues computing the hash.
>
>

based on description, you're trying to sign the messages? there was some 
effort done in oslo.messaging[1]

we do something similar in Ceilometer to sign IPC messages[2]. it does 
add overhead though.

[1] https://review.openstack.org/#/c/205330/
[2] 
https://github.com/openstack/ceilometer/blob/ffc9ee99c10ede988769907fdb0594a512c890cd/ceilometer/publisher/utils.py#L43-L58

cheers,
-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

2016-11-03 Thread Davanum Srinivas 
Does oslo.versionedobjects solve some of your needs?

http://www.slideshare.net/davanum/ovo-deep-dive
https://gorka.eguileor.com/learning-something-new-about-oslo-versioned-objects/
http://www.danplanet.com/blog/2015/10/06/upgrades-in-nova-objects/

-- Dims

On Thu, Nov 3, 2016 at 2:24 PM, Amrith Kumar  wrote:
> TL;DR
>
>
>
> I want to take a python data structure (see later for details) and represent
> it in a format that will be stable across python versions, and platforms so
> that I can construct a stable hash. I’m looking for pointers to some best
> practices on how to do this.
>
>
>
> The longer version
>
>
>
> Assume that there’s some function in python that is:
>
>
>
> def fn(*args, **kwargs):
>
> …
>
>
>
> I’d like to take (args, kwargs) and construct a hash of some representation
> that is deterministic. Specifically, assume that fn() is a method in the API
> (oslo.messaging transport …). I’d like to construct the hash on the sender
> side and on the RPC server side and make sure that the parameters are the
> same.
>
>
>
> So, just before calling call() or cast(), I could compute the hash and stuff
> it into the dictionary that is being sent over, and I can do the same on the
> receiving side. But since I cannot guarantee that the representation on the
> receiving side is necessarily identical to the representation on the sending
> side, I have issues computing the hash.
>
>
>
> In IRC, jroll suggested json; can one safely assume that json.dumps() is a
> deterministic representation?
>
>
>
> Thanks for any pointers and suggestions.
>
>
>
> -amrith
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev