Hi Gordon,
Thanks for looking at this, response below:
On Fri, Feb 15, 2013 at 02:06:05PM +0000, Gordon Sim wrote:
> On 02/14/2013 10:58 AM, Steven Hardy wrote:
> >On Wed, Feb 13, 2013 at 05:38:46PM +0000, Gordon Sim wrote:
> >>On 02/13/2013 08:49 AM, Steven Hardy wrote:
> >>>Hi,
> >>>
> >>>I'm trying to understand some differences encoding map datatypes between
> >>>qpid and rabbitmq (openstack project which needs to support both)
> >>>
> >>>It seems that we hit a limit of 2^16 bytes when encoding a map on qpid,
> >>>because the map appears to be encoded as a string (write_map in
> >>>codec010.py)
> >>>
> >>>Looking at the amqp 0-10 spec, it says "An encoded map may contain up to
> >>>(4294967295 - 4) octets worth of encoded entries.", so I'm trying to
> >>>understand the 65535 byte limitation I'm hitting, since we do not appear to
> >>>hit the same problem when configured to use rabbit.
> >>>
> >>>Can anyone please advise - is this a bug in python-qpid, or an expected
> >>>limitation of the qpid implementation?
> >>
> >>I think that is a bug in python-qpid. Do you have a simple
> >>reproducer you could attach to a JIRA?
> >
> >JIRA raised with reproducer :
> >
> >https://issues.apache.org/jira/browse/QPID-4583
>
> It turns out you can encode larger values as binary by calling
> 'buffer()' on them. E.g. with the attached patch applied to your
> reproducer the message gets sent and received without error.
>
> My view is therefore that this is not a bug, since it is indeed true
> that a string larger than 2^16 cannot be encoded as a str16, and
> there is a reasonable simple way to indicate that the value should
> be treated as binary which can then be encoded as vbin32.
>
> Do you agree?
Well, no - this is only addressing half of the problem - on the receiver
side the binary buffer gets returned to the user as a string, but it's not
converted back to the original format - so the message is corrupted and we
have no way of knowing that conversion from binary is required.
See my attached revised reproducer - maybe I'm missing something, but I
don't see how we can detect the _context_aws_creds key has been binary
encoded on the receiver side.
It seems to me that the problem here is python-qpid encoding individual
dict/map keys - wouldn't it be better to just dump the whole dict to a
string (ie json), then send the whole thing over as vbin32?
Steve
import json
import copy
import sys
from qpid.messaging import *
broker = "localhost:5672" if len(sys.argv)<2 else sys.argv[1]
address = "amq.topic" if len(sys.argv)<3 else sys.argv[2]
connection = Connection(broker)
connection.open()
session = connection.session()
sender = session.sender(address)
receiver = session.receiver(address)
a_template = open('breaks.json').read()
print "SHDEBUG loaded template, len=%s" % len(a_template)
orig_map = json.loads(a_template)
a_map = json.loads(a_template)
# If any key is > 2^16 bytes, str16 encoding fails
for key in a_map:
print "SHDEBUG len(str(a_map[%s])) = %s" % (key, len(str(a_map[key])))
if len(str(a_map[key])) > 65534:
a_map[key] = buffer(a_map[key])
print "encoding value of length %s as binary for key %s" %
(len(str(a_map[key])), key)
print "SHDEBUG created a_map, type=%s" % type(a_map)
sender.send(Message(content=a_map))
message = receiver.fetch(timeout=1)
print "got message, type(message.content) = %s" % type(message.content)
rx_map = message.content
for key in rx_map:
rx_element = rx_map[key]
if rx_element != orig_map[key]:
print "Oh no, key %s does not match!" % key
print "len rx_element = %s, type %s" % (len(rx_element),
type(rx_element))
print "len orig_map[key] = %s. type %s" %(len(orig_map[key]),
type(orig_map[key]))
# print "SHDEBUG rx_element = %s" % rx_element
# print "SHDEBUG orig_map = %s" % orig_map[key]
else:
print "key %s OK" % key
session.acknowledge()
connection.close()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
