Re: encoding newlines in attribute values
On 27-Apr-07, at 3:46 PM, Johnny Bufu wrote: > The default encoding would then be applied only when a attribute- > specific encoding was not used. With help from Mark Wahl I've put this into the spec and wrapped up the remaining issues. The latest version is in SVN here: http://openid.net/svn/listing.php?repname=specifications&path=% 2F&rev=0&sc=1 And I've also uploaded html and text versions for convenience: http://openid.net/svn/specifications/attribute_exchange/1.0/trunk/ openid-attribute-exchange.txt http://openid.net/svn/specifications/attribute_exchange/1.0/trunk/ openid-attribute-exchange.html Please have a look and point out what still needs to be fixed. I'd like to tag draft6 at the end of the week so that we can reference it easier during IIW. Thanks, Johnny ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
RE: encoding newlines in attribute values
Cool, thanks! > -Original Message- > From: Johnny Bufu [mailto:[EMAIL PROTECTED] > Sent: Monday, April 30, 2007 9:51 AM > To: Granqvist, Hans > Cc: OpenID specs list > Subject: Re: encoding newlines in attribute values > > Hans, > > On 30-Apr-07, at 9:22 AM, Granqvist, Hans wrote: > > Just so we're all on the same page: Can you post a link to the > > referenced proposal? > > Mark has announced it here on the list: > > http://openid.net/pipermail/specs/2007-April/001630.html > > > Johnny > > > ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
Hans, On 30-Apr-07, at 9:22 AM, Granqvist, Hans wrote: > Just so we're all on the same page: Can you post a link > to the referenced proposal? Mark has announced it here on the list: http://openid.net/pipermail/specs/2007-April/001630.html Johnny ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
RE: encoding newlines in attribute values
Hi Johnny, Just so we're all on the same page: Can you post a link to the referenced proposal? Thanks, Hans > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Johnny Bufu > Sent: Friday, April 27, 2007 3:46 PM > To: OpenID specs list > Subject: Re: encoding newlines in attribute values > > > On 20-Apr-07, at 11:18 AM, Dick Hardt wrote: > > To expand on that and include Mark Wahl's proposal about locale > > encoding[1] in a standard way for attributes so that the > libraries can > > do the right thing 99% of the time. > > > > I would propose that AX data have a default encoding that can be > > overrode by the attribute metadata. The default would be: > > > > URL encoding for text data > > escape sequence for locale using mechanism in RFC 3866 > escape sequence > > to indicate binary data that is then base64 encoded > > Does this work for everyone? If there are no issues, I'd like > to summarize it in the spec so that: > > - a "default encoding" is defined as described above > - attribute specific encodings can be used, and their > presence is signaled with an escape sequence similar to the > ones used in Mark's language tags proposal. > > The default encoding would then be applied only when a > attribute- specific encoding was not used. > > Agreed? > > > Johnny > > > ___ > specs mailing list > specs@openid.net > http://openid.net/mailman/listinfo/specs > ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On 20-Apr-07, at 11:18 AM, Dick Hardt wrote: > To expand on that and include Mark Wahl's proposal about locale > encoding[1] in a standard way for attributes so that the libraries > can do the right thing 99% of the time. > > I would propose that AX data have a default encoding that can be > overrode by the attribute metadata. The default would be: > > URL encoding for text data > escape sequence for locale using mechanism in RFC 3866 > escape sequence to indicate binary data that is then base64 encoded Does this work for everyone? If there are no issues, I'd like to summarize it in the spec so that: - a "default encoding" is defined as described above - attribute specific encodings can be used, and their presence is signaled with an escape sequence similar to the ones used in Mark's language tags proposal. The default encoding would then be applied only when a attribute- specific encoding was not used. Agreed? Johnny ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On 20-Apr-07, at 11:05 AM, Douglas Otis wrote: > > On Apr 20, 2007, at 10:56 AM, Johnny Bufu wrote: > >> >> On Apr 19, 2007, at 10:46 AM, Josh Hoyt wrote: >>> Each attribute already has to define its encoding rules and data- >>> type. The mechanism for encoding a newline can be part of this >>> encoding, if newlines are allowed in the value. Once there is one >>> attribute that has a defined encoding for newline, when new >>> attributes are defined, they can re-use this encoding. Does that >>> sound reasonable? >> >> So are you proposing that AX only accepts strings without newline >> characters in them, and the encoding to such a string should be >> handled by the parties who actually consume the attributes, according >> to the type / metadata specs? >> >> This would be nice and simple for the AX itself, however it would >> require everyone defining attributes to also define a 'encoding to >> strings without newlines' for them. > > The use of base64 can be confined to individual elements within an > attribute where newlines are _not_ affected. There should be a > standardized template for declaring base64 encoding starts with 'X' > and ends with 'Y'; Great idea Douglas. To expand on that and include Mark Wahl's proposal about locale encoding[1] in a standard way for attributes so that the libraries can do the right thing 99% of the time. I would propose that AX data have a default encoding that can be overrode by the attribute metadata. The default would be: URL encoding for text data escape sequence for locale using mechanism in RFC 3866 escape sequence to indicate binary data that is then base64 encoded [1] I see that Mark's proposal is not up on the site yet. It is a good proposal though! ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On Apr 20, 2007, at 10:56 AM, Johnny Bufu wrote: > > On Apr 19, 2007, at 10:46 AM, Josh Hoyt wrote: >> Each attribute already has to define its encoding rules and data- >> type. The mechanism for encoding a newline can be part of this >> encoding, if newlines are allowed in the value. Once there is one >> attribute that has a defined encoding for newline, when new >> attributes are defined, they can re-use this encoding. Does that >> sound reasonable? > > So are you proposing that AX only accepts strings without newline > characters in them, and the encoding to such a string should be > handled by the parties who actually consume the attributes, according > to the type / metadata specs? > > This would be nice and simple for the AX itself, however it would > require everyone defining attributes to also define a 'encoding to > strings without newlines' for them. The use of base64 can be confined to individual elements within an attribute where newlines are _not_ affected. There should be a standardized template for declaring base64 encoding starts with 'X' and ends with 'Y'; ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On Apr 19, 2007, at 10:46 AM, Josh Hoyt wrote: > Each attribute already has to define its encoding rules and data- > type. The mechanism for encoding a newline can be part of this > encoding, if newlines are allowed in the value. Once there is one > attribute that has a defined encoding for newline, when new > attributes are defined, they can re-use this encoding. Does that > sound reasonable? So are you proposing that AX only accepts strings without newline characters in them, and the encoding to such a string should be handled by the parties who actually consume the attributes, according to the type / metadata specs? This would be nice and simple for the AX itself, however it would require everyone defining attributes to also define a 'encoding to strings without newlines' for them. Johnny ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On Apr 19, 2007, at 10:46 AM, Josh Hoyt wrote: > On 4/19/07, Marius Scurtescu <[EMAIL PROTECTED]> wrote: >> I think we do need pre-URL-encoding, mainly because of signatures. >> In order to calculate the signature the parameters must be put >> together in a special way and new line characters are not allowed. > > Yes. The key-value encoding that's used for POST responses (to > associate and check_authentication) is also used in signature > generation. This is the source of the restriction on newlines in > values, not anything to do with URL encoding. > > Each attribute already has to define its encoding rules and data- > type. The mechanism for encoding a newline can be part of this > encoding, if newlines are allowed in the value. Once there is one > attribute that has a defined encoding for newline, when new > attributes are defined, they can re-use this encoding. Does that > sound reasonable? That sounds fair, however consistent encoding methods with a standardized syntax should be recommended. Elements like icons, voice, signatures blobs could adopt some type of standardized an overlay template. -Doug ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On 4/19/07, Marius Scurtescu <[EMAIL PROTECTED]> wrote: > I think we do need pre-URL-encoding, mainly because of signatures. In > order to calculate the signature the parameters must be put together > in a special way and new line characters are not allowed. Yes. The key-value encoding that's used for POST responses (to associate and check_authentication) is also used in signature generation. This is the source of the restriction on newlines in values, not anything to do with URL encoding. Each attribute already has to define its encoding rules and data-type. The mechanism for encoding a newline can be part of this encoding, if newlines are allowed in the value. Once there is one attribute that has a defined encoding for newline, when new attributes are defined, they can re-use this encoding. Does that sound reasonable? Josh ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On 19-Apr-07, at 7:29 AM, Rowan Kerr wrote:
> On 18-Apr-07, at 9:47 PM, Johnny Bufu wrote:
>> The core spec doesn't allow newline characters ("\n") in any openid.*
>> values. Currently, Attribute Exchange doesn't specify a way to encode
>> newlines in attribute values.
>
> Every indirect OpenID message would seem to be already url-encoded by
> the browser, or server as post data .. so "\n" => %0A (i.e.
> application/x-www-form-urlencoded mime type)
>
> Do we need a pre-url-encode encoding, or can we rely on browsers to
> do the right thing... I suppose it's helpful to spell it out for non-
> browser agents that want to pass OpenID messages.
I think we do need pre-URL-encoding, mainly because of signatures. In
order to calculate the signature the parameters must be put together
in a special way and new line characters are not allowed.
>
> If we want to define sending binary data in OpenID messages, maybe we
> should leverage multipart/form-data.
Same as above, need to encode for signatures to work.
Marius
___
specs mailing list
specs@openid.net
http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On 18-Apr-07, at 9:47 PM, Johnny Bufu wrote:
> The core spec doesn't allow newline characters ("\n") in any openid.*
> values. Currently, Attribute Exchange doesn't specify a way to encode
> newlines in attribute values.
Every indirect OpenID message would seem to be already url-encoded by
the browser, or server as post data .. so "\n" => %0A (i.e.
application/x-www-form-urlencoded mime type)
Do we need a pre-url-encode encoding, or can we rely on browsers to
do the right thing... I suppose it's helpful to spell it out for non-
browser agents that want to pass OpenID messages.
If we want to define sending binary data in OpenID messages, maybe we
should leverage multipart/form-data.
___
specs mailing list
specs@openid.net
http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On Wed, 2007-18-04 at 23:25 -0700, Douglas Otis wrote: > On Apr 18, 2007, at 8:31 PM, Marius Scurtescu wrote: > > > Base64 encoding is a pretty good candidate for binary data, but you > > cannot apply the same encoding to text fields. > > RFC4648 "URL and Filename safe" Base 64 Alphabet might be a good choice. > > > Applying base64, or similar encoding appropriate for binary data, to > > text fields has two drawbacks: > > - renders the field unreadable > > Binary data is often unreadable. True, but text is readable and ideally it should stay like that. > > > - increases the size of the field > > Base 64 increases the size of the encoded element by about 30%. > > > URL-encoding has the advantage that probably all web frameworks will > > have functions to encode and decode this format. > > URL-encoding increases the size of the encoded element by 300%. Only unsafe characters are encoded, not the whole string. Not sure how UTF-8 changes any of this. Marius ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
Just because one or two characters (entities) might be expanded in size by 300% doesn't justify expanding an entire attribute by 30%, especially if it's a got enough length to it to justify having newlines in it. - Original Message - From: "Douglas Otis" <[EMAIL PROTECTED]> To: "Marius Scurtescu" <[EMAIL PROTECTED]> Cc: "Mark Wahl" <[EMAIL PROTECTED]>, "OpenID specs list" Sent: Wednesday, April 18, 2007 11:25:17 PM (GMT-0800) America/Los_Angeles Subject: Re: encoding newlines in attribute values On Apr 18, 2007, at 8:31 PM, Marius Scurtescu wrote: > Base64 encoding is a pretty good candidate for binary data, but you > cannot apply the same encoding to text fields. RFC4648 "URL and Filename safe" Base 64 Alphabet might be a good choice. > Applying base64, or similar encoding appropriate for binary data, to > text fields has two drawbacks: > - renders the field unreadable Binary data is often unreadable. > - increases the size of the field Base 64 increases the size of the encoded element by about 30%. > URL-encoding has the advantage that probably all web frameworks will > have functions to encode and decode this format. URL-encoding increases the size of the encoded element by 300%. -Doug ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On Apr 18, 2007, at 8:31 PM, Marius Scurtescu wrote: > Base64 encoding is a pretty good candidate for binary data, but you > cannot apply the same encoding to text fields. RFC4648 "URL and Filename safe" Base 64 Alphabet might be a good choice. > Applying base64, or similar encoding appropriate for binary data, to > text fields has two drawbacks: > - renders the field unreadable Binary data is often unreadable. > - increases the size of the field Base 64 increases the size of the encoded element by about 30%. > URL-encoding has the advantage that probably all web frameworks will > have functions to encode and decode this format. URL-encoding increases the size of the encoded element by 300%. -Doug ___ specs mailing list specs@openid.net http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
On Wed, 2007-18-04 at 21:05 -0500, Mark Wahl wrote:
> Johnny Bufu wrote:
> > The core spec doesn't allow newline characters ("\n") in any openid.*
> > values. Currently, Attribute Exchange doesn't specify a way to encode
> > newlines in attribute values.
> >
> > At a minimum, we could specify a way to escape just the \n character.
> > Other option would be to do something more generic, e.g. URL-encoding
> > the values.
>
> It would be nice if there was a mechanism defined that could
> transfer binary-valued (non-string) attribute values, such as avatar
> 'icon' images, as OpenID AX attribute values. This is currently
> forbidden by section 3 of AX protocol, IIRC. I suggest that if such a
> binary transfer mechanism were defined, then it could be leveraged
> for transferring strings which contain illegal string value characters,
> such as newline.
Base64 encoding is a pretty good candidate for binary data, but you
cannot apply the same encoding to text fields.
Whatever encoding is used for text fields it must be applied every
single time, not only if the string contains illegal characters.
Otherwise there is no way to properly decode the string (how do you know
if you have to decode or not?). Adding an encoding flag for every field
is an overkill IMO.
Applying base64, or similar encoding appropriate for binary data, to
text fields has two drawbacks:
- renders the field unreadable
- increases the size of the field
URL-encoding has the advantage that probably all web frameworks will
have functions to encode and decode this format. The only minor drawback
I can see is the fact that during transport values will end up being
double encoded and this could generate confusion in some instances (like
debugging).
Marius
___
specs mailing list
specs@openid.net
http://openid.net/mailman/listinfo/specs
Re: encoding newlines in attribute values
Johnny Bufu wrote:
> The core spec doesn't allow newline characters ("\n") in any openid.*
> values. Currently, Attribute Exchange doesn't specify a way to encode
> newlines in attribute values.
>
> At a minimum, we could specify a way to escape just the \n character.
> Other option would be to do something more generic, e.g. URL-encoding
> the values.
It would be nice if there was a mechanism defined that could
transfer binary-valued (non-string) attribute values, such as avatar
'icon' images, as OpenID AX attribute values. This is currently
forbidden by section 3 of AX protocol, IIRC. I suggest that if such a
binary transfer mechanism were defined, then it could be leveraged
for transferring strings which contain illegal string value characters,
such as newline.
Mark Wahl
Informed Control Inc.
___
specs mailing list
specs@openid.net
http://openid.net/mailman/listinfo/specs
