Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-04-04 Thread Joshua Bell
Any further input on Kenneth's suggestions? Re: ArrayBufferView vs. DataView - I'm tempted to make the switch to just DataView. As discussed below, data parsing/serialization operations will tend to be associated with DataViews. As Glenn has mentioned elsewhere recently, it is possible to

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-04-04 Thread Glenn Maynard
On Wed, Apr 4, 2012 at 11:09 AM, Joshua Bell jsb...@chromium.org wrote: Any further input on Kenneth's suggestions? I largely disagree with those suggestions, because I don't believe they align with the natural, intuitive usage of the API. Re: ArrayBufferView vs. DataView - I'm tempted to

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-04-01 Thread Jonas Sicking
On Sat, Mar 31, 2012 at 6:13 PM, Glenn Maynard gl...@zewt.org wrote: On Wed, Mar 28, 2012 at 1:44 AM, Jonas Sicking jo...@sicking.cc wrote: Scanning over the buffer twice will cause a lot more memory IO and will definitely be slower. That's what cache is for.  But: benchmarks... We can

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-04-01 Thread Glenn Maynard
On Sun, Apr 1, 2012 at 5:28 PM, Jonas Sicking jo...@sicking.cc wrote: I'm saying that if an API is better in every way then it doesn't seem like an interesting discussion how much better, we should clearly go with that API. It's not a different API, it's an *additional* API. (Assuming that

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-31 Thread Glenn Maynard
On Wed, Mar 28, 2012 at 1:44 AM, Jonas Sicking jo...@sicking.cc wrote: Scanning over the buffer twice will cause a lot more memory IO and will definitely be slower. That's what cache is for. But: benchmarks... We can argue weather it's meaningfully slower or harder. But it seems like we

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-28 Thread Jonas Sicking
On Tue, Mar 27, 2012 at 4:45 PM, Glenn Maynard gl...@zewt.org wrote: On Tue, Mar 27, 2012 at 12:41 AM, Jonas Sicking jo...@sicking.cc wrote: The memchr is purely overhead, I.e. we are comparing memchr+decoding to decoding. So I don't see what's backing up the probably the fastest thing claim.

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-27 Thread Glenn Maynard
On Tue, Mar 27, 2012 at 12:41 AM, Jonas Sicking jo...@sicking.cc wrote: The memchr is purely overhead, I.e. we are comparing memchr+decoding to decoding. So I don't see what's backing up the probably the fastest thing claim. If you don't do it as an initial pass, then you have to embed null

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-27 Thread Kenneth Russell
On Mon, Mar 26, 2012 at 10:28 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, Mar 26, 2012 at 6:11 PM, Kenneth Russell k...@google.com wrote: On Mon, Mar 26, 2012 at 5:33 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, Mar 26, 2012 at 4:40 PM, Joshua Bell jsb...@chromium.org wrote: * We

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-27 Thread Glenn Maynard
On Tue, Mar 27, 2012 at 7:12 PM, Kenneth Russell k...@google.com wrote: - I think it should reference DataView directly rather than ArrayBufferView. The typed array spec was specifically designed with two use cases in mind: in-memory assembly of data to be sent to the graphics card or audio

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-27 Thread Kenneth Russell
On Tue, Mar 27, 2012 at 6:44 PM, Glenn Maynard gl...@zewt.org wrote: On Tue, Mar 27, 2012 at 7:12 PM, Kenneth Russell k...@google.com wrote:   - I think it should reference DataView directly rather than ArrayBufferView. The typed array spec was specifically designed with two use cases in

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Boris Zbarsky
On 3/25/12 7:45 AM, Geoffrey Sneddon wrote: On 21/03/12 04:31, Mark Callow wrote: On 17/03/2012 08:19, Boris Zbarsky wrote: I think that trying to get web developers to do this right is a lost cause, esp. because none of them (to a good approximation) have any big-endian systems to test on.

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Joshua Bell
On Sat, Mar 24, 2012 at 6:52 AM, Glenn Maynard gl...@zewt.org wrote: On Thu, Mar 22, 2012 at 8:58 AM, Anne van Kesteren ann...@opera.com wrote: Another way would be to have a second optional argument that indicates whether more bytes are coming (defaults to false), but I'm not sure of the

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Anne van Kesteren
On Mon, 26 Mar 2012 17:56:41 +0100, Joshua Bell jsb...@chromium.org wrote: Bikeshed: The |continues| term doesn't completely thrill me; it's clear in context, but not necessarily what someone might go searching for. {eof:true} would be lovely except we want the default to be yes-EOF but a

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Joshua Bell
On Mon, Mar 26, 2012 at 2:42 PM, Anne van Kesteren ann...@opera.com wrote: On Mon, 26 Mar 2012 17:56:41 +0100, Joshua Bell jsb...@chromium.org wrote: Bikeshed: The |continues| term doesn't completely thrill me; it's clear in context, but not necessarily what someone might go searching for.

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Glenn Maynard
On Mon, Mar 26, 2012 at 4:49 PM, Joshua Bell jsb...@chromium.org wrote: * A |stream| option, per the above Does this make sense when you're using stream: false to flush the stream? It's still a streaming operation. I guess it's close enough. * A |nullTerminator| option eliminates the need

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Jonas Sicking
On Mon, Mar 26, 2012 at 2:49 PM, Joshua Bell jsb...@chromium.org wrote: On Mon, Mar 26, 2012 at 2:42 PM, Anne van Kesteren ann...@opera.com wrote: On Mon, 26 Mar 2012 17:56:41 +0100, Joshua Bell jsb...@chromium.org wrote: Bikeshed: The |continues| term doesn't completely thrill me; it's

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Joshua Bell
On Mon, Mar 26, 2012 at 4:12 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 26, 2012 at 4:49 PM, Joshua Bell jsb...@chromium.org wrote: * A |stream| option, per the above Does this make sense when you're using stream: false to flush the stream? It's still a streaming operation. I

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Glenn Maynard
On Mon, Mar 26, 2012 at 6:27 PM, Jonas Sicking jo...@sicking.cc wrote: * It appears that we lost the ability to measure how long a resulting buffer was going to be and then decode into the buffer. I don't know if this is an issue. The theory is that it probably isn't a real performance issue

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Jonas Sicking
On Mon, Mar 26, 2012 at 4:40 PM, Joshua Bell jsb...@chromium.org wrote: * We lost the ability to decode from a arraybuffer and see how many bytes were consumed before a null-terminator was hit. One not terribly elegant solution would be to add a TextDecoder.decodeWithLength method which return

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Kenneth Russell
On Mon, Mar 26, 2012 at 5:33 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, Mar 26, 2012 at 4:40 PM, Joshua Bell jsb...@chromium.org wrote: * We lost the ability to decode from a arraybuffer and see how many bytes were consumed before a null-terminator was hit. One not terribly elegant

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread John Tamplin
On Mon, Mar 26, 2012 at 9:11 PM, Kenneth Russell k...@google.com wrote: The rationale for specifying the string encoding and decoding functionality outside the typed array specification is to keep the typed array spec small and easily implementable. The indexed property getters and setters on

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Glenn Maynard
On Mon, Mar 26, 2012 at 7:33 PM, Jonas Sicking jo...@sicking.cc wrote: Requiring callers to find the null character first, and then use that will require one additional pass over the encoded binary data though. That's extremely fast (memchr), and it's probably the fastest thing to do anyway,

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Joshua Bell
On Mon, Mar 26, 2012 at 6:24 PM, Glenn Maynard gl...@zewt.org wrote: I guess. It doesn't seem that important, since it's just a few lines of code. If this is done, I'd suggest that this helper API *not* have any special support for streaming (not to disallow it, but not to have any special

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Jonas Sicking
On Mon, Mar 26, 2012 at 6:11 PM, Kenneth Russell k...@google.com wrote: On Mon, Mar 26, 2012 at 5:33 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, Mar 26, 2012 at 4:40 PM, Joshua Bell jsb...@chromium.org wrote: * We lost the ability to decode from a arraybuffer and see how many bytes were

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-26 Thread Jonas Sicking
On Mon, Mar 26, 2012 at 6:24 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 26, 2012 at 7:33 PM, Jonas Sicking jo...@sicking.cc wrote: Requiring callers to find the null character first, and then use that will require one additional pass over the encoded binary data though. That's

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-24 Thread Glenn Maynard
On Thu, Mar 22, 2012 at 8:58 AM, Anne van Kesteren ann...@opera.com wrote: Another way would be to have a second optional argument that indicates whether more bytes are coming (defaults to false), but I'm not sure of the chances that would be used correctly. The reasons you outline are

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-22 Thread Anne van Kesteren
On Thu, 22 Mar 2012 03:12:25 +0100, Mark Callow callow_m...@hicorp.co.jp wrote: This has encode and decode reversed from my understanding. I regard the string (wide-char) as the canonical form and the bytes as the encoded form. This view is reflected in the widely used terminology charset

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-22 Thread Anne van Kesteren
On Wed, 21 Mar 2012 16:53:36 +0100, Joshua Bell jsb...@chromium.org wrote: Just to throw it out there - does anyone feel we can/should offer asymmetric encode/decode support, i.e. supporting more encodings for decode operations than for encode operations? XMLHttpRequest has that. You can

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-22 Thread Anne van Kesteren
On Thu, 22 Mar 2012 10:19:30 +0100, Anne van Kesteren ann...@opera.com wrote: They can use the prefixed variants :-) If we have to use a prefix String seems better, as Text is a node object in the platform. Simon pointed out Text as prefix is probably better (it is used elsewhere in the

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-22 Thread James Graham
On 03/21/2012 04:53 PM, Joshua Bell wrote: As for the API, how about: enc = new Encoder(euc-kr) string1 = enc.encode(bytes1) string2 = enc.encode(bytes2) string3 = enc.eof() // might return empty string if all is fine And similarly you would have dec = new Decoder(shift_jis)

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-22 Thread Glenn Maynard
On Wed, Mar 21, 2012 at 2:42 PM, Anne van Kesteren ann...@opera.com wrote: As for the API, how about: enc = new Encoder(euc-kr) string1 = enc.encode(bytes1) string2 = enc.encode(bytes2) string3 = enc.eof() // might return empty string if all is fine A problem with this is that the

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-22 Thread Anne van Kesteren
On Thu, 22 Mar 2012 14:47:05 +0100, Glenn Maynard gl...@zewt.org wrote: A problem with this is that the bugs resulting from not calling eof() are subtle. The only thing eof() would ever do, I think, is return U+FFFD characters if there are leftover characters in the internal buffer; if you

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-22 Thread NARUSE, Yui
2012/3/22 Anne van Kesteren ann...@opera.com: As for the API, how about:  enc = new Encoder(euc-kr)  string1 = enc.encode(bytes1)  string2 = enc.encode(bytes2)  string3 = enc.eof() // might return empty string if all is fine And similarly you would have  dec = new Decoder(shift_jis)  

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-21 Thread Jonas Sicking
On Tue, Mar 20, 2012 at 10:39 AM, Joshua Bell jsb...@chromium.org wrote: On Tue, Mar 20, 2012 at 7:26 AM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 19, 2012 at 11:52 PM, Jonas Sicking jo...@sicking.cc wrote: Why are encodings different than other parts of the API where you indeed have

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-21 Thread NARUSE, Yui
2012/3/21 Glenn Maynard gl...@zewt.org: On Tue, Mar 20, 2012 at 12:39 PM, Joshua Bell jsb...@chromium.org wrote: 1. Only support encodings with stateless coding (possibly down to a minimum of UTF-8) 2. Only provide an API supporting non-streaming coding (i.e. whole strings/whole buffers) 3.

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-21 Thread NARUSE, Yui
2012/3/21 Jonas Sicking jo...@sicking.cc: I'm pretty sure there is consensus for supporting UTF8. UTF8 is stateful though can be made not stateful by not consuming all characters and instead forcing the caller to keep the state (in the form of unconsumed text). Your use of the word stateful

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-21 Thread Anne van Kesteren
On Wed, 21 Mar 2012 01:27:47 -0700, Jonas Sicking jo...@sicking.cc wrote: This leaves us with 2 or 3. So the question is if we should support streaming or not. I suspect doing so would be worth it. For XMLHttpRequest it might be, yes. I think we should expose the same encoding set throughout

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-21 Thread Glenn Maynard
On Wed, Mar 21, 2012 at 3:27 AM, Jonas Sicking jo...@sicking.cc wrote: 1) Create an API which forces consumers to do state handling. Probably leading to people creating wrappers which essentially implement option 3 It's not the same. Please look at how ISO-2022 works: the stream has

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-21 Thread Joshua Bell
On Wed, Mar 21, 2012 at 12:42 PM, Anne van Kesteren ann...@opera.comwrote: On Wed, 21 Mar 2012 01:27:47 -0700, Jonas Sicking jo...@sicking.cc wrote: This leaves us with 2 or 3. So the question is if we should support streaming or not. I suspect doing so would be worth it. For

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-21 Thread Charles Pritchard
On 3/21/2012 8:53 AM, Joshua Bell wrote: On Wed, Mar 21, 2012 at 12:42 PM, Anne van Kesterenann...@opera.comwrote: On Wed, 21 Mar 2012 01:27:47 -0700, Jonas Sickingjo...@sicking.cc wrote: This leaves us with 2 or 3. So the question is if we should support streaming or not. I suspect

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-21 Thread Mark Callow
On 22/03/2012 04:42, Anne van Kesteren wrote: ... As for the API, how about: enc = new Encoder(euc-kr) string1 = enc.encode(bytes1) string2 = enc.encode(bytes2) string3 = enc.eof() // might return empty string if all is fine And similarly you would have dec = new

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-20 Thread Glenn Maynard
On Mon, Mar 19, 2012 at 11:52 PM, Jonas Sicking jo...@sicking.cc wrote: Why are encodings different than other parts of the API where you indeed have to know what works and what doesn't. Do you memorize lists of encodings? I certainly don't. I look them up as needed. UTF8 is stateful, so I

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-20 Thread Joshua Bell
On Tue, Mar 20, 2012 at 7:26 AM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 19, 2012 at 11:52 PM, Jonas Sicking jo...@sicking.cc wrote: Why are encodings different than other parts of the API where you indeed have to know what works and what doesn't. Do you memorize lists of

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-20 Thread Glenn Maynard
On Tue, Mar 20, 2012 at 12:39 PM, Joshua Bell jsb...@chromium.org wrote: 1. Only support encodings with stateless coding (possibly down to a minimum of UTF-8) 2. Only provide an API supporting non-streaming coding (i.e. whole strings/whole buffers) 3. Expand the API to return encoder/decoder

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-20 Thread Mark Callow
On 17/03/2012 08:19, Boris Zbarsky wrote: I think that trying to get web developers to do this right is a lost cause, esp. because none of them (to a good approximation) have any big-endian systems to test on. On what do you base this oft-repeated assertion? ARM CPUs can work either way. I

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-19 Thread Henri Sivonen
On Wed, Mar 14, 2012 at 12:49 AM, Jonas Sicking jo...@sicking.cc wrote: Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof) into a decoded string. Similarly being able to encode a string into an ArrayBuffer

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-19 Thread Jonas Sicking
On Mon, Mar 19, 2012 at 7:00 AM, Henri Sivonen hsivo...@iki.fi wrote: On Wed, Mar 14, 2012 at 12:49 AM, Jonas Sicking jo...@sicking.cc wrote: Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof) into a decoded

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-19 Thread Glenn Maynard
On Mon, Mar 19, 2012 at 12:46 PM, Joshua Bell jsb...@chromium.org wrote: I have edited the proposal to base the list of encodings on http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html - is there any reason that would not be sufficient or appropriate? (this appears to be a superset of

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-19 Thread Jonas Sicking
On Mon, Mar 19, 2012 at 5:10 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 19, 2012 at 5:54 PM, Jonas Sicking jo...@sicking.cc wrote: Yes, I think we should enumerate the set of encodings supported. Ideally we'd for simplicity support the same set of enumerated encodings everywhere in

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-19 Thread Glenn Maynard
On Mon, Mar 19, 2012 at 7:33 PM, Jonas Sicking jo...@sicking.cc wrote: What value are we adding, and to whom, by keeping the list the smallest it can be, even when that means keeping the lists of supported encodings different between different APIs? Not needlessly extending support for

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-19 Thread Glenn Maynard
On Mon, Mar 19, 2012 at 8:14 PM, Glenn Maynard gl...@zewt.org wrote: If this is the only reason that'd all have to be specified, that's probably another reason to consider it... (Well, there's form data either way. At least encoding is probably easier to spec, since it only has to deal with

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-19 Thread Jonas Sicking
On Mon, Mar 19, 2012 at 6:14 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 19, 2012 at 7:33 PM, Jonas Sicking jo...@sicking.cc wrote: What value are we adding, and to whom, by keeping the list the smallest it can be, even when that means keeping the lists of supported encodings

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Joshua Bell
On Thu, Mar 15, 2012 at 5:20 PM, Glenn Maynard gl...@zewt.org wrote: On Thu, Mar 15, 2012 at 6:51 PM, Jonas Sicking jo...@sicking.cc wrote: What's the use-case for the stringLength function? You can't decode into an existing datastructure anyway, so you're ultimately forced to call decode at

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Joshua Bell
On Fri, Mar 16, 2012 at 9:19 AM, Joshua Bell jsb...@chromium.org wrote: And just to be clear, the use case is decoding data formats where string fields are variable length null terminated. ... and the spec should include normative guidance that length-prefixing is strongly recommended for

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Glenn Maynard
On Fri, Mar 16, 2012 at 11:19 AM, Joshua Bell jsb...@chromium.org wrote: And just to be clear, the use case is decoding data formats where string fields are variable length null terminated. A concrete example is ZIP central directories. I think we want both encoding and destination to be

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread James Graham
On Fri, 16 Mar 2012, Glenn Maynard wrote: On Fri, Mar 16, 2012 at 11:19 AM, Joshua Bell jsb...@chromium.org wrote: And just to be clear, the use case is decoding data formats where string fields are variable length null terminated. A concrete example is ZIP central directories. I think

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Joshua Bell
On Fri, Mar 16, 2012 at 10:35 AM, Glenn Maynard gl...@zewt.org wrote: On Fri, Mar 16, 2012 at 11:19 AM, Joshua Bell jsb...@chromium.org wrote: ... where output === view if view is supplied, otherwise a new Uint8Array (or Uint8ClampedArray??) Uint8Array is correct. (Uint8ClampedArray is

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Boris Zbarsky
On 3/16/12 5:12 PM, Joshua Bell wrote: FYI, there was some follow up IRC conversation on this. With Typed Arrays as currently specified - that is, that Uint16Array has platform endianness For what it's worth, it seems like this is something we should seriously consider changing so as to make

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Charles Pritchard
On 3/16/2012 2:17 PM, Boris Zbarsky wrote: On 3/16/12 5:12 PM, Joshua Bell wrote: FYI, there was some follow up IRC conversation on this. With Typed Arrays as currently specified - that is, that Uint16Array has platform endianness For what it's worth, it seems like this is something we

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread James Graham
On Fri, 16 Mar 2012, Charles Pritchard wrote: On 3/16/2012 2:17 PM, Boris Zbarsky wrote: On 3/16/12 5:12 PM, Joshua Bell wrote: FYI, there was some follow up IRC conversation on this. With Typed Arrays as currently specified - that is, that Uint16Array has platform endianness For what it's

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Glenn Maynard
On Fri, Mar 16, 2012 at 4:44 PM, Charles Pritchard ch...@jumis.com wrote: The DataView set of methods already does this work. The raw arrays are supposed to have platform endianness. That's wrong. This is web API design 101; everyone should know better than this by now. Exposing platform

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Charles Pritchard
On 3/16/2012 3:26 PM, Glenn Maynard wrote: On Fri, Mar 16, 2012 at 4:44 PM, Charles Pritchard ch...@jumis.com mailto:ch...@jumis.com wrote: The DataView set of methods already does this work. The raw arrays are supposed to have platform endianness. That's wrong. This is web API

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Boris Zbarsky
On 3/16/12 5:44 PM, Charles Pritchard wrote: The DataView set of methods already does this work. The raw arrays are supposed to have platform endianness. I haven't seen anyone actually using the DataView stuff in practice, or presenting it to developers much... If you see some evangelists

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Boris Zbarsky
On 3/16/12 5:25 PM, Brandon Jones wrote: Everyone knows that typed arrays /can/ be Big Endian, but I'm not aware of any devices available right now that support WebGL that are. I believe that recent Firefox on a SPARC processor would fit that description. Of course the number of web

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Jonas Sicking
On Fri, Mar 16, 2012 at 9:19 AM, Joshua Bell jsb...@chromium.org wrote: On Thu, Mar 15, 2012 at 5:20 PM, Glenn Maynard gl...@zewt.org wrote: On Thu, Mar 15, 2012 at 6:51 PM, Jonas Sicking jo...@sicking.cc wrote: What's the use-case for the stringLength function? You can't decode into an

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Charles Pritchard
On 3/16/2012 4:25 PM, Boris Zbarsky wrote: On 3/16/12 5:25 PM, Brandon Jones wrote: Everyone knows that typed arrays /can/ be Big Endian, but I'm not aware of any devices available right now that support WebGL that are. I believe that recent Firefox on a SPARC processor would fit that

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread James Robinson
On Fri, Mar 16, 2012 at 4:25 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 3/16/12 5:25 PM, Brandon Jones wrote: Everyone knows that typed arrays /can/ be Big Endian, but I'm not aware of any devices available right now that support WebGL that are. I believe that recent Firefox on a SPARC

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Boris Zbarsky
On 3/16/12 7:43 PM, James Robinson wrote: You can s/web developers/users/ and the statement would still apply, wouldn't it? Sure, but so what? The upshot is that people are writing code that assumes little-endian hardware all over. We should just clearly make the spec say that that's what

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-16 Thread Charles Pritchard
On 3/16/2012 5:25 PM, Boris Zbarsky wrote: On 3/16/12 7:43 PM, James Robinson wrote: You can s/web developers/users/ and the statement would still apply, wouldn't it? Sure, but so what? The upshot is that people are writing code that assumes little-endian hardware all over. We should just

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-15 Thread Anne van Kesteren
On Wed, 14 Mar 2012 23:53:12 +0100, Glenn Maynard gl...@zewt.org wrote: On Wed, Mar 14, 2012 at 6:52 AM, Anne van Kesteren ann...@opera.com wrote: If we can make it a deterministic, unchanging, and defined algorithm, I think that would actually be acceptable. And ideally we do define that

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-15 Thread Jonas Sicking
On Wed, Mar 14, 2012 at 3:33 PM, Joshua Bell jsb...@chromium.org wrote: FYI, I've updated http://wiki.whatwg.org/wiki/StringEncoding A few comments: What's the use-case for the stringLength function? You can't decode into an existing datastructure anyway, so you're ultimately forced to call

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-15 Thread Glenn Maynard
On Thu, Mar 15, 2012 at 6:51 PM, Jonas Sicking jo...@sicking.cc wrote: What's the use-case for the stringLength function? You can't decode into an existing datastructure anyway, so you're ultimately forced to call decode at which point the stringLength function hasn't helped you.

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-14 Thread Anne van Kesteren
On Wed, 14 Mar 2012 01:01:42 +0100, Ian Hickson i...@hixie.ch wrote: Seems reasonable. If we have specific use cases for non-UTF-8 encodings, I agree we should support them; if that's the case, we should survey those use cases to work out what the set of encodings we need is, and add just

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-14 Thread Cedric Vivier
Hi, On Wed, Mar 14, 2012 at 06:49, Jonas Sicking jo...@sicking.cc wrote: Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof) into a decoded string. Similarly being able to encode a string into an ArrayBuffer

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-14 Thread James Graham
On 03/14/2012 12:38 AM, Tab Atkins Jr. wrote: On Tue, Mar 13, 2012 at 4:11 PM, Glenn Maynardgl...@zewt.org wrote: The API on that wiki page is a reasonable start. For the same reasons that we discussed in a recent thread (

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-14 Thread Anne van Kesteren
On Wed, 14 Mar 2012 00:50:43 +0100, Joshua Bell jsb...@chromium.org wrote: For both of the above: initially suggested use cases included parsing data as esoteric as ID3 tags in MP3 files, where encoding unspecified and is guessed at by decoders, and includes non-Unicode encodings. It was

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-14 Thread Joshua Bell
FYI, I've updated http://wiki.whatwg.org/wiki/StringEncoding * Rewritten in terms of Anne's Encoding spec and WebIDL, for algorithms, encodings, and encoding selection, which greatly simplifies the spec. This implicitly adds support for all of the other encodings defined therein - we may still

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-14 Thread Glenn Maynard
On Tue, Mar 13, 2012 at 9:47 PM, John Tamplin j...@google.com wrote: I am fine with strongly suggesting that only UTF8 be used for new things, but leaving out legacy support will severely limit the utility of this library. Not all limitations are bad, and I'd disagree with seriously. At a

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-14 Thread Joshua Bell
On Wed, Mar 14, 2012 at 3:53 PM, Glenn Maynard gl...@zewt.org wrote: It's more than a naming problem. With this string API, one side of the conversion is always a DOMString. Base64 conversion wants ArrayBuffer-ArrayBuffer conversions, so it would belong in a separate API. Huh. The

[whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Jonas Sicking
Hi All, Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof) into a decoded string. Similarly being able to encode a string into an ArrayBuffer (or part thereof). Something as simple as DOMString

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Tab Atkins Jr.
On Tue, Mar 13, 2012 at 3:49 PM, Jonas Sicking jo...@sicking.cc wrote: Hi All, Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof) into a decoded string. Similarly being able to encode a string into an

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Ian Hickson
On Tue, 13 Mar 2012, Jonas Sicking wrote: Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof) into a decoded string. Similarly being able to encode a string into an ArrayBuffer (or part thereof).

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Kenneth Russell
Joshua Bell has been working on a string encoding and decoding API that supports the needed encodings, and which is separable from the core typed array API: http://wiki.whatwg.org/wiki/StringEncoding This is the direction I prefer. String encoding and decoding seems to be a complex enough

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Jonas Sicking
On Tue, Mar 13, 2012 at 3:58 PM, Tab Atkins Jr. jackalm...@gmail.com wrote: On Tue, Mar 13, 2012 at 3:49 PM, Jonas Sicking jo...@sicking.cc wrote: Hi All, Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof)

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Jonas Sicking
On Tue, Mar 13, 2012 at 4:08 PM, Kenneth Russell k...@google.com wrote: Joshua Bell has been working on a string encoding and decoding API that supports the needed encodings, and which is separable from the core typed array API: http://wiki.whatwg.org/wiki/StringEncoding This is the

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Glenn Maynard
On Tue, Mar 13, 2012 at 5:49 PM, Jonas Sicking jo...@sicking.cc wrote: Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof) into a decoded string. Similarly being able to encode a string into an ArrayBuffer

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Kenneth Russell
On Tue, Mar 13, 2012 at 6:10 PM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Mar 13, 2012 at 4:08 PM, Kenneth Russell k...@google.com wrote: Joshua Bell has been working on a string encoding and decoding API that supports the needed encodings, and which is separable from the core typed array

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Ian Hickson
On Tue, 13 Mar 2012, Jonas Sicking wrote: Unfortunately I suspect getting anything added on the String object will take a few years given that it's too late to get into ES6 (and in any case I suspect adding ArrayBuffer dependencies to ES6 would be controversial). We can just define it

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Joshua Bell
On Tue, Mar 13, 2012 at 4:11 PM, Glenn Maynard gl...@zewt.org wrote: On Tue, Mar 13, 2012 at 5:49 PM, Jonas Sicking jo...@sicking.cc wrote: Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof) into a

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Joshua Bell
On Tue, Mar 13, 2012 at 4:11 PM, Glenn Maynard gl...@zewt.org wrote: On Tue, Mar 13, 2012 at 5:49 PM, Jonas Sicking jo...@sicking.cc wrote: Something that has come up a couple of times with content authors lately has been the desire to convert an ArrayBuffer (or part thereof) into a

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Ian Hickson
On Tue, 13 Mar 2012, Joshua Bell wrote: On Tue, Mar 13, 2012 at 4:10 PM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Mar 13, 2012 at 4:08 PM, Kenneth Russell k...@google.com wrote: Joshua Bell has been working on a string encoding and decoding API that supports the needed encodings,

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Ian Hickson
On Tue, 13 Mar 2012, Joshua Bell wrote: WHATWG makes sense, I just hadn't gotten around to shopping for a home. (Administrivia: Is there need to propose a charter addition?) You're welcome to use the WHATWG list for this. Charters are pointless and there's no need to worry about them here.

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Tab Atkins Jr.
On Tue, Mar 13, 2012 at 4:08 PM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Mar 13, 2012 at 3:58 PM, Tab Atkins Jr. jackalm...@gmail.com wrote: On Tue, Mar 13, 2012 at 3:49 PM, Jonas Sicking jo...@sicking.cc wrote: Hi All, Something that has come up a couple of times with content authors

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Tab Atkins Jr.
On Tue, Mar 13, 2012 at 4:11 PM, Glenn Maynard gl...@zewt.org wrote: The API on that wiki page is a reasonable start.  For the same reasons that we discussed in a recent thread ( http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/1589.html), conversion errors should use replacement

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Ian Hickson
On Tue, 13 Mar 2012, Joshua Bell wrote: For both of the above: initially suggested use cases included parsing data as esoteric as ID3 tags in MP3 files, where encoding unspecified and is guessed at by decoders, and includes non-Unicode encodings. It was suggested that the encoding

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread Glenn Maynard
Using Views instead of specifying the offset and length sounds good. On Tue, Mar 13, 2012 at 6:28 PM, Ian Hickson i...@hixie.ch wrote: - What's the use case for supporting anything but UTF-8? Other Unicode encodings may be useful, to decode existing file formats containing (most likely at a

Re: [whatwg] API for encoding/decoding ArrayBuffers into text

2012-03-13 Thread John Tamplin
On Tue, Mar 13, 2012 at 8:19 PM, Glenn Maynard gl...@zewt.org wrote: Using Views instead of specifying the offset and length sounds good. On Tue, Mar 13, 2012 at 6:28 PM, Ian Hickson i...@hixie.ch wrote: - What's the use case for supporting anything but UTF-8? Other Unicode encodings