I see. Thanks for the help!
>You could possibly capture the bytes of each compressed message, and
> then construct a new websocket-free program that starts with those
> bytes and knows what each decompressed message should be.
I went ahead and wrote some example code. It consists of two
implementations. One deflates the entire compressed stream in the expected
fashion. The second tries to decompress the payload of each message as it
arrives.
The data was dumped from a live websockets session between Chrome and a
Node.js server using Socket.io.
I added a delimiter between each message in the log file so I could
separate them out in the test code to simulate a series of individual
messages arriving.
Here's the common code for the two proof of concepts:
f, err := os.Open("flate_log.d")
if err != nil {
panic(err)
}
buf := bytes.NewBuffer(nil)
_, err = buf.ReadFrom(f)
if err != nil {
panic(err)
}
messages := bytes.Split(buf.Bytes(), magic_seperator)
compData := bytes.NewBuffer(nil)
deflater := flate.NewReader(compData)
This works as expected, so the data must be sound:
// Unsplit into a single contiguous stream
for _, msg := range messages {
// msg in this case is a single websockets message
compData.Write(msg)
}
decompressed, err := ioutil.ReadAll(deflater)
fmt.Printf("Err: %v. Decompressed:\n%s\n\n", err,
hex.Dump(decompressed[0:len(decompressed)]))
The following example only works for the first few messages before deflate
starts complaining about corrupt bytes. The implementation breaks the
streaming model like you said. But, AFAIK, some variant of it needs to work
to correctly support permessage-deflate with websockets.
for _, msg := range messages {
// msg in this case is a single websockets message
compData.Write(msg)
decompressed, err := ioutil.ReadAll(deflater)
fmt.Printf("Err: %v. Decompressed:\n%s\n\n", err,
hex.Dump(decompressed[0:len(decompressed)]))
// Interestingly, ignoring an unexpected EOF allowed more of the
messages to be decompressed
// provided we use Reset here.
// Without this conditional, things break sooner.
if err == io.ErrUnexpectedEOF {
deflater.(flate.Resetter).Reset(compData, nil)
continue
}
if err != nil {
break
}
compData.Reset()
}
Is there a different implementation of deflate that you think I should try?
On Thu, Jul 14, 2016 at 9:30 PM, Nigel Tao <nigel...@golang.org> wrote:
> On Fri, Jul 15, 2016 at 9:14 AM, Adam Keeton <ajkee...@gmail.com> wrote:
> > A small self-contained example with this code would be tricky with all of
> > the websocket handling stuff in the way.
>
> You could possibly capture the bytes of each compressed message, and
> then construct a new websocket-free program that starts with those
> bytes and knows what each decompressed message should be.
>
>
> > I'm not entirely clear with how to work with Reset, specifically the use
> of
> > dictionaries. Passing 'nil' as a dictionary hasn't gotten me anywhere.
> I'm
> > guessing that might be the key detail for maintaining the sliding window?
>
> You might need new API on a flate decompressor to e.g. ack and clear
> an explicit mypkg.EndOfMessage error that you repeatedly thread
> through the underlying reader. The compress/flate package, like all
> io.Reader implementations, expects to read and deliver streams of
> bytes, not delimited messages, but websocket (IIUC) isn't really a
> stream (two messages "abc" and "def" isn't semantically equivalent to
> a single message "abcdef"), and the two models don't quite fit well.
>
> That new API would need to be like the flate.Resetter dance, due to
> backwards compat on flate.NewReader returning an io.ReadCloser.
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
