Hi,
thanks for help!
But the problem might just be that you're writing many more bytes to
> your encoder than you're reading from the file.
>
Did you mean i should write it so ? Write(srcBuf[0:lenRead])
nrRead, err := src.Read(bufRead); // read from file
if err!= nil {
break
}
fmt.Printf("read read len: %d\n", nrRead)
nwEnc, err := encoder.Write(bufRead[0:nrRead]) // write to encoder
This means the size of buf2 could be much larger than the original
> file and also full of junk data from previous reads and zeroed bytes.
>
One q i have here, is not buf2 supposed to be drained by the following call
to Read, at least partially
_, err = buf2.Write(r) // write to decoder
fmt.Println(err)
if decoder==nil {
decoder = flate.NewReader(buf2)
}
r2 := make([]byte, dReadSize*2)
_, err = decoder.Read(r2) // read from decoder // <- here, it should
remove some data of buf2, no ?
fmt.Println(err)
Besides that, i totally agree this example code is full of wrongness, I
rewrote it,
package main
import (
"fmt"
"os"
"bytes"
// "encoding/base64"
"compress/flate"
"io"
"time"
)
func main () {
// okSize := 16384
// koSize := 64384
// koSize := 16385 // yes 1 more and it breaks :s
// change read size to 16384 and everything is ok
fReadSize := 16385
dReadSize := 64384 // dread seems not impacting
src, _ := os.Open("src.avi")
bufRead := make([]byte, fReadSize)
bufEnc := new(bytes.Buffer)
encoder, _ := flate.NewWriter(bufEnc, 9)
bufDec := new(bytes.Buffer)
var decoder io.ReadCloser
for {
fmt.Println("-----------")
nrRead, err := src.Read(bufRead); // read from file
if err!= nil {
break
}
fmt.Printf("read read len: %d\n", nrRead)
nwEnc, err := encoder.Write(bufRead[0:nrRead]) // write to encoder
fmt.Println(err)
fmt.Printf("encode write len: %d\n", nwEnc)
err = encoder.Flush()
fmt.Println(err)
encSlice := bufEnc.Bytes() // read from encoder
// fmt.Println(encSlice[0:20])
fmt.Printf("encode read len: %d\n", len(encSlice))
fmt.Printf("bufEnc len: %d\n", bufEnc.Len())
fmt.Printf("bufEnc cap: %d\n", bufEnc.Cap())
bufEnc.Truncate(0)
// fmt.Println("_______")
// fmt.Println(encSlice[0:20])
// fmt.Println(r)
nwDec, err := bufDec.Write(encSlice) // write to decoder
fmt.Println(err)
fmt.Printf("decode write len: %d\n", nwDec)
if decoder==nil {
decoder = flate.NewReader(bufDec)
}
sliceDec := make([]byte, dReadSize*2)
nrDec, err := decoder.Read(sliceDec) // read from decoder
fmt.Println(err)
fmt.Printf("decode read len: %d\n", nrDec)
fmt.Printf("bufDec len: %d\n", bufDec.Len())
fmt.Printf("bufDec cap: %d\n", bufDec.Cap())
time.Sleep( 1 * time.Second)
}
}
Which after few iterations will yield
read read len: 16385
<nil>
encode write len: 16385
<nil>
encode read len: 15150
bufEnc len: 15150
bufEnc cap: 16320
<nil>
decode write len: 15150
<nil>
decode read len: 8
bufDec len: 51234
bufDec cap: 140569
So definitely the decoder is holding on something. Its cap and len are
irremediably growing.
The thing now is if I try to read it twice time consecutively,
the second will read will always yield 0 bytes and provide me an
UnexpectedEOF error.
Does this error leaves the decoder in a unrecoverable state ? I m unsure.
Looks likes i can t drain it :x
Le lundi 4 juillet 2016 06:17:51 UTC+2, Jesse McNelis a écrit :
>
> On Mon, Jul 4, 2016 at 3:35 AM, mhhcbon <cpasmabo...@gmail.com
> <javascript:>> wrote:
> > Hi,
> >
> > I have this program which reads file, flate encode then flate decode the
> > data.
> >
> > I noticed that when i used different size for the slice of []byte to
> read
> > data, the program will retain memory when the size is > 16384.
> > When its lower than this value everything is fine, but 16385 breaks.
> >
> > I don t quite understand the reason of this behavior, can someone help
> me to
> > understand what s going on there ?
>
> I can't see anywhere that this program could be holding on to extra
> memory it's not using.
> But the problem might just be that you're writing many more bytes to
> your encoder than you're reading from the file.
>
> _, err := src.Read(b); // read from file
>
> Read() isn't required to fill the whole buffer it's given. It can read
> a single byte and return.
> Because you're ignoring the value telling you how many bytes it read
> you're passing the whole 16385 slice to your encoder even though you
> might have read much less than 16385 bytes.
>
> This means the size of buf2 could be much larger than the original
> file and also full of junk data from previous reads and zeroed bytes.
>
> Read() is a low level call that you should avoid calling directly
> because it's tricky to get right.
>
> For an example of how to properly call a Read() see the implementation
> of io.Copy()
> https://golang.org/src/io/io.go?#L366
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
