nice! a god old decorator example! thanks El sábado, 31 de octubre de 2020 a las 15:16:12 UTC-3, Severyn Lisovsky escribió:
> for everyone interested this is the solution in Go Playground: > https://play.golang.org/p/Rxwcwhai4Gl > > On Saturday, October 31, 2020 at 7:01:08 PM UTC+1 Severyn Lisovsky wrote: > >> Tamás Gulácsi, wow didn't know that providing bufio.Reader to >> bufio.NewReader doesn't wrap your reader. Looks like this is the solution >> I've been looking for. Thanks! >> >> On Saturday, October 31, 2020 at 6:50:18 PM UTC+1 Tamás Gulácsi wrote: >> >>> Why do you need an access to the internal bufio.Reader? >>> >>> If you provide a bufio.Reader to bufio.NewReader, then it will NOT >>> create a new reader, but give back your reader. >>> So if you keep your bufio.Reader, and give it to csv.NewReader, than you >>> will have the same *bufio.Reader >>> as what the csv.Reader's inner r ! >>> >>> Severyn Lisovsky a következőt írta (2020. október 31., szombat, 18:31:34 >>> UTC+1): >>> >>>> Tamás Gulácsi, this was basically my initial idea to do that, but >>>> unfortunately there is no access to internal bufio.Reader. See: >>>> https://golang.org/src/encoding/csv/reader.go#L170 >>>> >>>> peterGo, my file is ~100GB so downloading it just for sake of splitting >>>> doesn't make sense to me. I want for each worker to make use of >>>> NewRangeReader >>>> method >>>> <https://godoc.org/cloud.google.com/go/storage#ObjectHandle.NewRangeReader> >>>> to >>>> download only related piece of the file. >>>> >>>> ren...@ix.netcom.com ByteCount reader that wraps the underlying reader >>>> wouldn't help because csv.Reader doesn't read from underlying reader >>>> synchronically, it reads from bufio.Reader which buffers the bytes. So for >>>> example if you read 1 row from CSV (eg. 10 bytes) from underlying >>>> io.Reader >>>> will be 4096 bytes read. On the next csv.Reader.Read() call none of bytes >>>> will be read from underlying io.Reader because it will take next row out >>>> of >>>> the buffer >>>> >>>> On Saturday, October 31, 2020 at 6:02:32 PM UTC+1 Tamás Gulácsi wrote: >>>> >>>>> Give csv.NewReader your own *bufio.Reader. >>>>> Regarding (https://pkg.go.dev/pkg/bufio/#NewReaderSize) if the >>>>> underlying io.Reader is already a *bufio.Reader with a big enough size >>>>> (and >>>>> csv.NewReader uses the default 4k), >>>>> then the underlying reader is used, no new wrapping is introduced. >>>>> >>>>> This way if you use >>>>> cr := countingReader{Reader:r} >>>>> br := bufio.NewReader(cr) >>>>> csvR := csv.NewReader(br) >>>>> >>>>> then cr.N - br.Buffered() is the number of bytes read by csv.Reader, >>>>> the end of the last line read. >>>>> >>>>> Hope this helps. >>>>> >>>>> Severyn Lisovsky a következőt írta (2020. október 31., szombat, >>>>> 3:17:26 UTC+1): >>>>> >>>>>> Hi, >>>>>> >>>>>> I have difficulty counting bytes that were processed by csv.Reader >>>>>> because it reads from internally created bufio.Reader. If I pass some >>>>>> counting reader to csv.NewReader it will show not the actual number >>>>>> bytes >>>>>> "processed" by csv.Reader to receive the output I get calling >>>>>> csv.Reader.Read method, but the number of bytes copied to bufio.Reader's >>>>>> buffer internally (some bytes may be read during next csv.Reader.Read >>>>>> call >>>>>> from the buffer). >>>>>> >>>>>> Is there a way I can deal with this issue by not forking encoding/csv >>>>>> package? >>>>>> >>>>>> To give you more high-level picture - I want to split remote csv file >>>>>> to chunks. Each chunk should be standalone csv file - starting from >>>>>> actual >>>>>> beginning of the line, ending with newline byte. So I'm trying to do the >>>>>> following - split file size by the number of chunks, and for each chunk >>>>>> - >>>>>> skip first bytes up to newline symbol and read to >>>>>> offset+chunkSize+[number >>>>>> of bytes to the next newline symbol] >>>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5f0de599-972b-43c0-b88b-1337bcde8f59n%40googlegroups.com.
