Re: [go-nuts] how is xml.Decoder.CharsetReader supposed to be held?

2022-05-06 Thread 'Dan Kortschak' via golang-nuts 
On Fri, 2022-05-06 at 15:55 -0700, Ian Lance Taylor wrote:
> On Fri, May 6, 2022 at 3:07 AM 'Dan Kortschak' via golang-nuts
>  wrote:
> >
> > On Fri, 2022-05-06 at 11:22 +0200, Diego Joss wrote:
> > > Does this work for you?
> > >
> > > https://go.dev/play/p/xLRawVhcRtF
> > >
> >
> > Thanks. No, the documents are in UTF-16, and the procinst will be
> > too.
> > So it looks more like this https://go.dev/play/p/4IcXNI3yd2M. If I
> > pull
> > the proc inst out of the UTF-16, then I can get it to work;
> > https://go.dev/play/p/kHwkVWtxbNO. But this leads to the issue
> > where at
> > that point I could just decode the whole message and pass it
> > through.
> > So I don't really see the point of using CharsetReader (at least
> > not
> > with UTF-16).
>
> Yeah, that's not the kind of thing that CharsetReader can help with.
> You'll need a plain io.Reader that converts from UTF-16 to UTF-8.
>
> CharsetReader only works if the character set name is available in
> plain ASCII in the first XML definitions, but the data doesn't use
> UTF-8.  It can be used with the kinds of encodings found in the
> subdirectories of https://pkg.go.dev/golang.org/x/text/encoding.
>
> Ian
>

Thanks, Ian.

It might be moot, because it looks like the encoding declaration in the
XML that I have is lying. But in general the solution would need to
sniff the first line and then try for finding the encoding declaration.
I suspect that this is what other languages do in this situation.

Dan


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3277068af53ec326a6b3163e4f6d0242b96aa81a.camel%40kortschak.io.

Re: [go-nuts] how is xml.Decoder.CharsetReader supposed to be held?

2022-05-06 Thread Ian Lance Taylor 
On Fri, May 6, 2022 at 3:07 AM 'Dan Kortschak' via golang-nuts
 wrote:
>
> On Fri, 2022-05-06 at 11:22 +0200, Diego Joss wrote:
> > Does this work for you?
> >
> > https://go.dev/play/p/xLRawVhcRtF
> >
>
> Thanks. No, the documents are in UTF-16, and the procinst will be too.
> So it looks more like this https://go.dev/play/p/4IcXNI3yd2M. If I pull
> the proc inst out of the UTF-16, then I can get it to work;
> https://go.dev/play/p/kHwkVWtxbNO. But this leads to the issue where at
> that point I could just decode the whole message and pass it through.
> So I don't really see the point of using CharsetReader (at least not
> with UTF-16).

Yeah, that's not the kind of thing that CharsetReader can help with.
You'll need a plain io.Reader that converts from UTF-16 to UTF-8.

CharsetReader only works if the character set name is available in
plain ASCII in the first XML definitions, but the data doesn't use
UTF-8.  It can be used with the kinds of encodings found in the
subdirectories of https://pkg.go.dev/golang.org/x/text/encoding.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWUsmnRpC_a_HausGTAasgcT1862F%3Dcae04cs%3DTP2uRVA%40mail.gmail.com.

Re: [go-nuts] Re: GO program's memory footprint is confusing

2022-05-06 Thread Robert Engels 
https://www.arp242.net/static-go.html

> On May 5, 2022, at 11:27 PM, garenchan  wrote:
> 
> The binary file is not a dynamic executable.   So I don't think that will 
> happen.
> 
> 在2022年5月6日星期五 UTC+8 11:54:44 写道：
>> Are you certain that on the low memory instance there isn’t another process 
>> - maybe Go - that hasn’t already loaded the shared libraries - and on the 
>> other it hasn’t - so the RSS is different. Compare the VSZ sizes and see if 
>> they are the same. 
>> 
 On May 5, 2022, at 10:28 PM, garenchan  wrote:
 
>>> @jake, I tried 1.18 version of Go, but the problem still exists. 
>> 
>>> 
>>> I also tried using C to write program that includes memory allocation and 
>>> usage,  and this problem does not occur.
>>> 在2022年5月5日星期四 UTC+8 21:25:20 写道：
 Since no one has responded with concrete ideas, I'll throw out two 
 suggestions. They may seem obvious.
 
  First have you tries the latest version of Go? and do you get the same 
 results?
 
  Second have you run the experiment with a small binaries not from Go? I 
 would suggest something that does allocate some real memory, not a "hello 
 world" C program or something.
> On Thursday, May 5, 2022 at 7:21:39 AM UTC-4 garenchan wrote:
> 
> Both hosts have 8 cores and 16GB RAM.
> 在2022年4月30日星期六 UTC+8 00:19:44 写道：
>> What version of Go are you using (go version)?
>> 
>> $ go version
>> go version go1.17.6 linux/amd64
>> 
>> Does this issue reproduce with the latest release?
>> 
>> uncertain
>> 
>> What operating system and processor architecture are you using (go env)?
>> 
>> $ go env
>> GO111MODULE="on"
>> GOARCH="amd64"
>> GOBIN=""
>> GOCACHE="/root/.cache/go-build"
>> GOENV="/root/.config/go/env"
>> GOEXE=""
>> GOEXPERIMENT=""
>> GOFLAGS=""
>> GOHOSTARCH="amd64"
>> GOHOSTOS="linux"
>> GOINSECURE=""
>> GOMODCACHE="/root/go/pkg/mod"
>> GONOPROXY=""
>> GONOSUMDB=""
>> GOOS="linux"
>> GOPATH="/root/go"
>> GOPRIVATE=""
>> GOPROXY=""
>> GOROOT="/home/go"
>> GOSUMDB="off"
>> GOTMPDIR=""
>> GOTOOLDIR="/home/go/pkg/tool/linux_amd64"
>> GOVCS=""
>> GOVERSION="go1.17.6"
>> GCCGO="gccgo"
>> AR="ar"
>> CC="gcc"
>> CXX="g++"
>> CGO_ENABLED="1"
>> GOMOD="/home/demo/go.mod"
>> CGO_CFLAGS="-g -O2"
>> CGO_CPPFLAGS=""
>> CGO_CXXFLAGS="-g -O2"
>> CGO_FFLAGS="-g -O2"
>> CGO_LDFLAGS="-g -O2"
>> PKG_CONFIG="pkg-config"
>> GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 
>> -fdebug-prefix-map=/tmp/go-build4023324410=/tmp/go-build 
>> -gno-record-gcc-switches"
>> 
>> What did you do?
>> 
>> I encountered a memory problem with the GO program, see here for 
>> details.(https://stackoverflow.com/questions/71994609/memory-footprint-of-the-same-program-varies-greatly-in-two-similar-environments)
>> 
>> In order to simplify the analysis, I wrote a simple program to test.
>> 
>> ```go
>> package main
>> 
>> import (
>> "time"
>> )
>> 
>> func main() {
>> time.Sleep(60*time.Second)
>> }
>> ```
>> 
>> I compiled it into binary file on a linux host `host1` with kernel 4.18. 
>> Then I run it on `host1` and the process takes up close to 5MB RSS.
>> I then copy the binary file to another host `host2` with kernel 4.18. I 
>> also ran it on `host2`, but this time the process took up less than 1MB 
>> RSS.
>> I repeated the test many times and observed the same thing.
>> 
>> ```
>> $ uname -a
>> Linux host1 4.18.0 #1 SMP Wed Nov 10 20:46:19 CST 2021 x86_64 x86_64 
>> x86_64 GNU/Linux
>> 
>> $ uname -a
>> Linux host2 4.18.0 #1 SMP Fri May 8 10:59:10 UTC 2021 x86_64 x86_64 
>> x86_64 GNU/Linux
>> ```
>> 
>> Why is memory footprint of the same program in similar environments so 
>> different? What factors might be contributing to this problem?
>> 
>> What did you expect to see?
>> 
>> I would expect to see the memory footprint of the same program in 
>> similar environments be close. I look forward to your answers. Thank you 
>> very much.
>>> 
>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/d8a3e1d9-e578-4668-9b19-38b49e29cd11n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
>

Re: [go-nuts] how is xml.Decoder.CharsetReader supposed to be held?

2022-05-06 Thread 'Dan Kortschak' via golang-nuts 
On Fri, 2022-05-06 at 11:22 +0200, Diego Joss wrote:
> Does this work for you?
>
> https://go.dev/play/p/xLRawVhcRtF
>

Thanks. No, the documents are in UTF-16, and the procinst will be too.
So it looks more like this https://go.dev/play/p/4IcXNI3yd2M. If I pull
the proc inst out of the UTF-16, then I can get it to work;
https://go.dev/play/p/kHwkVWtxbNO. But this leads to the issue where at
that point I could just decode the whole message and pass it through.
So I don't really see the point of using CharsetReader (at least not
with UTF-16).

Dan


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b3554811bb056f6741c4b2f484abd2fb7e801f6f.camel%40kortschak.io.

Re: [go-nuts] Re: DeepCloning a datastructure

2022-05-06 Thread Zhaoxun Yan 
Yes there are two open source packages regarding "DeepCopy"

https://github.com/mohae/deepcopy/
https://github.com/barkimedes/go-deepcopy/

Has anyone tried them yet?

在2020年4月19日星期日 UTC+8 01:26:11 写道：

> How about doing it manually? You can define a DeepCopy method on the type 
> and manually allocate a new struct and initialize the maps and slices by 
> looping over them. That way you get to have more control over all the 
> fields and problem of private fields is solved as well. I would guess this 
> would be faster as well because no reflection is required.
>
> As far as I know Kubernetes uses this approach and they have a 
> code-generator which can conveniently generate the code for the structs.
>
> https://github.com/kubernetes/code-generator/tree/master/cmd/deepcopy-gen
>
>
>
> On Fri, Apr 17, 2020 at 2:07 PM Carsten Orthbandt  
> wrote:
>
>> The easiest way is to marshal everything into some sort of byte slice, 
>> then unmarshal back. JSON doesn't work for this if there are private 
>> fields. But gotiny (https://github.com/niubaoshu/gotiny) does. If you 
>> use interfaces in your data, you have to register their types, everthing 
>> else is automatic.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/17810cd4-dab5-4583-8944-aed38b773232%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/573fca81-61b0-4e32-b051-4a7bcdb2f6f1n%40googlegroups.com.

Re: [go-nuts] how is xml.Decoder.CharsetReader supposed to be held?

2022-05-06 Thread Diego Joss 
Does this work for you?

https://go.dev/play/p/xLRawVhcRtF

-- Diego

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGjxhKmCJWczpUhtgyWsn0dJ1_E9PcTUBOZW112bGsVQyMZq0g%40mail.gmail.com.