[go-nuts] Re: Can't read >32KBytes from http response body.

[Miha Vrhovnik]

I'm reading a lot of gzipped responses with Go client, so I'd say that this 
is probably problematic http server and not the client.

On Thursday, June 23, 2022 at 2:00:18 AM UTC+2 Kevin Chowski wrote:

> Do you have a self-contained reproducer? E.g., a single Go program that 
> hosts a server (either using the server functionality from package 
> net/http, or hard-coded HTTP responses) and shows that using the client 
> logic in net/http causes this problem? Preferably shared via 
> play.golang.org link. If the problem is in the Go standard library, that 
> is the best avenue for getting the root cause fixed, which usually is 
> better than using workarounds in your application.
>
> On Wednesday, June 22, 2022 at 8:41:54 AM UTC-6 minxie...@gmail.com wrote:
>
>> We've managed to identify the issue and found a temporary solution. 
>>
>> The issue only happens when the stream encoding is "gzip" and the 
>> uncompressed size is greater than 32KBytes. Side by side we've compared the 
>> server stream with plain text and gzip. The gzip uncompression is 
>> automatically engaged if server response stream encoding is detected. If 
>> the uncompressed data is truncated at 32768 bytes.
>>
>>
>> https://go.googlesource.com/go/+/refs/heads/dev.boringcrypto.go1.17/src/net/http/h2_bundle.go#8835.
>>  
>> (same in 1.19).
>>
>> The client could use either "Accept-Encoding: identity" header or disable 
>> the compression in the transport instance to ask for an uncompressed stream.
>>
>> Look like a Golang net/http bug to me.
>>
>> On Monday, June 20, 2022 at 3:19:26 AM UTC-5 Brian Candler wrote:
>>
>>> 1. What's the web server you're connecting to?  (Is it written in go, 
>>> and if so, can you share it?)
>>> 2. What happens if you use io.Read instead of io.ReadAtLeast?
>>> 3. What happens if you remove the time.Sleep()? (Maybe the target 
>>> webserver has a timeout which requires the transaction to be completed 
>>> within 30 seconds?)
>>>
>>> *io.ReadAtLeast* does exactly what it says: read *at least* that number 
>>> of bytes.  If the sender has nothing more to send, and does not end the 
>>> stream, then it will wait indefinitely for more data.  Note that 
>>> io.ReadAtLeast never returns io.EOF, but it may return io.ErrUnexpectedEOF 
>>> if it's unable to read the number of requested bytes.  For source code see 
>>> here 
>>> 
>>> .
>>>
>>> However, even if the sender does drop the connection, it does seem weird 
>>> that you don't get an error.  (I don't suppose you are going through a 
>>> badly-configured firewall with an aggressive TCP timeout and which doesn't 
>>> send RST?)
>>>
>>> On Monday, 20 June 2022 at 06:44:38 UTC+1 minxie...@gmail.com wrote:
>>>
 All,

 I got puzzled by how to read a response size > 32KBytes from http 
 resp.Body. All I get is 32768 bytes. With either io.Read functions or 
 bufio.Reader.


















 *buffer := make([]byte, 1024)var total_bytes_read int for {  len, err 
 := io.ReadAtLeast(res.Body, buffer, 1024)  total_bytes_read += len  
 log.Println(len, "bytes recevied, total received bytes: ", 
 total_bytes_read)  if err != nil {if err == io.EOF {  
 log.Println("EOF: last chunk received")} else {  log.Println(err)  
   }break  }   time.Sleep(1 * time.Second)}   log.Println("Total bytes 
 received:", total_bytes_read)err = res.Body.Close()*

 What I see is: 

 *{"level":"info","msg":"1024 bytes recevied, total received bytes: 
 1024","time":"2022-06-19T19:47:48-05:00"} *
 *{"level":"info","msg":"1024 bytes recevied, total received bytes: 
 2048","time":"2022-06-19T19:47:49-05:00"} *

 *{"level":"info","msg":"1024 bytes recevied, total received bytes: 
 3072","time":"2022-06-19T19:47:50-05:00"}*
 *...*
 *{"level":"info","msg":"1024 bytes recevied, total received bytes: 
 30720","time":"2022-06-19T19:48:17-05:00"}*
 *{"level":"info","msg":"1024 bytes recevied, total received bytes: 
 31744","time":"2022-06-19T19:48:18-05:00"} *
 *{"level":"info","msg":"1024 bytes recevied, total received bytes: 
 32768","time":"2022-06-19T19:48:19-05:00"}*

 Then it got stuck and waiting for return from the io.Read. Not sure if 
 what I did is correct. Got the same observation when using more popular 
 bufio.Read as well.

 Thanks!
 -Min



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b0021441-b29d-4356-b56a-a79f78173c71n%40googlegroups.com.

Re: [go-nuts] Memory usage

[Miha Vrhovnik]

And if you are using a cgo and didn't replace the standard malloc with 
jemaloc you are probably suffering from memory fragmentation or whatever 
the hell happens.
We had to restart the processors every 1000 iterations because of that. 
There was no memory leak according to valgrind. (about 300kb), yet the RSS 
was constantly rising. Calling exactly the same C api with a C shim the RSS 
was constant about 100M. And api was extremely simplified all large memory 
allocations happened on the C side. pprof showed small amount of used 
memory yet the RSS was in gigabytes.


BR;
Miha
On Monday, December 13, 2021 at 7:24:57 PM UTC+1 kra...@skepticism.us wrote:

> If you're using cgo to link with C/C++ code then a memory leak in that API 
> is the first place to start looking..
>
> On Mon, Dec 13, 2021 at 10:14 AM Gareth Owenson  
> wrote:
>
>> Hi all
>>
>> We have a go application that processes large numbers of documents.  It 
>> behaves like it has a memory leak, eventually exhausting system ram after a 
>> few hours.  I've setup pprof and looked at the various outputs, and that is 
>> adamant that the application is only using some 50MB of ram, yet Linux top 
>> reports it using much more (and growing).
>>
>> I've tried setting madvdontneed to no avail.
>>
>> Any help on better ways to identify the cause would be greatly 
>> appreciated.
>>
>> best
>> Gareth
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/409a950d-b270-4ef7-a965-ea3d450e74f5n%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0707e376-d6d7-40ff-9803-e5534c98b62dn%40googlegroups.com.

[go-nuts] cgo and fatal: morestack on g0

[Miha Vrhovnik]

Hey,

we are using a library via CGO that is the emulator for a real device. 
That library also contains a javascript engine which calls to Go for 
communication to the outside world. The problem we are facing is that we 
are getting more and more crashes with "morestack on g0". 
I have found tickets on the go github (for windows) that show that this 
occurs where go is out of stack. When looking at the gdb bt, the callstack 
doesn't seem so deep. 
And before anyone asks, yes the call stack looks like this  
go->c->c->.->c->go (crash).

The main is locked to the Os thread with runtime.LockOSThread() however 
removing this doesn't help.

Is there any chance of upping up the available stack in CGO cases.

The behavior is the same on 1.16.7 and newly released 1.17.

BR,
Miha

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/01278325-357b-4ec5-95a2-d126bd38009dn%40googlegroups.com.

[go-nuts] interface implementing subset not compatible

[Miha Vrhovnik]

I'm trying to put external library under the interface but it seems that 
this is not possible the way I thought it would be...

My interfaces are as following...
















*type Beginner interface {Begin(ctx context.Context) (Tx, error)}type 
Querier interface {Exec(ctx context.Context, sql string, arguments 
...interface{}) (commandTag pgconn.CommandTag, err error)Query(ctx 
context.Context, sql string, args ...interface{}) (pgx.Rows, error)   
 QueryRow(ctx context.Context, sql string, args ...interface{}) 
pgx.Row}type Tx interface {Commit(ctx context.Context) error   
 Rollback(ctx context.Context) errorQuerier}*

Now I have a function


*func NewFoo (b Beginer) {}*

Into that function I want to pass a struct instance with the method that 
has a following signature. (This function is not under my control).

*Begin(ctx context.Context) (pgx.Tx, error)*

Now the* pgx.Tx* has a following signature



















*type Tx interface {Begin(ctx context.Context) (Tx, error)   
 Commit(ctx context.Context) errorRollback(ctx context.Context) error   
 CopyFrom(ctx context.Context, tableName Identifier, columnNames []string, 
rowSrc CopyFromSource) (int64, error)SendBatch(ctx context.Context, b 
*Batch) BatchResultsLargeObjects() LargeObjectsPrepare(ctx 
context.Context, name, sql string) (*pgconn.StatementDescription, error)   
 Exec(ctx context.Context, sql string, arguments ...interface{}) 
(commandTag pgconn.CommandTag, err error)Query(ctx context.Context, sql 
string, args ...interface{}) (Rows, error)QueryRow(ctx context.Context, 
sql string, args ...interface{}) Row// Conn returns the underlying 
*Conn that on which this transaction is executing.Conn() *Conn}*

The go is giving me the following error.

cannot use me.DBpgx() (type *pgxpool.Pool) as type db.Beginner in argument 
to service.NewFoo:
*pgxpool.Pool does not implement db.Beginner (wrong type for Begin 
method)
have Begin(context.Context) (pgx.Tx, error)
want Begin(context.Context) (db.Tx, error)

how come that this is forbidden as my Tx interface is clearly a subset of 
pgx.Tx

BR,
Miha

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9fc0ff89-867a-464b-a955-07a63eec4296n%40googlegroups.com.

[go-nuts] Re: Mysterious RSS memory spike

[Miha Vrhovnik]

Hey, Vlad,
it's not that. we have a similar or rather the exactly the same problem... 
where the OS doesn't claim the memory back.. this is also annoying to the 
point we had to disable memory under pressure notifications for that 
specific host. 
Also valgrind doesn't report any leaks... simulating the same with C app 
the memory is returned to the OS almost immediately.

Fortunately for us this is in the worker processes where we can restart 
them after about 1000 iterations.

BR,
Miha


On Monday, August 24, 2020 at 11:05:00 AM UTC+2 vlad...@varank.in wrote:

> Hey,
>
> I haven't looked deep but I recall there had been a note about runtime 
> change in Go 1.13's https://golang.org/doc/go1.13#runtime That is
>
> > The runtime is now more aggressive at returning memory to the operating 
> system to make it available to co-tenant applications [..] However, on many 
> OSes, including Linux, the OS itself reclaims memory lazily, so process RSS 
> will not decrease until the system is under memory pressure. 
>
> Could that be the behaviour you observe (although, since you don't see the 
> same in C implementation, I might be confused, sorry in advance).
>
> On Sunday, August 23, 2020 at 5:05:22 PM UTC+2 Manish Rai Jain wrote:
>
>> Hey Gophers,
>>
>> I'm puzzled by a mysterious RSS memory spike in my Go program, when all 
>> memory allocations are happening via Cgo. I assert that there are no memory 
>> leaks in the program. And have written another C program with similar logic 
>> which does NOT show RSS memory spiking. So, I suspect this is something to 
>> do with Go memory.
>>
>> 
>> Program:
>>
>> https://github.com/dgraph-io/ristretto/pull/186
>>
>> This PR creates a Go memtest program, which does this:
>> - Uses z.Calloc and z.Free to allocate Go struct (S) and a byte slice 
>> inside it. All allocations are happening in Cgo, and being type casted into 
>> Go. No allocations are happening in Go (except a 32 MB fill slice).
>> - z.NumAllocBytes is tracking memory allocated and freed by these calls.
>> - Increases memory usage to 16 GB (as reported by z.NumAllocBytes).
>> - Decreases it back to 1 GB.
>> - Repeats this cycle.
>> - On Ctrl+C, it deallocates everything and asserts that Cgo memory 
>> allocated is zero.
>>
>> I was concerned about memory fragmentation, so created a very similar C 
>> program which does the same thing (memtestc).
>>
>> Please feel free to run either of the Go or C programs. They should 
>> compile and run easily.
>>
>> Behavior:
>>
>> Run the program with: `go build . && ./memtest` . Go pprof heap shows 32 
>> MB used, to account for the fill slice. However, RSS reported keeps roughly 
>> increasing every cycle.
>>
>> I'm using Go 1.14.4 and on it, RSS jumps to 22GB after a few cycles. 
>> memtestc 
>> (C equivalent, compiled with gcc) does not show this behavior. The RSS goes 
>> down to 1GB-ish every cycle.
>>
>> Any clues why the RSS is much higher than expected in Go and keeps 
>> climbing in Go?
>>
>> —
>> Manish
>> Founder, https://dgraph.io
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/039ca670-90fd-488d-a5ad-7bf6d3303885n%40googlegroups.com.

Re: [go-nuts] An important proposal will fail without your support

[miha . vrhovnik]

https://github.com/golang/text/pull/9

And I cannot find the issue, but there are a few with extract command not 
working and crashing.

On Wednesday, March 11, 2020 at 9:40:36 AM UTC+1, kortschak wrote:
>
> Why do you say that? 
>
> ~/src/golang.org/x/text/message [master*]$ go env GO111MODULE 
> on 
> ~/src/golang.org/x/text/message [master*]$ go version 
> go version go1.14 linux/amd64 
> ~/src/golang.org/x/text/message [master*]$ go test 
> PASS 
> ok  golang.org/x/text/message0.024s 
>
>
> On Wed, 2020-03-11 at 01:08 -0700, miha.v...@gmail.com  
> wrote: 
> > AFAIK this doesn't work in recent go versions and even less if you 
> > are using modules. 
> > 
> > On Wednesday, March 11, 2020 at 6:39:36 AM UTC+1, Tamás Gulácsi 
> > wrote: 
> > > Hi Jon, 
> > > Have you read https://godoc.org/golang.org/x/text/message ? 
> > > It's not gettext, but a better - though not perfect - solution for 
> > > this i18n problem. 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "golang-nuts" group. 
> > To unsubscribe from this group and stop receiving emails from it, 
> > send an email to golan...@googlegroups.com . 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/golang-nuts/e700071c-1138-439c-83c5-93eac98ecb0f%40googlegroups.com
>  
> > . 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1b001ff5-10ec-4d9e-a78e-c89b7b13b096%40googlegroups.com.

Re: [go-nuts] An important proposal will fail without your support

[miha . vrhovnik]

AFAIK this doesn't work in recent go versions and even less if you are 
using modules.

On Wednesday, March 11, 2020 at 6:39:36 AM UTC+1, Tamás Gulácsi wrote:
>
> Hi Jon,
>
> Have you read https://godoc.org/golang.org/x/text/message ?
>
> It's not gettext, but a better - though not perfect - solution for this 
> i18n problem.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e700071c-1138-439c-83c5-93eac98ecb0f%40googlegroups.com.

Re: [go-nuts] ultra slow os.Exec

[miha . vrhovnik]

The first wait is on read, the 2nd on futex
[pid 20441] read(5, 
"\0\0\0@\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 64) 
= 64
[pid 20441] rt_sigprocmask(SIG_SETMASK, [HUP INT USR1 USR2 ALRM CHLD IO], 
NULL, 8) = 0
[pid 20441] rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 USR2 ALRM CHLD IO], 
[HUP INT USR1 USR2 ALRM CHLD IO], 8) = 0
[pid 20441] rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 USR2 ALRM CHLD IO], 
[HUP INT USR1 USR2 ALRM CHLD IO], 8) = 0
[pid 20441] writev(3, 
[{iov_base="\34\0\0\\0\0\0\0\0\0\0\2\0\0\0\250r\204\377\0\0\0\0\0ty\270\377\377\377\377"...,
 
iov_len=64}, 
{iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
 
iov_len=40}, {iov_base="\1\0\0\0\20\0\0\0", iov_len=8}], 3) = 112
[pid 20441] read(5, 
"\3\1\0\0\0\0\0\0\330\2656\374w\237\325\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
64) = 64
[pid 20441] rt_sigprocmask(SIG_SETMASK, [HUP INT USR1 USR2 ALRM CHLD IO], 
NULL, 8) = 0
[pid 20441] rt_sigprocmask(SIG_SETMASK, [HUP INT USR1 USR2 ALRM CHLD IO], 
NULL, 8) = 0
[pid 20441] read(7, 


 
[pid 20437] <... futex resumed> )   = -1 ETIMEDOUT (Connection timed 
out)
[pid 20437] nanosleep({tv_sec=0, tv_nsec=2}, NULL) = 0
[pid 20437] futex(0x59fb70, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0}


BR,
Miha

On Tuesday, November 19, 2019 at 10:09:12 PM UTC+1, Ian Lance Taylor wrote:
>
> On Tue, Nov 19, 2019 at 11:45 AM > 
> wrote: 
> > 
> > > Doesn't seem to happen with a exec of "/usr/bin/date" so it is most 
> likely something wine+Go specific. 
> > Yeah, I did try catting the file and it was fast enough. So you are 
> probably right. 
> > 
> > Whats interesting is if I do it indirectly via shell script (then this 
> is only 5 times slower not 28): But still too slow. Hell PHP can't be 
> faster at this than Go. 
>
> Use strace -f to see what the Go program is doing. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c12d38c3-618f-499a-bcaf-d2862272cfed%40googlegroups.com.

Re: [go-nuts] ultra slow os.Exec

[miha . vrhovnik]

> Doesn't seem to happen with a exec of "/usr/bin/date" so it is most 
likely something wine+Go specific. 
Yeah, I did try catting the file and it was fast enough. So you are 
probably right.

Whats interesting is if I do it indirectly via shell script (then this is 
only 5 times slower not 28): But still too slow. Hell PHP can't be faster 
at this than Go.

cat /tmp/test 
#!/bin/sh
WINEDEBUG=err-all,fixme-all WINEARCH=win32 /opt/wine-staging/bin/wine 
cmd.exe /c help

== go ==
func main() {
now := time.Now()
ctx, cancel := context.WithTimeout(context.Background(), 
120*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "/tmp/test")
//cmd := exec.CommandContext(ctx, "/opt/wine-staging/bin/wine", "cmd", 
"/c", "help")
//cmd.Env = []string{"WINEDEBUG=err-all,fixme-all", "WINEARCH=win32"}
output, err := cmd.CombinedOutput()
if err != nil {
fmt.Println(err)
}
fmt.Println(string(output))
fmt.Println(time.Now().Sub(now))
}


time bin/osexec 

real0m5,256s
user0m0,015s
sys0m0,221s

BR,
Miha

On Tuesday, November 19, 2019 at 6:13:23 PM UTC+1, Robert Engels wrote:
>
>
>
>
>
>
>
> -Original Message- 
> >From: Robert Engels > 
> >Sent: Nov 19, 2019 10:54 AM 
> >To: Ian Lance Taylor >, 
> miha.v...@gmail.com  
> >Cc: golang-nuts > 
> >Subject: Re: [go-nuts] ultra slow os.Exec 
> > 
> > 
> >I am guessing - because it is close to 30 secs (which sounds a lot like a 
> timeout value) - that there is some problem in hand-off with the file 
> descriptors and the descriptor is getting locked, causing some sort of 
> timeout. 
> > 
> >If it was "any exe" I'm sure someone would of reported it before. It 
> should be trivial to write a test. I'll be back... 
> > 
> > 
> > 
> > 
> >-Original Message- 
> >>From: Ian Lance Taylor > 
> >>Sent: Nov 19, 2019 10:28 AM 
> >>To: miha.v...@gmail.com  
> >>Cc: golang-nuts > 
> >>Subject: Re: [go-nuts] ultra slow os.Exec 
> >> 
> >>On Tue, Nov 19, 2019 at 7:29 AM > 
> wrote: 
> >>> 
> >>> Robert is right, all 3 examples are the same (they execute the same 
> command with wine being set up and then teared down again). wine itself is 
> not an issue. It's go's exec that does something extremely funny. 
> >> 
> >>Sorry, I misread the original note.  But if I'm reading it correctly 
> >>now, I find these results pretty hard to believe.  Can anybody else 
> >>replicate them? 
> >> 
> >>Ian 
> >> 
> >> 
> >>> On Tuesday, November 19, 2019 at 4:22:47 PM UTC+1, Robert Engels 
> wrote: 
>  
>  
>  I think the point the OP is making is that when he runs the command 
> from the Linux command line it completes in 2 sec - so the Wine startup 
> time should not be the issue. 
>  
>  
>  -Original Message- 
>  >From: Ian Lance Taylor  
>  >Sent: Nov 19, 2019 9:15 AM 
>  >To: miha.v...@gmail.com 
>  >Cc: golang-nuts  
>  >Subject: Re: [go-nuts] ultra slow os.Exec 
>  > 
>  >On Tue, Nov 19, 2019 at 6:52 AM  wrote: 
>  >> 
>  >> I'm running windows cli exe on linux via wine using os.Exec 
> command.. this takes 28s to complete. 
>  >> If I run the same command via linux cli it takes 2s to complete, 
> if I run the same command via php's exec it also takes about 2 seconds to 
> complete. 
>  >> 
>  >> == linux cli == 
>  >> 
>  >> time WINEDEBUG=err-all,fixme-all WINEARCH=win32 
> /opt/wine-staging/bin/wine cmd.exe /c help 
>  >>  
>  >> real0m1,366s 
>  >> user0m0,002s 
>  >> sys0m0,225s 
>  >> 
>  >> == php == 
>  >> cat t.php 
>  >>   >> $out=array(); 
>  >> 
>  >> exec('WINEDEBUG=err-all,fixme-all WINEARCH=win32 
> /opt/wine-staging/bin/wine cmd.exe /c help', $out); 
>  >> 
>  >> var_dump($out); 
>  >> 
>  >> . 
>  >> 
>  >> real0m1,427s 
>  >> user0m0,035s 
>  >> sys0m0,216s 
>  >> 
>  >> 
>  >> ==go exec== 
>  >> 
>  >> func main() { 
>  >> now := time.Now() 
>  >> ctx, cancel := context.WithTimeout(context.Background(), 
> 120*time.Second) 
>  >> defer cancel() 
>  >> cmd := exec.CommandContext(ctx, "/opt/wine-staging/bin/wine", 
> "cmd", "/c", "help") 
>  >> cmd.Env = []string{"WINEDEBUG=err-all,fixme-all", 
> "WINEARCH=win32"} 
>  >> output, err := cmd.CombinedOutput() 
>  >> if err != nil { 
>  >> fmt.Println(err) 
>  >> } 
>  >> fmt.Println(string(output)) 
>  >> fmt.Println(time.Now().Sub(now)) 
>  >> } 
>  >> 
>  >> time bin/osexec 
>  >> 
>  >> real0m39,915s 
>  >> user0m17,062s 
>  >> sys0m0,195s 
>  >> 
>  >> 
>  >> * I have tried to set stdin and std out directly to the one from 
> os (no difference) 
>  >> * I have replaced cmd.CombinedOutput with cmd.Run() (no 
> difference) 
>  >> 
>  >> go version 
>  >> go version go1.13.3 

Re: [go-nuts] ultra slow os.Exec

[miha . vrhovnik]

Robert is right, all 3 examples are the same (they execute the same command 
with wine being set up and then teared down again). wine itself is not an 
issue. It's go's exec that does something extremely funny.

BR,
Miha

On Tuesday, November 19, 2019 at 4:22:47 PM UTC+1, Robert Engels wrote:
>
>
> I think the point the OP is making is that when he runs the command from 
> the Linux command line it completes in 2 sec - so the Wine startup time 
> should not be the issue. 
>
>
> -Original Message- 
> >From: Ian Lance Taylor > 
> >Sent: Nov 19, 2019 9:15 AM 
> >To: miha.v...@gmail.com  
> >Cc: golang-nuts > 
> >Subject: Re: [go-nuts] ultra slow os.Exec 
> > 
> >On Tue, Nov 19, 2019 at 6:52 AM > 
> wrote: 
> >> 
> >> I'm running windows cli exe on linux via wine using os.Exec command.. 
> this takes 28s to complete. 
> >> If I run the same command via linux cli it takes 2s to complete, if I 
> run the same command via php's exec it also takes about 2 seconds to 
> complete. 
> >> 
> >> == linux cli == 
> >> 
> >> time WINEDEBUG=err-all,fixme-all WINEARCH=win32 
> /opt/wine-staging/bin/wine cmd.exe /c help 
> >>  
> >> real0m1,366s 
> >> user0m0,002s 
> >> sys0m0,225s 
> >> 
> >> == php == 
> >> cat t.php 
> >>  >> $out=array(); 
> >> 
> >> exec('WINEDEBUG=err-all,fixme-all WINEARCH=win32 
> /opt/wine-staging/bin/wine cmd.exe /c help', $out); 
> >> 
> >> var_dump($out); 
> >> 
> >> . 
> >> 
> >> real0m1,427s 
> >> user0m0,035s 
> >> sys0m0,216s 
> >> 
> >> 
> >> ==go exec== 
> >> 
> >> func main() { 
> >> now := time.Now() 
> >> ctx, cancel := context.WithTimeout(context.Background(), 
> 120*time.Second) 
> >> defer cancel() 
> >> cmd := exec.CommandContext(ctx, "/opt/wine-staging/bin/wine", 
> "cmd", "/c", "help") 
> >> cmd.Env = []string{"WINEDEBUG=err-all,fixme-all", "WINEARCH=win32"} 
> >> output, err := cmd.CombinedOutput() 
> >> if err != nil { 
> >> fmt.Println(err) 
> >> } 
> >> fmt.Println(string(output)) 
> >> fmt.Println(time.Now().Sub(now)) 
> >> } 
> >> 
> >> time bin/osexec 
> >> 
> >> real0m39,915s 
> >> user0m17,062s 
> >> sys0m0,195s 
> >> 
> >> 
> >> * I have tried to set stdin and std out directly to the one from os (no 
> difference) 
> >> * I have replaced cmd.CombinedOutput with cmd.Run() (no difference) 
> >> 
> >> go version 
> >> go version go1.13.3 linux/amd64 
> >> I'm totally clueless on what's going on. 
> > 
> >How long does it take to just run the wine command from the shell? 
> > 
> >My first guess would certainly be that the problem is in wine, not in 
> >Go's os/exec package.  Starting up a Windows emulator has to take a 
> >certain amount of time. 
> > 
> >Ian 
> > 
> >-- 
> >You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group. 
> >To unsubscribe from this group and stop receiving emails from it, send an 
> email to golan...@googlegroups.com . 
> >To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXJZy61dX_UetsKY0r%3DNMcoBcE4sCuFZVzvvPme%3DjTTDg%40mail.gmail.com.
>  
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/037e16e2-158a-4cf7-8a5f-8b940d55e64a%40googlegroups.com.

[go-nuts] ultra slow os.Exec

[miha . vrhovnik]

Hey,

I'm running windows cli exe on linux via wine using os.Exec command.. this 
takes 28s to complete. 
If I run the same command via linux cli it takes 2s to complete, if I run 
the same command via php's exec it also takes about 2 seconds to complete.

== linux cli ==

time WINEDEBUG=err-all,fixme-all WINEARCH=win32 /opt/wine-staging/bin/wine 
cmd.exe /c help

real0m1,366s
user0m0,002s
sys0m0,225s

== php ==
cat t.php 
https://groups.google.com/d/msgid/golang-nuts/9d6620a7-4c3a-4f3a-86a0-6222b898aa6c%40googlegroups.com.

Re: [go-nuts] cgo memory fragmetation?

[miha . vrhovnik]

It's not that. The system had 4G of memory initially as it is supposed to 
be enough for 18 processors (They should max out at around 250M but have 
the typical of about 100M) and it started swapping heavily. (I then raised 
the limits to 16G) so things can continue to run.

BR,
Miha

On Monday, October 7, 2019 at 4:46:32 PM UTC+2, Ian Lance Taylor wrote:
>
> On Mon, Oct 7, 2019 at 12:54 AM > 
> wrote: 
> > 
> > We are having a very weird problem, where the process memory keeps 
> rising, but both the pprof and valgrind report no memory leaks. But the RSS 
> just keeps rising.The C part is in external so library 
>
> In the absence of other memory pressure on the system as a whole, 
> increasing RSS just means that your program is using different 
> addresses for allocated memory.  This isn't necessarily unexpected. 
> If overall memory usage is not increasing, there may not be a problem 
> here. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3456b19b-e9a3-44a3-848a-23f7d90412cd%40googlegroups.com.

Re: [go-nuts] cgo memory fragmetation?

[miha . vrhovnik]

This doesn't seem to help...

gc 589 @171.013s 0%: 0.012+0.69+0.17 ms clock, 0.41+0.079/2.9/1.6+5.5 ms 
cpu, 6->6->5 MB, 7 MB goal, 32 P
scvg: 0 MB released
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 52, consumed: 8 (MB)
gc 590 @171.018s 0%: 0.25+0.89+0.020 ms clock, 8.2+0.28/3.8/2.6+0.65 ms 
cpu, 7->7->6 MB, 8 MB goal, 32 P
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)

The ps shows 3674660 826844 (VSZ RSS)

BR,
Miha

On Monday, October 7, 2019 at 10:04:41 AM UTC+2, Jamil Djadala wrote:
>
> On Mon, 7 Oct 2019 00:54:32 -0700 (PDT) 
> miha.v...@gmail.com  wrote: 
>
> > Hi guys, 
> > 
> > We are having a very weird problem, where the process memory keeps 
> > rising, but both the pprof and valgrind report no memory leaks. But 
> > the RSS just keeps rising.The C part is in external so library 
> > 
> > To be more exact, the pprof when the process becomes idle reports a 
> > few megabytes used and the Valgrind reports 88bytes lost(this are 
> > global objects) But the RSS for example is ~1G and virtual 4G+ 
> > I've run also run the app with the GODEBUG=madvdontneed=1,gctrace=1 
> > but madvdontneed doesn't seem to help. 
> > 
> > If I describe the MO of communicating with the library's API (all 
> > memory is allocated on the library side). 
> > 
> > for { 
> >   * initialize the processor 
> >   * say to the library to create the blob of size x on it's side 
> >   * convert the returned pointer to a fake slice (code below) 
> >   * copy data to the fake slice 
> >   * instruct processor to do it's thing (e.g make 
> > calculations/compress the data,...) 
> >   * process the data on Go side (depending on the processor this 
> > either makes a copy of the data or creates a fake slice from it) 
> >   * destroy the processor (this frees the memory on the library's 
> > side) 
> >   * upload the data somewhere 
> > } 
> > 
> > func cVoidPtrToByteSlice(data unsafe.Pointer, size int, bytes 
> > *[]byte) { header := (*reflect.SliceHeader)(unsafe.Pointer(bytes)) 
> > header.Data = uintptr(data) 
> > header.Len = size 
> > header.Cap = size 
> > } 
> > 
> > BR, 
> > Miha 
> > 
>
> Hi, can you try to run your program with GOGC=10 environment ? 
>
> -- 
> Jamil Djadala 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/39e78a46-50e0-411d-b826-c4ac6497fda6%40googlegroups.com.

[go-nuts] cgo memory fragmetation?

[miha . vrhovnik]

Hi guys,

We are having a very weird problem, where the process memory keeps rising, 
but both the pprof and valgrind report no memory leaks. But the RSS just 
keeps rising.The C part is in external so library

To be more exact, the pprof when the process becomes idle reports a few 
megabytes used and the Valgrind reports 88bytes lost(this are global 
objects) But the RSS for example is ~1G and virtual 4G+
I've run also run the app with the GODEBUG=madvdontneed=1,gctrace=1 but 
madvdontneed doesn't seem to help.

If I describe the MO of communicating with the library's API (all memory is 
allocated on the library side). 

for {
  * initialize the processor 
  * say to the library to create the blob of size x on it's side
  * convert the returned pointer to a fake slice (code below)
  * copy data to the fake slice
  * instruct processor to do it's thing (e.g make calculations/compress the 
data,...)
  * process the data on Go side (depending on the processor this either 
makes a copy of the data or creates a fake slice from it)
  * destroy the processor (this frees the memory on the library's side)
  * upload the data somewhere
}

func cVoidPtrToByteSlice(data unsafe.Pointer, size int, bytes *[]byte) {
header := (*reflect.SliceHeader)(unsafe.Pointer(bytes))
header.Data = uintptr(data)
header.Len = size
header.Cap = size
}

BR,
Miha

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/de3655e0-62b4-4f39-ac52-82483b9009fc%40googlegroups.com.

[go-nuts] Re: go modules and vendor: redundant features?

[miha . vrhovnik]

Vendoring MUST stay. Athens is another piece of software that has to be set 
up and keep update. Venoring solves this porblem and ols the one when 
dependencies go missing.

On Saturday, November 17, 2018 at 5:33:55 AM UTC+1, Henry wrote:
>
> Hi, 
>
> It seems to me that go modules and vendor attempt to solve the same 
> problem. I wonder whether we should just choose one and scrap the other, or 
> find a way to consolidate them under a single unified feature. I am a bit 
> concerned with the direction Go is going. People keep adding stuffs into Go 
> and later find themselves unable to remove existing features due to the 
> backward compatibility promise. Go 1.11 is a different beast than Go 1.0, 
> and is significantly more complex. 
>
> I think Go2 is an opportunity to learn from Go1, and to start from a clean 
> slate by scraping and consolidating features. Go2 needs to be simpler than 
> Go1.11
>
> Henry
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.