Re: [go-nuts] cgo memory fragmetation?
It's not that. The system had 4G of memory initially as it is supposed to be enough for 18 processors (They should max out at around 250M but have the typical of about 100M) and it started swapping heavily. (I then raised the limits to 16G) so things can continue to run. BR, Miha On Monday, October 7, 2019 at 4:46:32 PM UTC+2, Ian Lance Taylor wrote: > > On Mon, Oct 7, 2019 at 12:54 AM > > wrote: > > > > We are having a very weird problem, where the process memory keeps > rising, but both the pprof and valgrind report no memory leaks. But the RSS > just keeps rising.The C part is in external so library > > In the absence of other memory pressure on the system as a whole, > increasing RSS just means that your program is using different > addresses for allocated memory. This isn't necessarily unexpected. > If overall memory usage is not increasing, there may not be a problem > here. > > Ian > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/3456b19b-e9a3-44a3-848a-23f7d90412cd%40googlegroups.com.
Re: [go-nuts] cgo memory fragmetation?
On Mon, Oct 7, 2019 at 12:54 AM wrote: > > We are having a very weird problem, where the process memory keeps rising, > but both the pprof and valgrind report no memory leaks. But the RSS just > keeps rising.The C part is in external so library In the absence of other memory pressure on the system as a whole, increasing RSS just means that your program is using different addresses for allocated memory. This isn't necessarily unexpected. If overall memory usage is not increasing, there may not be a problem here. Ian -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcX1ViOC%2BWucfQfCjj0g8DCU7_O%2Bj1ZYJXbqCZ%3DZFHEPeg%40mail.gmail.com.
Re: [go-nuts] cgo memory fragmetation?
This doesn't seem to help...
gc 589 @171.013s 0%: 0.012+0.69+0.17 ms clock, 0.41+0.079/2.9/1.6+5.5 ms
cpu, 6->6->5 MB, 7 MB goal, 32 P
scvg: 0 MB released
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: inuse: 7, idle: 54, sys: 61, released: 54, consumed: 7 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 52, consumed: 8 (MB)
gc 590 @171.018s 0%: 0.25+0.89+0.020 ms clock, 8.2+0.28/3.8/2.6+0.65 ms
cpu, 7->7->6 MB, 8 MB goal, 32 P
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
scvg: 0 MB released
scvg: inuse: 8, idle: 53, sys: 61, released: 53, consumed: 8 (MB)
The ps shows 3674660 826844 (VSZ RSS)
BR,
Miha
On Monday, October 7, 2019 at 10:04:41 AM UTC+2, Jamil Djadala wrote:
>
> On Mon, 7 Oct 2019 00:54:32 -0700 (PDT)
> miha.v...@gmail.com wrote:
>
> > Hi guys,
> >
> > We are having a very weird problem, where the process memory keeps
> > rising, but both the pprof and valgrind report no memory leaks. But
> > the RSS just keeps rising.The C part is in external so library
> >
> > To be more exact, the pprof when the process becomes idle reports a
> > few megabytes used and the Valgrind reports 88bytes lost(this are
> > global objects) But the RSS for example is ~1G and virtual 4G+
> > I've run also run the app with the GODEBUG=madvdontneed=1,gctrace=1
> > but madvdontneed doesn't seem to help.
> >
> > If I describe the MO of communicating with the library's API (all
> > memory is allocated on the library side).
> >
> > for {
> > * initialize the processor
> > * say to the library to create the blob of size x on it's side
> > * convert the returned pointer to a fake slice (code below)
> > * copy data to the fake slice
> > * instruct processor to do it's thing (e.g make
> > calculations/compress the data,...)
> > * process the data on Go side (depending on the processor this
> > either makes a copy of the data or creates a fake slice from it)
> > * destroy the processor (this frees the memory on the library's
> > side)
> > * upload the data somewhere
> > }
> >
> > func cVoidPtrToByteSlice(data unsafe.Pointer, size int, bytes
> > *[]byte) { header := (*reflect.SliceHeader)(unsafe.Pointer(bytes))
> > header.Data = uintptr(data)
> > header.Len = size
> > header.Cap = size
> > }
> >
> > BR,
> > Miha
> >
>
> Hi, can you try to run your program with GOGC=10 environment ?
>
> --
> Jamil Djadala
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/39e78a46-50e0-411d-b826-c4ac6497fda6%40googlegroups.com.
Re: [go-nuts] cgo memory fragmetation?
On Mon, 7 Oct 2019 00:54:32 -0700 (PDT)
miha.vrhov...@gmail.com wrote:
> Hi guys,
>
> We are having a very weird problem, where the process memory keeps
> rising, but both the pprof and valgrind report no memory leaks. But
> the RSS just keeps rising.The C part is in external so library
>
> To be more exact, the pprof when the process becomes idle reports a
> few megabytes used and the Valgrind reports 88bytes lost(this are
> global objects) But the RSS for example is ~1G and virtual 4G+
> I've run also run the app with the GODEBUG=madvdontneed=1,gctrace=1
> but madvdontneed doesn't seem to help.
>
> If I describe the MO of communicating with the library's API (all
> memory is allocated on the library side).
>
> for {
> * initialize the processor
> * say to the library to create the blob of size x on it's side
> * convert the returned pointer to a fake slice (code below)
> * copy data to the fake slice
> * instruct processor to do it's thing (e.g make
> calculations/compress the data,...)
> * process the data on Go side (depending on the processor this
> either makes a copy of the data or creates a fake slice from it)
> * destroy the processor (this frees the memory on the library's
> side)
> * upload the data somewhere
> }
>
> func cVoidPtrToByteSlice(data unsafe.Pointer, size int, bytes
> *[]byte) { header := (*reflect.SliceHeader)(unsafe.Pointer(bytes))
> header.Data = uintptr(data)
> header.Len = size
> header.Cap = size
> }
>
> BR,
> Miha
>
Hi, can you try to run your program with GOGC=10 environment ?
--
Jamil Djadala
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/20191007110422.325e6ae5%40bee.datamax.bg.
