Re: [go-nuts] Compiling Go plugins with playground/NACL/GAE-like isolation

[voidlogic]

For some use cases I am sure this kind of integration would be just fine. 
But if functions within plugins are called often enough the overhead 
difference between calling a function in a dynamically linked library (.so) 
and communicating via a UNIX socket will quickly become non-trivial. Even 
if you move from pipe based IPC to shared memory this is still a large gap 
as compared to direct calls.

For my use case, I may have resort to having users submit source code that 
my system then compiles (to .so) and vets (so I can black list packages, 
etc) if I can't find a better method of isolation.

On Wednesday, May 17, 2017 at 5:20:34 PM UTC-7, Aldrin Leal wrote:
>
> go-plugin wouldn't work?
>
> github.com/hashicorp/go-plugin
>
>
> --
> -- Aldrin Leal,  / 
> http://about.me/aldrinleal
>
> On Wed, May 17, 2017 at 7:05 PM, voidlogic  > wrote:
>
>> Hey Everyone,
>>
>> I'm working on a project to allow other teams within my company to submit 
>> plugins that are executing as optional event handlers within my 
>> application. We currently support Lua but with the addition of Go plugin 
>> support we would like to support Go as well (our app is written in Go 
>> itself).
>>
>> The new plugin package looks like it will work well: 
>> https://golang.org/pkg/plugin/
>>
>> The only caveat is it would be nice to have the plugins have similar 
>> filesystem, unsafe, etc isolation that the playground has. One idea would 
>> be to try to maintain a fork of Go that allows these GOOS=nacl mockups to 
>> be enabled for amd64 plugins- would that work or does a plugin share a 
>> runtime with the loading application? I don't think pure NACL with work 
>> without linking the NACL loader into the application...
>>
>> How does Google's app engines isolation work? A forked Go runtime? With 
>> the difference that no part of the app needs privileges (unlike here were 
>> the app doing the loading should be privileged) 
>>
>> If anyone has thoughts on loading plugins with some isolation, I would 
>> love to hear them. Thanks!
>>
>> -Tylor
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] beginner seeks peer review

[kbfastcat]

Thanks much, will review this, and incorporate changes...

On Wednesday, May 17, 2017 at 12:56:59 AM UTC-7, Lutz Horn wrote:
>
> Hi, 
>
> the more I work with Go the more I like it. Reading code written by 
> other people, even by beginners, is straight forward. 
>
> > https://github.com/kbfastcat/nrmetrics 
>
> go_vet and the misspell check found some minor issues: 
>
> https://goreportcard.com/report/github.com/kbfastcat/nrmetrics 
>
> My observations: 
>
> -- 
>
> In the lines 
>
> https://github.com/kbfastcat/nrmetrics/blob/master/nrpost/nrpost.go#L19 
>
> https://github.com/kbfastcat/nrmetrics/blob/master/nrpost/nrpost.go#L30 
>
>
> https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/config.go#L27 
>
>
> https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/config.go#L33 
>
> I would 
>
>return nil, err 
>
> instead of an unused first item. 
>
> -- 
>
> Why don't you export metric on 
>
>
> https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/getstats.go#L38
>  
>
> when you do export the other types? 
>
> -- 
>
> Could you write tests for parseInt64 and parseFloat64 that actually have 
> multiple lines as the info argument? The slice you get at 
>
>
> https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/getstats.go#L127
>  
>
> only contains one element in your tests. 
>
> -- 
>
> The names parseInt64 and parseFloat64 are a little misleading. I would 
> expect them to only accept on string argument that contains an int64 or 
> a float64 plus some to be ignored characters. But your functions look 
> for the marker given as the s argument and then get the number right of 
> the colon. 
>
> Maybe names that better tell the reader what the functions are doing 
> could be chosen. 
>
> -- 
>
> The help strings of the two flags at 
>
>
> https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/redis-agent.go#L21
>  
>
> just duplicate the flag names in a unhelpful way. You should describe 
> the meaning of the flags. 
>
> -- 
>
> I am sure others here can give you more helpful advice. But this what I, 
> a beginner like you, find from reading your code. 
>
> Lutz 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: nested template question

[Tieson Molly]

thanks Billie,  I will check it out

On Wednesday, May 17, 2017 at 1:56:46 PM UTC-4, Billie H Cleek wrote:
>
> Ty,
>
> The block template can satisfy your need: 
> https://godoc.org/text/template#example-Template--Block
>
> On Wednesday, May 17, 2017 at 4:55:28 AM UTC-7, Tieson Molly wrote:
>>
>> Is there a way to defined a template name as a fallback if one does not 
>> exist?
>>
>> Say you have a nested template in your code
>>
>> {{template "javascript" .}}
>>
>>
>> if the template name "javascript" is not defined because you did not 
>> include that file during the parsing, is there a way to 
>>
>> generate that name dynamically?
>>
>> Best regards,
>>
>> Ty
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Compiling Go plugins with playground/NACL/GAE-like isolation

[Aldrin Leal]

go-plugin wouldn't work?

github.com/hashicorp/go-plugin


--
-- Aldrin Leal,  / http://about.me/aldrinleal

On Wed, May 17, 2017 at 7:05 PM, voidlogic  wrote:

> Hey Everyone,
>
> I'm working on a project to allow other teams within my company to submit
> plugins that are executing as optional event handlers within my
> application. We currently support Lua but with the addition of Go plugin
> support we would like to support Go as well (our app is written in Go
> itself).
>
> The new plugin package looks like it will work well:
> https://golang.org/pkg/plugin/
>
> The only caveat is it would be nice to have the plugins have similar
> filesystem, unsafe, etc isolation that the playground has. One idea would
> be to try to maintain a fork of Go that allows these GOOS=nacl mockups to
> be enabled for amd64 plugins- would that work or does a plugin share a
> runtime with the loading application? I don't think pure NACL with work
> without linking the NACL loader into the application...
>
> How does Google's app engines isolation work? A forked Go runtime? With
> the difference that no part of the app needs privileges (unlike here were
> the app doing the loading should be privileged)
>
> If anyone has thoughts on loading plugins with some isolation, I would
> love to hear them. Thanks!
>
> -Tylor
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Compiling Go plugins with playground/NACL/GAE-like isolation

[voidlogic]

Hey Everyone,

I'm working on a project to allow other teams within my company to submit 
plugins that are executing as optional event handlers within my 
application. We currently support Lua but with the addition of Go plugin 
support we would like to support Go as well (our app is written in Go 
itself).

The new plugin package looks like it will work well: 
https://golang.org/pkg/plugin/

The only caveat is it would be nice to have the plugins have similar 
filesystem, unsafe, etc isolation that the playground has. One idea would 
be to try to maintain a fork of Go that allows these GOOS=nacl mockups to 
be enabled for amd64 plugins- would that work or does a plugin share a 
runtime with the loading application? I don't think pure NACL with work 
without linking the NACL loader into the application...

How does Google's app engines isolation work? A forked Go runtime? With the 
difference that no part of the app needs privileges (unlike here were the 
app doing the loading should be privileged) 

If anyone has thoughts on loading plugins with some isolation, I would love 
to hear them. Thanks!

-Tylor

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] x/term: go-keycodes

[Nigel Tao]

On Sun, May 14, 2017 at 6:02 PM, anatoly techtonik  wrote:
> On Fri, May 12, 2017 at 7:54 AM, Nigel Tao  wrote:
>> On Fri, May 12, 2017 at 3:11 AM, anatoly techtonik  
>> wrote:
>>> I am new to Go and I need a cross-platform value for keyboard key codes to
>>> make sure I can react to user key presses regardless of operating system
>>> layer.
>>
>> FYI, there's already the key.Code type defined in
>> https://github.com/golang/mobile/blob/master/event/key/key.go
>
> 1. So does that mean it will conflict if both are imported?
>
> Right now https://github.com/yakshaveinc/go-keycodes/blob/master/keycodes.go
> uses KeyF11 while mobile lib uses CodeF11. For me Code is less clear
> than Key, because from user point of view they press keys, not
> symbols/characters of codes.

It will not conflict.

As for names, note that users of that package will refer to it by its
qualified name: key.CodeF11, not a bare CodeF11.

For example, with the net/http package in the standard library, the
types are called Client and Server, which users of that package will
see as http.Client and http.Server. It's not "type HTTPClient" which
would become "http.HTTPClient". Even so, http.Client and rpc.Client
(from the net/rpc package) will not conflict.


> 2. Does it provide reverse mapping, so that if user press F11, go
> program could write "KeyF11" to console?

Yes, just call the String method, or just pass it to e.g. fmt.Printf("%v", etc).


> 3. How should keycodes package be renames and placed so that I could
> be reused in mobile?

I'm not sure that it needs to be renamed, or that you need a new
package. It might be fine for golang.org/x/term code to import
golang.org/x/mobile packages.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Golang map is getting corrupted with Memory mapped files

[Tamás Gulácsi]

I don't really understand what are you doing.
You serialize with gob.Encode, but does NOT deserialize, just use the encoded 
bytes as the backing memory of an unitialized map. This won't work.

Either deserialize with gob.Decode, or use your own custom data structure, 
which can be backed by memory mapped disk - for example a B+-tree.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: nested template question

[bhcleek]

Ty,

The block template can satisfy your need: 
https://godoc.org/text/template#example-Template--Block

On Wednesday, May 17, 2017 at 4:55:28 AM UTC-7, Tieson Molly wrote:
>
> Is there a way to defined a template name as a fallback if one does not 
> exist?
>
> Say you have a nested template in your code
>
> {{template "javascript" .}}
>
>
> if the template name "javascript" is not defined because you did not 
> include that file during the parsing, is there a way to 
>
> generate that name dynamically?
>
> Best regards,
>
> Ty
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Golang map is getting corrupted with Memory mapped files

[chaitanya reddy]

Thanks Jan for the reply.

So is there any way I can make this program to work?

On Wednesday, May 17, 2017 at 6:15:12 AM UTC-7, Jan Mercl wrote:
>
> On Wed, May 17, 2017 at 2:49 PM chaitanya reddy  > wrote:
>
> Maps must be initialized before mutating them, which the program does not 
> do. However, when initializing a map it's not possible to chose where the 
> map instance will be allocated neither where the map will acquire memory 
> when needed.
> -- 
>
> -j
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Realizing SSD random read IOPS

[Ian Lance Taylor]

On Wed, May 17, 2017 at 12:29 AM, Manish Rai Jain  wrote:
>
>> libaio sounds good on paper, but at least on GNU/Linux it's all in user
>> space.
>
> I see. That makes sense. Reading a bit more, Linux native I/O sounds like it
> does exactly what we expect, i.e. save OS threads, and push this to kernel:
> http://man7.org/linux/man-pages/man2/io_submit.2.html
> But, I suppose this can't be part of Go, because it's not portable. Is my
> understanding correct?

We could use io_submit and friends on GNU/Linux.  We want to provide a
consistent API to Go code, but the internal code can be different on
different operating systems.  For example the implementations on
WIndows and Unix systems are of course quite different.

It's not obvious to me that io_submit would be a win for normal
programs, but if anybody wants to try it out and see that would be
great.


> Also, any explanations about why GOMAXPROCS causes throughput to increase,
> if new OS threads are being spawned by blocked goroutines anyway? I thought
> I understood it before but now I don't.

My guess is that it's the timing.  The current runtime doesn't spawn a
new OS thread until an existing thread has been blocked in a syscall
for 20us or more.  Having more threads ready to go avoids that delay.

I agree with Dave that looking at the execution tracer is likely to help.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Why no return statement for *timerCtx in func parentCancelCtx(context.go)

[Ian Davis]

On Wed, 17 May 2017, at 01:40 PM, Guohua Ouyang wrote:
> I was confusing when I read the lines #L279-#L280
> https://github.com/golang/go/blob/master/src/context/context.go#L279

it will loop with the new value of parent


> --
>  You received this message because you are subscribed to the Google
>  Groups "golang-nuts" group.>  To unsubscribe from this group and stop 
> receiving emails from it,
>  send an email to golang-nuts+unsubscr...@googlegroups.com.>  For more 
> options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: http: TLS handshake error from :: EOF

[James Hartig]

I've posted a more detailed explanation of this to the net-dev mailing 
list: https://groups.google.com/a/chromium.org/forum/#!topic/net-dev/IsYHz3hNlNU

What we're seeing is clients making 2 back-to-back TCP connections to port 
443 but only ever sending Client Hello (and continuing TLS handshake) to 
the first opened socket. The second socket is closed by the client after 
~10 seconds without anything over the socket after the server acknowledging 
the SYN.

It seems like this isn't a Golang issue but still seems like Go shouldn't 
be logging that the handshake failed when the handshake was never started.

Should I file a bug to remove the log in that case?

On Wednesday, May 3, 2017 at 9:45:00 AM UTC-4, James Hartig wrote:
>
> We're also seeing 100k of these errors per day per server. The majority of 
> the time its just a result of:
> client SYN
> server SYN-ACK
> client FIN
> server FIN-ACK
>
> or
>  
> client SYN
> server SYN-ACK
> Retransmission
> Retransmission
> RST
> ...
>
> The error if the version is unsupported is:
> http: TLS handshake error from ip:port : tls: client offered an 
> unsupported, maximum protocol version of 300
>
> Can we ignore this error if there was no handshake ever initiated?
>
> Thanks!
>
> On Sunday, January 1, 2017 at 1:10:54 PM UTC-5, paperboardinc wrote:
>>
>> No worries J at least your trying to help. I think the minimum must be 
>> TLS 1.0 because I know I didn’t set the tls server config value, however, I 
>> do know exactly where to set it. I asked this question in the Go repo on 
>> Github, a core golang developer told me to use the forums for asking 
>> questions, I suppose he knows the answer since he wasn’t really surprised 
>> about the behavior. We have about 2 million people using the site on a 
>> monthly basis, so it really floods our logs. @bratfpatrick please help!
>>
>>  
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Golang map is getting corrupted with Memory mapped files

[Jan Mercl]

On Wed, May 17, 2017 at 2:49 PM chaitanya reddy 
wrote:

Maps must be initialized before mutating them, which the program does not
do. However, when initializing a map it's not possible to chose where the
map instance will be allocated neither where the map will acquire memory
when needed.
-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: [go/loader] write a conf to load faster a program ?

[mhhcbon]

Do you think i could take advantage of importer to serialize stdlib, 
then unserialize it rather than process it every times ?

Are those types linked to some sort of unserializable resource that would 
prevent that to happen ?

On Tuesday, May 16, 2017 at 1:43:44 AM UTC+2, mhh...@gmail.com wrote:
>
> Hi,
>
> I wrote this func to load a program,
> its faster but it has drawbacks about error managements.
> It might hide some messages.
>
> the first time i ran this function,
> without a special typechecker error handler,
> I felt overwhelmed by the amount of errors generated.
>
> in this func s is an import path, 
> and only that path should be loaded (so far).
> The import is very superficial.
>
> Is there a better/correct way to write a conf 
> that achieves similar speed goal 
> but with good error triage ?
>
> Should i use another api ?
>
> I m mostly interested into 
> inspecting the definitions,
> create new definitions,
> update some definitions in place,
> to be able to process the source even if it contains errors,
> unless they are critical.
>
> Thanks
>
>
> // GetFastProgramLoader returns a fast program loader
> func GetFastProgramLoader(s string) loader.Config {
> var conf loader.Config
> conf.ParserMode = parser.ParseComments
> conf.TypeChecker.IgnoreFuncBodies = true
> conf.TypeChecker.DisableUnusedImportCheck = true
> conf.TypeChecker.Error = func(err error) {
> if !err.(*types.Error).Soft {
> panic(err)
> }
> if strings.Index(err.Error(), "could not import") == -1 ||
> strings.Index(err.Error(), "undeclared name:") == -1 {
> return
> }
> log.Println(err)
> }
>
> // this really matters otherise its a pain to generate a partial 
> program.
> conf.AllowErrors = true
> originalPkgFinder := (*build.Context).Import
> conf.FindPackage = func(ctxt *build.Context, fromDir, importPath 
> string, mode build.ImportMode) (*build.Package, error) {
> if fromDir == s {
> return originalPkgFinder(ctxt, fromDir, importPath, mode)
> }
> return nil, fmt.Errorf("skipped %v %v", fromDir, importPath)
> }
> return conf
> }
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Realizing SSD random read IOPS

['David Klempner' via golang-nuts]

On May 16, 2017 22:03, "Ian Lance Taylor"  wrote:

On Tue, May 16, 2017 at 9:26 PM, Manish Rai Jain 
wrote:
>> The runtime will spawn a new thread to replace the one that is blocked.
>
> Realized that after writing my last mail. And that actually explains some
of
> the other crashes we saw, about "too many threads", if we run tens of
> thousands of goroutines to do these reads, one goroutine per read.
>
> It is obviously lot more expensive to spawn a new OS thread. It seems like
> this exact same problem was already solved for network via netpoller
> (https://morsmachine.dk/netpoller). Blocking OS threads for disk reads
made
> sense for HDDs, which could only do 200 IOPS; for SSDs we'd need a
solution
> based on async I/O.

Note that in the upcoming Go 1.9 release we now use the netpoller for
the os package as well.  However, it's not as effective as one would
hope, because on GNU/Linux you can't use epoll for disk files.


There's a not very well documented API to make AIO completions kick an
eventfd.

It
mainly helps with pipes.

Ian

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


smime.p7s
Description: S/MIME Cryptographic Signature

[go-nuts] Casbin: an authorization library that supports access control models like ACL, RBAC, ABAC

[Yang Luo]

Hi there,

casbin (https://github.com/casbin/casbin)  
is a powerful and efficient open-source access control library for Golang 
projects. It provides support for enforcing authorization based on various 
models. By far, the access control models supported by casbin are:

   1. ACL (Access Control List) 
   
   2. ACL with superuser 
   3. ACL without users: especially useful for systems that don't have 
   authentication or user log-ins.
   4. ACL without resources: some scenarios may target for a type of 
   resources instead of an individual resource by using permissions like 
   write-article, read-log. It doesn't control the access to a specific 
   article or log.
   5. RBAC (Role-Based Access Control) 
   
   6. RBAC with resource roles: both users and resources can have roles (or 
   groups) at the same time.
   7. ABAC (Attribute-Based Access Control) 
   
   8. RESTful 
   

Features

What casbin does:

   1. enforce the policy in the classic {subject, object, action} form or a 
   customized form as you defined.
   2. handle the storage of the access control model and its policy.
   3. manage the role-user mappings and role-role mappings (aka role 
   hierarchy in RBAC).
   4. support built-in superuser like root or administrator. A superuser 
   can do anything without explict permissions.
   5. multiple built-in operators to support the rule matching. For 
   example, keyMatch can map a resource key /foo/bar to the pattern /foo*.

What casbin does NOT do:

   1. authentication (aka verify username and password when a user logs in)
   2. manage the list of users or roles. I believe it's more convenient for 
   the project itself to manage these entities. Users usually have their 
   passwords, and casbin is not designed as a password container. However, 
   casbin stores the user-role mapping for the RBAC scenario.

Installation

go get github.com/casbin/casbin


Please advise, raise your issues and star, thanks!


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Golang map is getting corrupted with Memory mapped files

[chaitanya reddy]

Hi

I was trying to write persistent map using mmap. But my map is getting 
corrupted by mmaps. I think I am using them wrongly. I need your advise.

Thanks in advance.

package main


import (
"bytes"
"encoding/gob"
"fmt"
"os"
"syscall"
"unsafe"
)


func GetBytes(key interface{}) ([]byte, error) {
var buf bytes.Buffer
enc := gob.NewEncoder()
err := enc.Encode(key)
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}


func main() {
const n = 1e3
t := int(unsafe.Sizeof(0)) * n


map_file, err := os.Create("/tmp/test1.dat")
if err != nil { 

  [0/89]
fmt.Println(err)
os.Exit(1)
}
_, err = map_file.Seek(int64(t-1), 0)
if err != nil {
fmt.Println(err)
os.Exit(1)
}


dummyMapBytes, err := GetBytes(map[string]string{"dummy": "dummy"})
if err != nil {
fmt.Println(err)
os.Exit(1)
}
_, err = map_file.Write(dummyMapBytes)
if err != nil {
fmt.Println(err)
os.Exit(1)
}


mmap, err := syscall.Mmap(int(map_file.Fd()), 0, int(t), syscall.
PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
kv := (*map[string]string)(unsafe.Pointer())
fmt.Println(*kv)


for i := 0; i < n; i++ {
(*kv)[string(i)] = string(i * i)
}


fmt.Println(*kv)


for k, v := range *kv {
fmt.Printf("%s - %s", k, v)
}


err = syscall.Munmap(mmap)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
err = map_file.Close()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
}

Output:
map[]
map[D:ሐ ˦: l:ⶐ ǵ: ̕: Ξ: |:㰐 ý:降 ˤ: [:⁙ ¶:腤 ʼn: ʳ: : ·:苑 IJ: ˗: û: ć: ʊ: ͂: 
^:⊄ ˀ: ;: :䘤 ŕ: ʨ: ͜: Ǎ: dz: Ʉ: ʠ: ω: h:⩀ ə: Ȑ: ˽: :喐 ª:烤 ɝ: Ǣ: ɿ: i:⬑ ĩ: ƾ: 
ǟ: ˖: ̅: G:Ꮁ p: þ:ﰄ ś:홙 ť: Ű:ᄀ Ǿ: ̗: ͻ: 5: :呩 ķ: Η: ϒ: ŧ: ͆: ͝:
: Ũ: ǀ: ß:쉁 Ŕ: ̐: Ƥ: Dž: ʗ: Ύ: ĝ: ͘: a:Ⓛ ȇ: À:退 ð: ȹ: ɋ: ˞: ͙: 
ͩ::@ :Ɛ »:袙 Ž:㜉 ƀ:䀀 Ɩ:菤 ț: ζ: I:ᓑ Ɵ:ꃁ ʲ: ͳ: :岤 ě: ǔ: ̹: ψ: ϙ: :䡀 Ʊ: ǭ: 
Ȍ: έ: Ł: Ʀ: ʅ: ͖: ǎ: ǧ: ɲ: ˭: Ά: Ē: Ů:ୄ ̽: ͋: œ: Ș: ̌: ƞ:鶄 Ơ:ꐀ ˺: ͷ: w:㝑 
ˇ: ̄: ̫: ͚: ʖ: ͡: ΄: 3: ɩ: ʄ: ͤ: ¥:橙 ¾:贄 ô: ̠: Ϝ: ï:� ͏: :埤 ÿ:︁ Ĉ: ɧ: ˁ: 
Ρ: Τ: ϊ: DŽ: Ǐ: Ǽ: ʻ: 6: ¯:瞡 Ă:Є ʟ: ű:Ꮱ ο: :婀 ğ: ȓ: ɘ: ̴: ͔: ͭ: :̐ ŭ:ࡩ ƣ: 
Ȯ: ´:纐 Ǧ: ȉ: ˴: Ä:阐 ʈ: ĵ: ł: Ń: ɞ: ɦ: ͍: ʣ
: ˬ: í:� î:� nj: ɕ: ː: Ś: :䶩 ©:澑 Ƃ:䘄 ̈: :ɱ :嚹 Î:ꗄ Ð:꤀ ü: Ȫ: ˘: ş: ɺ: ˔: ̏: 
Έ: ϡ:ு <:ฐ ȝ: ʢ: ͟: ϗ: b:▄ Ƚ: ̖: ̱: z:㨤 ê:헤 Ĥ: Ȓ: ʮ: ę: ɳ: ͊: :ɀ ë: Ę: ƨ: 
ɖ: ͦ: Ί: ̃: ͒: Ͱ: :䐐 «:爹 ǿ: Q:ᦡ X:Ṁ Ə:淡 ˏ: ͓: 1:ॡ º:蜤 Æ:餤 Ê:齤 ˛: ̥: W:ᶑ :憄 
Ǝ:櫄 ƹ: Ȩ: ̨: ͇: ͧ: Ċ: ɟ: ʞ: ˹: ʹ: ʸ: ˥: : Ϥ:⌐ ƫ: Ǘ: &:֤ *:ۤ ŋ: ɢ: :Ā Ĵ: 
̿: ͛: ͨ: : ũ: ƺ: Ɍ: ɛ: :䥑 Õ:넹 Ǜ: ̸: ͮ: j: Å:鞙 Υ: λ: χ: Ϧ:㊤ Ą:ࠐ ǰ: Ȱ: ɤ: 
½:讉 ã:쥉 ơ: ȋ: é:퐑 Ņ: Ş: ſ:㴁 ǣ: ȵ: ̘: ͑: ˱: ̰: :信 Μ: ϥ:⫙ ą:ਙ ɮ: ̢: -:ߩ 
9:ಱ Ψ: ŵ:ό ȯ: ɉ: ύ: ƍ:枩 Ȟ: ɰ: Ě: Ƭ: ̀: :巙 ¤:椐 Ř: Ɣ:綐 ʯ: ˿: ̲: Ζ: ϝ: :卄 
į: Ȧ: ˊ: ͵: Ϊ: ώ: P:ᤀ Ȳ: ő: ƕ:肹 Ǹ: ȅ:  :Ѐ ˒: ñ: ļ: ƒ:睄 ǡ: Ŗ: Ϛ: Þ:삄 å:쳙 
Ħ: B:ᄄ µ:翹 ǒ: ̳: ¸:葀 ǝ:
 ɼ: ˍ: ή: #:Ӊ ģ: Ǖ: ȣ: ̬: Ϙ: ×:뒑 ɯ: ʔ: ̂: :ġ !:с :仄 ǐ: ǯ: ȗ: ˋ: :䜱 ɣ: ͈: ȁ: 
Α: :Ä r:㋄ :刡 ů:ม ʤ: ί: Ï:ꝡ ɴ: ʌ: ̜: c:♉ :䀀 ă: Ȇ: ɠ: ̍: ̙: Ñ: Ț: ̀: Π: ϖ: 
Ĭ: Dz: ʪ: ˠ: Λ: ε:  :Q :ρ ė: Ɲ:驉 Ǭ: ̵: ĉ: >:༄ £:柉 :á Ȃ: ɑ: ̻: :1 °:礀 Ç:骱 ǘ: 
ǫ: ȏ: Ŋ: ʱ: σ: υ: K:ᗹ q: ˚: :  :ń e:⟙ Ū: Ȕ: ̟: ͣ: g:⥱ Í:ꐩ ö: ņ: ̪: ʴ: 
˶: ̔: ä:쬐 ĺ: Ǵ: ù: ͺ: ·: Ώ: Ȝ: ɐ: ̯: Ō: Ǟ: ʘ: ̡: : μ: Ȋ: ɓ: ı: ˙: ξ:Ą ::ത 
L:ᚐ ř:탱 Š: ţ: ͞: Ġ: Ƙ:詀 ʶ: ʹ: M:ᜩ Ƌ:
慹 Ƕ: ˨: ɵ: ʑ: ˯: Ϟ: à:쐀 Ų:ᛄ ƭ: ɚ: :͉ Ö:다 ̋: :ũ E:ኙ Ô:꾐 ƈ:塀 ɫ: α: ²:範 Ȅ: 
ȿ: ̊:
:d ˟: ˮ: 7: Ľ: š: ș: ˷: ̾: ¼:訐 Ƨ: ƻ: ͉: Ɓ:䌁 ʍ: ̈́: +:ܹ Ɯ:霐 {:㬙  :搀 ˕: ̶: 
ͅ: : Ø:뙀 İ: ȟ: ɶ: ͽ: Γ: Ĺ: :$ :ʤ ā: ̤: ǻ: ͼ: β: ʺ: `:␀ ĥ: ˎ: ͗: H:ᑀ Ģ: ):ڑ 
Ň: ʜ: Ŀ: Ť: Β: A:ႁ ư: Â:錄 ʥ: : ȍ: Ȣ: ʇ: Ȗ: ò: ˄: ˌ: : γ: ƌ:撐 Κ: ¡:敁 Ź:⬱ 
ƚ:邤 Ì:ꊐ ǁ: Ǻ: ˜: ~:㸄 Ĝ: dž: Ȁ: ˳: ϔ: 4:ઐ É:鷑 Ü:봐 ȕ: ˃: .: _:⍁ Ƽ: Ǒ: ʀ: O:ᡡ 
:儀 Ć: ŝ: Ŭ:֐ lj:

   :y Y:ự ċ: Ǩ: ɗ: ʚ: ˩: ͢: φ: ¹:薱 ɍ: 
v:㙤 }:㴉 :䩤 ˂: :ȑ 2:ৄ F:ጤ Ƞ: δ: Ā: ɻ: ,:ސ Ƣ: ɷ: Ν: J:ᕤ Ɇ: ˈ: : :Ǥ ¦:殤 ƥ: 
Ǯ: Ɔ:判 ƶ: ʾ: ̛: π: Ğ: ͪ: Ɛ:焀 ɾ: ̼: Đ:℀ Ɨ:蜑 ȳ: Ȼ: ɔ: ˡ: ̮: : â:임 ĭ: ɥ: 
˝: ̭: R: ʬ: k:ⲹ ō: ȑ: Ϣ:ᎄ ˙ Œ: ͌: ͐: N:ោ Ĕ: ƽ: ˣ: ΐ: Ó:귩 ħ: Ŧ: Ȉ: C:ᆉ Ƅ:䰐 
ˢ: Ď: Χ: U: DZ: ̣: Σ: ŗ: ź:⸤ ȡ: ɏ: ʭ: ʷ: ̞: ͫ: η: Û:뭙 ū:ʹ ȶ: ς: ¬:玐 Ɖ:孑 
Ʋ: ʰ: Ÿ:⡀ ̇: ̉: ̦: Ͷ: Θ: ι: ƛ:鏙 Ʈ: ɀ: Ω: :䄁 NJ: ǜ: ˧: ̓: Ë:ꃹ ij: ʉ: :䲐 Ī: 
Ŝ: ɂ: ʏ: ̺: ͥ: ͬ: : (:ـ T:ᮐ ȩ: Ⱦ: Ɏ: ά: : ':ױ ǚ: :΄ :䈄 ±:穡 Ò:걄 ÷: ƪ: =: 
u:㕹 ó: Ŏ: ȸ: ν: \:ℐ f:⢤ È:鱀 ǂ: ȫ: θ: ¢:暄 ˰: τ: ď: ʦ: Ƴ: ǹ: Ƶ: Ǫ: Ι: ϓ: Ǥ: 
˓: Ε: ì:� ŷ:║ Ż:ㄙ Ɠ:穩 ȴ: ʒ: ʩ: ʫ: ˫: ų:ᦩ ˑ: ˪: ́: ¨:湀 Ǚ: ̧: $:Ԑ n:⽄ ɨ: 
Č: ʕ: ˉ: ̑: ͠: ρ: ǃ: ;: 

[go-nuts] Why no return statement for *timerCtx in func parentCancelCtx(context.go)

[Guohua Ouyang]

I was confusing when I read the lines #L279-#L280
https://github.com/golang/go/blob/master/src/context/context.go#L279

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] nested template question

[Tieson Molly]

Is there a way to defined a template name as a fallback if one does not 
exist?

Say you have a nested template in your code

{{template "javascript" .}}


if the template name "javascript" is not defined because you did not 
include that file during the parsing, is there a way to 

generate that name dynamically?

Best regards,

Ty

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] The zero value of a slice is nil

[mhhcbon]

hi

At
https://blog.golang.org/go-slices-usage-and-internals#TOC_3.

It says,

> The zero value of a slice is nil. The len and cap functions will both 
return 0 for a nil slice. 


Then looked at https://youtu.be/ynoY2xz-F8s?t=10m20s


I found the second explanation much better to explain

https://play.golang.org/p/bSTmNrAtjA

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Stop HTTP Server with Context cancel

[mhhcbon]

> Is it OK to use context cancellation for stopping long running functions ?

yes, i d say so. 

About contexts, 
https://www.youtube.com/watch?v=LSzR0VEraWw
https://www.youtube.com/watch?v=8M90t0KvEDY

>From scratch, 
with some mocks to test

package main

import (
"context"
"log"
"time"
)

func main() {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.
Millisecond)
defer cancel()
var err error
listenSig := make(chan error)
go func() { listenSig <- <-listenAndServe() }()//IRL, not sure if that 
will return if server has shutdown.
// this is unclear at 
https://beta.golang.org/src/net/http/server.go?s=79791:79837#L2615
// i d say its all about timeouts.
select {
case err = <-listenSig:
log.Println("server soft end")

case <-ctx.Done():
log.Println("context ended")

log.Println("server hard stop")
if err = shutdown( /*context.Background()*/ ); err == nil {
err = ctx.Err()
}
log.Println("server ended")
}
log.Println(err)

}

func listenAndServe() chan error {
e := make(chan error)
go func() {
<-time.After(time.Second)
e <- nil
}()
return e
}

func shutdown() error {
<-time.After(2 * time.Second)
return nil
}


that being said, server.Shutdown seems blocking,
https://beta.golang.org/pkg/net/http/#Server.Shutdown
> Shutdown ... waiting indefinitely for connections to return to idle and 
then shut down.
https://beta.golang.org/src/net/http/server.go?s=74304:74358#L2440

If so, d go func that one and put its err to a sink logger.
It will hang there until all clients are out, so the server must be 
configured with timeouts.
Or use a context with deadline ?
I m not sure what s the best, maybe both are needed ?
It looks likes this ctx parameter is sugar only.

Apart, does server.Shutdown allows a straight restart ?
+/- like if i d use socket_reuse port/addr?

> This code ensures that there is not leaking goroutine.

How do you know ? 

I mean for race detection there is a tool, 

for routine leak see those findings,
https://github.com/fortytw2/leaktest
https://godoc.org/github.com/anacrolix/missinggo#GoroutineLeakCheck
https://github.com/google/gops
https://medium.com/golangspec/goroutine-leak-400063aef468

https://github.com/golang/go/issues/12989
https://github.com/golang/go/issues/5308
https://github.com/golang/go/issues/6705

It has been set to
"Adding Release=None to all Priority=Someday bugs."


On Tuesday, April 4, 2017 at 8:02:16 PM UTC+2, Pierre Durand wrote:
>
> Hello
>
> I wrote a small helper to stop an HTTP Server when a Context is canceled.
> https://play.golang.org/p/Gl8APynVdh
>
> What do you think ?
> Is it OK to use context cancellation for stopping long running functions ?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] beginner seeks peer review

[Lutz Horn]

Hi,

the more I work with Go the more I like it. Reading code written by 
other people, even by beginners, is straight forward.



https://github.com/kbfastcat/nrmetrics


go_vet and the misspell check found some minor issues:

https://goreportcard.com/report/github.com/kbfastcat/nrmetrics

My observations:

--

In the lines

https://github.com/kbfastcat/nrmetrics/blob/master/nrpost/nrpost.go#L19

https://github.com/kbfastcat/nrmetrics/blob/master/nrpost/nrpost.go#L30

https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/config.go#L27

https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/config.go#L33

I would

  return nil, err

instead of an unused first item.

--

Why don't you export metric on

https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/getstats.go#L38

when you do export the other types?

--

Could you write tests for parseInt64 and parseFloat64 that actually have 
multiple lines as the info argument? The slice you get at


https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/getstats.go#L127

only contains one element in your tests.

--

The names parseInt64 and parseFloat64 are a little misleading. I would 
expect them to only accept on string argument that contains an int64 or 
a float64 plus some to be ignored characters. But your functions look 
for the marker given as the s argument and then get the number right of 
the colon.


Maybe names that better tell the reader what the functions are doing 
could be chosen.


--

The help strings of the two flags at

https://github.com/kbfastcat/nrmetrics/blob/master/redis-agent/redis-agent.go#L21

just duplicate the flag names in a unhelpful way. You should describe 
the meaning of the flags.


--

I am sure others here can give you more helpful advice. But this what I, 
a beginner like you, find from reading your code.


Lutz


--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] adding context.Context to new code

[Nazri Ramliy]

I'm about to embark on doing the same, and my first pit stop is at
using the callgraph to generate the caller-callee data but stumbled
upon this error while trying out the example given by the "callgraph"
command when run without arguments:

$ callgraph -format '{{.Caller.Pkg.Object.Path}} ->
{{.Callee.Pkg.Object.Path}}' /usr/local/go/src/net/http/triv.go | sort
| uniq
callgraph: template: -format:1:9: executing "-format" at
<.Caller.Pkg.Object.P...>: can't evaluate field Object in type
*ssa.Package

any idea what I'm doing wrong?

nazri

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Realizing SSD random read IOPS

[Dave Cheney]

Can you post the svg versions of those profiles?

Also, I recommend the execution trace profiler for this job, it'll show you
a lot of detail about how the runtime is interacting with your program.

On Wed, 17 May 2017, 17:29 Manish Rai Jain  wrote:

> > libaio sounds good on paper, but at least on GNU/Linux it's all in user
> space.
>
> I see. That makes sense. Reading a bit more, Linux native I/O sounds like
> it does exactly what we expect, i.e. save OS threads, and push this to
> kernel: http://man7.org/linux/man-pages/man2/io_submit.2.html
> But, I suppose this can't be part of Go, because it's not portable. Is my
> understanding correct?
>
> Also, any explanations about why GOMAXPROCS causes throughput to increase,
> if new OS threads are being spawned by blocked goroutines anyway? I thought
> I understood it before but now I don't.
>
> Dave, profiler doesn't show any issues with the code itself. It's just
> blocked waiting on syscalls.
>
> $ go tool pprof randread /tmp/profile398062565/cpu.pprof
>  ~/go/src/github.com/dgraph-io/badger-bench/randread
> Entering interactive mode (type "help" for commands)
> (pprof) top
> 19.48s of 19.76s total (98.58%)
> Dropped 27 nodes (cum <= 0.10s)
>   flat  flat%   sum%cum   cum%
> 19.34s 97.87% 97.87% 19.52s 98.79%  syscall.Syscall6
>  0.07s  0.35% 98.23%  0.11s  0.56%  runtime.exitsyscall
>  0.03s  0.15% 98.38% 19.56s 98.99%  os.(*File).ReadAt
>  0.02s   0.1% 98.48%  0.10s  0.51%  math/rand.(*Rand).Intn
>  0.01s 0.051% 98.53% 19.70s 99.70%  main.Conc2.func1
>  0.01s 0.051% 98.58% 19.53s 98.84%  syscall.Pread
>  0 0% 98.58%  0.13s  0.66%  main.getIndices
>  0 0% 98.58% 19.53s 98.84%  os.(*File).pread
>  0 0% 98.58% 19.70s 99.70%  runtime.goexit
> (pprof)
>
>
> $ go tool pprof randread /tmp/profile192709852/block.pprof
>  ~/go/src/github.com/dgraph-io/badger-bench/randread
> Entering interactive mode (type "help" for commands)
> (pprof) top
> 58.48s of 58.48s total (  100%)
> Dropped 8 nodes (cum <= 0.29s)
>   flat  flat%   sum%cum   cum%
> 58.48s   100%   100% 58.48s   100%  sync.(*WaitGroup).Wait
>  0 0%   100% 58.48s   100%  main.Conc2
>  0 0%   100% 58.48s   100%  main.main
>  0 0%   100% 58.48s   100%  runtime.goexit
>  0 0%   100% 58.48s   100%  runtime.main
> (pprof)
>
>
> On Wed, May 17, 2017 at 3:25 PM, Dave Cheney  wrote:
>
>> Rather than guessing what is going on, I think it's time to break out the
>> profiling tools Manish.
>>
>> On Wed, 17 May 2017, 15:23 David Klempner  wrote:
>>
>>>
>>> On May 16, 2017 22:03, "Ian Lance Taylor"  wrote:
>>>
>>> On Tue, May 16, 2017 at 9:26 PM, Manish Rai Jain 
>>> wrote:
>>> >> The runtime will spawn a new thread to replace the one that is
>>> blocked.
>>> >
>>> > Realized that after writing my last mail. And that actually explains
>>> some of
>>> > the other crashes we saw, about "too many threads", if we run tens of
>>> > thousands of goroutines to do these reads, one goroutine per read.
>>> >
>>> > It is obviously lot more expensive to spawn a new OS thread. It seems
>>> like
>>> > this exact same problem was already solved for network via netpoller
>>> > (https://morsmachine.dk/netpoller). Blocking OS threads for disk
>>> reads made
>>> > sense for HDDs, which could only do 200 IOPS; for SSDs we'd need a
>>> solution
>>> > based on async I/O.
>>>
>>> Note that in the upcoming Go 1.9 release we now use the netpoller for
>>> the os package as well.  However, it's not as effective as one would
>>> hope, because on GNU/Linux you can't use epoll for disk files.
>>>
>>>
>>> There's a not very well documented API to make AIO completions kick an
>>> eventfd.
>>>
>>> It
>>> mainly helps with pipes.
>>>
>>>
>>> Ian
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Realizing SSD random read IOPS

[Manish Rai Jain]

> libaio sounds good on paper, but at least on GNU/Linux it's all in user
space.

I see. That makes sense. Reading a bit more, Linux native I/O sounds like
it does exactly what we expect, i.e. save OS threads, and push this to
kernel: http://man7.org/linux/man-pages/man2/io_submit.2.html
But, I suppose this can't be part of Go, because it's not portable. Is my
understanding correct?

Also, any explanations about why GOMAXPROCS causes throughput to increase,
if new OS threads are being spawned by blocked goroutines anyway? I thought
I understood it before but now I don't.

Dave, profiler doesn't show any issues with the code itself. It's just
blocked waiting on syscalls.

$ go tool pprof randread /tmp/profile398062565/cpu.pprof
 ~/go/src/github.com/dgraph-io/badger-bench/randread
Entering interactive mode (type "help" for commands)
(pprof) top
19.48s of 19.76s total (98.58%)
Dropped 27 nodes (cum <= 0.10s)
  flat  flat%   sum%cum   cum%
19.34s 97.87% 97.87% 19.52s 98.79%  syscall.Syscall6
 0.07s  0.35% 98.23%  0.11s  0.56%  runtime.exitsyscall
 0.03s  0.15% 98.38% 19.56s 98.99%  os.(*File).ReadAt
 0.02s   0.1% 98.48%  0.10s  0.51%  math/rand.(*Rand).Intn
 0.01s 0.051% 98.53% 19.70s 99.70%  main.Conc2.func1
 0.01s 0.051% 98.58% 19.53s 98.84%  syscall.Pread
 0 0% 98.58%  0.13s  0.66%  main.getIndices
 0 0% 98.58% 19.53s 98.84%  os.(*File).pread
 0 0% 98.58% 19.70s 99.70%  runtime.goexit
(pprof)


$ go tool pprof randread /tmp/profile192709852/block.pprof
 ~/go/src/github.com/dgraph-io/badger-bench/randread
Entering interactive mode (type "help" for commands)
(pprof) top
58.48s of 58.48s total (  100%)
Dropped 8 nodes (cum <= 0.29s)
  flat  flat%   sum%cum   cum%
58.48s   100%   100% 58.48s   100%  sync.(*WaitGroup).Wait
 0 0%   100% 58.48s   100%  main.Conc2
 0 0%   100% 58.48s   100%  main.main
 0 0%   100% 58.48s   100%  runtime.goexit
 0 0%   100% 58.48s   100%  runtime.main
(pprof)


On Wed, May 17, 2017 at 3:25 PM, Dave Cheney  wrote:

> Rather than guessing what is going on, I think it's time to break out the
> profiling tools Manish.
>
> On Wed, 17 May 2017, 15:23 David Klempner  wrote:
>
>>
>> On May 16, 2017 22:03, "Ian Lance Taylor"  wrote:
>>
>> On Tue, May 16, 2017 at 9:26 PM, Manish Rai Jain 
>> wrote:
>> >> The runtime will spawn a new thread to replace the one that is blocked.
>> >
>> > Realized that after writing my last mail. And that actually explains
>> some of
>> > the other crashes we saw, about "too many threads", if we run tens of
>> > thousands of goroutines to do these reads, one goroutine per read.
>> >
>> > It is obviously lot more expensive to spawn a new OS thread. It seems
>> like
>> > this exact same problem was already solved for network via netpoller
>> > (https://morsmachine.dk/netpoller). Blocking OS threads for disk reads
>> made
>> > sense for HDDs, which could only do 200 IOPS; for SSDs we'd need a
>> solution
>> > based on async I/O.
>>
>> Note that in the upcoming Go 1.9 release we now use the netpoller for
>> the os package as well.  However, it's not as effective as one would
>> hope, because on GNU/Linux you can't use epoll for disk files.
>>
>>
>> There's a not very well documented API to make AIO completions kick an
>> eventfd.
>>
>> It
>> mainly helps with pipes.
>>
>>
>> Ian
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>>
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.