Re: [go-nuts] Field undefined error when trying to compile go file for parsing xml's

[Konstantin Khomoutov]

On Thu, 24 Nov 2016 22:20:44 -0800 (PST)
Chris S  wrote:

>  I am working on trying to parse an xml file with my go code. But I
> keep on getting an error saying:
> 
> ./main_v4.go:155: aggInfoXml.IpAddr.Hports undefined (type []Addr has
> no field or method Hports)
[...]
> type AggInfoXml struct {
> Percent IntPercent  `xml:"taskprogress,omitempty"`
> IpAddr  []Addr  `xml:"host>address,omitempty"`
> }
> 
> type IntPercent struct {
> Value float64 `xml:"percent,attr,omitempty"` // works
> }
> 
> 
> type Addr struct {
> Ip  string   `xml:"addr,attr"`
> Hports  []Ports  `xml:"host>ports>port,omitempty"` //failing to
> be recognized
> }
[...]
> for _, port := range aggInfoXml.IpAddr.Hports {
> fmt.Printf("Port: %s\n", port.ports())
> }

The IpAddr field of the AggInfoXml struct is a slice.
Slices have no fields; but they have elements which should be accessed
through the indexing operation, like in aggInfoXml[0] for instance.

Hence "aggInfoXml.IpAddr.Hports" has no sense at all.

You might want to use

  for _, addr := range range aggInfoXml.IpAddr {
for _, port := range addr.Hports {
}
  }

or something like this -- that is, Hports is only defined on instances
of type Addr, not on slices of this type.

[...]

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] bug report?: go font box drawing is wrong

[seanphaugh]

The ascenders for blocks, lines, etc are all wrong and it's very 
distracting. I'm enjoying go mono in the terminal so far but it's tedious 
to go through every file and delete the problematic character ranges. Can 
these be removed from the font until they're done properly or just copied 
from a font that does it right like DejaVu?

Sorry if this is the wrong place. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GoLang assembly support shifted operands?

[Wei . Xiao]

I have filed the issue: https://github.com/golang/go/issues/18070
and submit a patch for it: https://go-review.googlesource.com/#/c/33595/

On Tuesday, November 15, 2016 at 9:50:00 AM UTC+8, william@gmail.com 
wrote:
>
> Do you have plan to restore them back for reducing code size?
>
> On Tuesday, November 15, 2016 at 5:19:56 AM UTC+8, Aram Hăvărneanu wrote:
>>
>> They used to be supported but I think I dropped them with the old C 
>> assembler. 
>>
>> -- 
>> Aram Hăvărneanu 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Deleting the /r/golang subreddit

[vacuuz]

I would just like to point out what a genuinely vile response this is. Your 
vague and menacing implication is that Florian is somehow a sexist because 
he disagrees with you is beneath you, Dave, and you should apologise. 


On Friday, November 25, 2016 at 6:26:40 PM UTC+13, Dave Cheney wrote:
>
> On Fri, Nov 25, 2016 at 4:24 PM, Florian Weimer  > wrote: 
> > * Brad Fitzpatrick: 
> > 
> >> In light of the CEO of Reddit admitting to editing user comments (see 
> >> dozen news stories today), I propose we delete the /r/golang subreddit. 
> >> 
> >> That is so beyond unethical and immature, 
> > 
> > Was it immature because they didn't make any money out of it, at least 
> > not directly?  Modifying user-generated content without informed 
> > consent is standard business practice.  You must be aware of that. 
> > (Many people who read this will see totally misleading ads next to 
> > this post, for instance, and I obviously never agreed to that.) 
> > 
> > It was a bad prank for sure.  But it was just a prank. 
>
> Do you normalise the behaviour of all men, or only those at the C level? 
>
> > 
> > -- 
> > You received this message because you are subscribed to a topic in the 
> Google Groups "golang-nuts" group. 
> > To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/golang-nuts/XoOhzUClDPs/unsubscribe. 
> > To unsubscribe from this group and all its topics, send an email to 
> golang-nuts...@googlegroups.com . 
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Checking whether html/css toggle switch has been set to on/off position in go

[Chris S]

Hi,

I am trying to implement a toggle switch in my webpage. I've followed the 
site below for doing so:

http://www.w3schools.com/howto/howto_css_switch.asp

I currently have my html file set up with a button and this toggle switch. 
I also configured my webserver in go to be listening on localhost:8080. And 
I have a websocket handler configured, so that I can easily pass data 
through to my webpage on the click of the button.

What I want to do create a toggle switch on my webpage that the user can 
switch on and off, and then have them click a button. After that button is 
clicked I want to analyse the users selection using an if condition in my 
golang code based on whether this toggle switch is on/off, but I cannot 
figure out how to access this value in go. Any suggestions would be 
helpful. Also, it'd be ideal to have a toggle switch implemented, but if 
anyone has any simpler ideas for this use case then I'd be open to them.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Hide struct fields from frontend code (JSON REST API)

[Mirco Zeiss]

That might work. Thank you! Will try and report back.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] CFG for a Go program

[Michael Jones]

Details of this would make a great Go Blog post…

 

From: adonovan via golang-nuts 
Reply-To: 
Date: Sunday, November 27, 2016 at 6:07 PM
To: golang-nuts 
Cc: 
Subject: Re: [go-nuts] CFG for a Go program

 

If you're building tools for source code analysis, you may find the 
golang.org/x/go/ssa representation easier to work with than the internals of 
the compiler.  Build and run this command to see an example:

 

  $ go get golang.org/x/tools/cmd/ssadump

  $ ssadump -build=F fmt

 

Alternatively, the cmd/vet tool in the standard library has an internal 
subpackage that constructs the control-flow graph of a function.  In this 
representation, each block contains a sequence of Go statements and 
expressions, not low-level SSA instructions.

 

Which of these forms of control-flow graph is most appropriate depends on the 
(unmentioned) problem you're trying to solve.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: goroutine scheduler is a trap?????!!!!!

[370265036]

thanks.
i have checked my code and analyze the cpu prof file for many times.i do 
not see any problems.

在 2016年11月26日星期六 UTC+8上午2:45:13，Didier Spezia写道：
>
> Any non trivial program is scalable up a certain point.
> In your code, you may have some contention.
> It can be physical (CPU, memory, network, disks, etc ...) or logical 
> (mutexes, channels, shared memory, etc ...)
> If it is not in your own code, then perhaps it may be in the packages you 
> use, including the Go standard libraries or the runtime itself.
> The exact cause is difficult to guess, that's why profiling is useful.
>
> Also, the Go scheduler is currently not NUMA aware. It does not exploit 
> well the locality in a machine having multiple sockets.
> On a two sockets box, it is usually more efficient to run two instances of 
> the application. and pin them to their own NUMA node, rather than one 
> single instance sharing all the CPUs.
>
> Regards.
> Didier.
>
> On Thursday, November 24, 2016 at 10:02:21 PM UTC+1, 3702...@qq.com wrote:
>>
>> today i am testing my service(something like cdn).system is centos 
>> 6.5,memory=32GB,bandwidth=10Gb and 4000 live stream.one mechine is 16-core 
>> cpu and the other is 8-core cpu.the testing result makes me feel 
>> upset,16-core cpu mechine run  about 1000% cpu and play is not smooth,but 
>> the 8-core cpu mechine run about 600% cpu and play 
>> smooth!!!why?could someone help??help me
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: goroutine scheduler is a trap?????!!!!!

[370265036]

thanks for response.
i am using the version is go1.7, i do not know it's detail version.
yes, i have profiled my program,the cpu profile,16-core cpu and 8-core cpu 
both done.the percent of the same functions almost at the same,but the 
total cpu.this is interesting.

i will follow your suggestion,use the Go 1.7.3 to see what will be happened.

在 2016年11月25日星期五 UTC+8上午6:43:04，Dave Cheney写道：
>
> I'm sorry you're having problems.
>
> Before we can help you need to help us by giving us details.
>
> - Which version of Go are you using? If you are using less than Go 1.7.3, 
> please upgrade.
> - Have you profiled your program? This is the first step to diagnosing any 
> performance problem, here is a presentation with many ways to profile a Go 
> program. 
> http://go-talks.appspot.com/github.com/davecheney/presentations/seven.slide
>
> On Friday, 25 November 2016 08:02:21 UTC+11, 3702...@qq.com wrote:
>>
>> today i am testing my service(something like cdn).system is centos 
>> 6.5,memory=32GB,bandwidth=10Gb and 4000 live stream.one mechine is 16-core 
>> cpu and the other is 8-core cpu.the testing result makes me feel 
>> upset,16-core cpu mechine run  about 1000% cpu and play is not smooth,but 
>> the 8-core cpu mechine run about 600% cpu and play 
>> smooth!!!why?could someone help??help me
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Inevitable unexpected EOF

[Connor Peet]

That's exactly what I was look for, thanks so much! I've prefixed each 
message with its original size and it works perfectly.

On Sunday, November 27, 2016 at 6:52:30 PM UTC-5, thebroke...@gmail.com 
wrote:
>
> This is entirely proper behavior of the gzip implementation (or more 
> specifically, compress/flate). At the edges of the LZ77 window 
> (approximately 32KiB), you are not guaranteed that you can simply for loop 
> over the gzip.Reader until the input buffer is zero since the 
> implementation of the decompressor tries to avoid ever returning from Read 
> with (0, nil), which is discouraged in the io.Reader implementation. Since 
> it avoids returning (0, nil), it tries to read the next byte, which causes 
> the io.ErrUnexpectedEOF error you see.
>
> If you want to use Write+Flush in conjunction with Read, you must provide 
> some type of framing such that the decompressor knows how many bytes to 
> read before it had hit the flush point.
>
> See this example: 
> https://tip.golang.org/pkg/compress/flate/#example__synchronization
>  
> On Sunday, November 27, 2016 at 2:04:09 PM UTC-8, kortschak wrote:
>>
>> I don't see anywhere that you are closing the gzip.Writer. Does doing 
>> that fix the problem? 
>>
>> On Fri, 2016-11-25 at 05:12 -0800, Connor wrote: 
>> > Hi all. 
>> > 
>> > I'm trying to implement a structure which gzips multiple  
>> > individually-inflatable messages in the same data stream. I've built 
>> > an  
>> > example implementation here: https://play.golang.org/p/hwdrVtI29t. 
>> > While  
>> > this works initially, eventually the gzip reader throws an 
>> > "unexpected EOF"  
>> > at me. I've tracked the problem down to L609 in 
>> > compress/flate/inflate.go  
>> > ; the value 
>> > returned  
>> > by f.dict.availWrite() reaches zero, causing the reader to transition 
>> > from  
>> > the nextBlock "state" to reading huffmanBlock, ending in an 
>> > unexpected EOF.  
>> > For what it's worth, I've also tested it with this  
>> >  LZ4 package which provides almost 
>> > the same  
>> > API as compress/gzip, and it works flawlessly. 
>> > 
>> > Could someone provide some pointers here? Is this the result of my 
>> > own  
>> > limited understanding of gzip/DEFLATE's mechanics, or is this 
>> > possible a  
>> > flaw in compress/(flate|gzip)? 
>> >  
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] CFG for a Go program

[adonovan via golang-nuts]

If you're building tools for source code analysis, you may find the 
golang.org/x/go/ssa representation easier to work with than the internals 
of the compiler.  Build and run this command to see an example:

  $ go get golang.org/x/tools/cmd/ssadump
  $ ssadump -build=F fmt

Alternatively, the cmd/vet tool in the standard library has an internal 
subpackage that constructs the control-flow graph of a function.  In this 
representation, each block contains a sequence of Go statements and 
expressions, not low-level SSA instructions.

Which of these forms of control-flow graph is most appropriate depends on 
the (unmentioned) problem you're trying to solve.

>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Eliminating redundant map lookups

['Keith Randall' via golang-nuts]

Yes, it currently requires two lookups (two hashes, two bucket searches, 
etc.).  This general problem is issue 17133 
(https://github.com/golang/go/issues/17133).  Your example has the extra 
complication that the update operation is an append, not just a +=.


On Sunday, November 27, 2016 at 7:00:23 AM UTC-8, Ian Cottrell wrote:
>
> Yes you will get two map operations happen, one lookup and one assign.
>
> Wrapping the code into a (mildly modified for pretty output) runnable 
> example: https://play.golang.org/p/oGKvfS9ssH
>
> In the basic case of a new key, there is no way to avoid doing both a 
> lookup and assign, but in the case where the key exists, you can avoid the 
> assign by adding another level of indirection, like this: 
> https://play.golang.org/p/Rsm1uT4tlP
> You can see the only changes are the type of the likes map (from 
> map[string]People to map[string]*People), and the way it's modified in the 
> loop.
> I don't recommend actually doing this in general though. The result is not 
> horrific, but it's definitely less clear, and most cases where the map 
> lookup is actually a significant factor, you have probably already picked 
> the wrong algorithm, so micro optimising it is not going to get you far. At 
> the very least you would want to see it actually improve a benchmark before 
> going with it.
>
>
> On Sun, 27 Nov 2016 at 13:40  wrote:
>
>> I should clarify that this actually came up in practice and I have no 
>> clue on how I can eliminate them
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] RDF "encoding"

[Patrick Logan]

Some CSV-specific specs are here...
https://www.w3.org/blog/news/archives/5232?pk_campaign=feed_kwd=csv-on-the-web-recommendations-published

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Inevitable unexpected EOF

[thebrokentoaster]

This is entirely proper behavior of the gzip implementation (or more 
specifically, compress/flate). At the edges of the LZ77 window 
(approximately 32KiB), you are not guaranteed that you can simply for loop 
over the gzip.Reader until the input buffer is zero since the 
implementation of the decompressor tries to avoid ever returning from Read 
with (0, nil), which is discouraged in the io.Reader implementation. Since 
it avoids returning (0, nil), it tries to read the next byte, which causes 
the io.ErrUnexpectedEOF error you see.

If you want to use Write+Flush in conjunction with Read, you must provide 
some type of framing such that the decompressor knows how many bytes to 
read before it had hit the flush point.

See this 
example: https://tip.golang.org/pkg/compress/flate/#example__synchronization
 
On Sunday, November 27, 2016 at 2:04:09 PM UTC-8, kortschak wrote:
>
> I don't see anywhere that you are closing the gzip.Writer. Does doing 
> that fix the problem? 
>
> On Fri, 2016-11-25 at 05:12 -0800, Connor wrote: 
> > Hi all. 
> > 
> > I'm trying to implement a structure which gzips multiple  
> > individually-inflatable messages in the same data stream. I've built 
> > an  
> > example implementation here: https://play.golang.org/p/hwdrVtI29t. 
> > While  
> > this works initially, eventually the gzip reader throws an 
> > "unexpected EOF"  
> > at me. I've tracked the problem down to L609 in 
> > compress/flate/inflate.go  
> > ; the value 
> > returned  
> > by f.dict.availWrite() reaches zero, causing the reader to transition 
> > from  
> > the nextBlock "state" to reading huffmanBlock, ending in an 
> > unexpected EOF.  
> > For what it's worth, I've also tested it with this  
> >  LZ4 package which provides almost 
> > the same  
> > API as compress/gzip, and it works flawlessly. 
> > 
> > Could someone provide some pointers here? Is this the result of my 
> > own  
> > limited understanding of gzip/DEFLATE's mechanics, or is this 
> > possible a  
> > flaw in compress/(flate|gzip)? 
> >  
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] RDF "encoding"

[Dan Kortschak]

On Sun, 2016-11-27 at 01:46 -0800, Johann Höchtl wrote:
> Is there a standard to perform IRIs encoding? Would URL-encoding the
> read 
> part be acceptable and interoperable?


There is a published grammar for RDF, which defines the IRI grammar
used. This can be used to write a parser. An example of this is here
for RDF n-quads:
https://godoc.org/github.com/cayleygraph/cayley/quad/nquads

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Inevitable unexpected EOF

[Dan Kortschak]

I don't see anywhere that you are closing the gzip.Writer. Does doing
that fix the problem?

On Fri, 2016-11-25 at 05:12 -0800, Connor wrote:
> Hi all.
> 
> I'm trying to implement a structure which gzips multiple 
> individually-inflatable messages in the same data stream. I've built
> an 
> example implementation here: https://play.golang.org/p/hwdrVtI29t.
> While 
> this works initially, eventually the gzip reader throws an
> "unexpected EOF" 
> at me. I've tracked the problem down to L609 in
> compress/flate/inflate.go 
> ; the value
> returned 
> by f.dict.availWrite() reaches zero, causing the reader to transition
> from 
> the nextBlock "state" to reading huffmanBlock, ending in an
> unexpected EOF. 
> For what it's worth, I've also tested it with this 
>  LZ4 package which provides almost
> the same 
> API as compress/gzip, and it works flawlessly.
> 
> Could someone provide some pointers here? Is this the result of my
> own 
> limited understanding of gzip/DEFLATE's mechanics, or is this
> possible a 
> flaw in compress/(flate|gzip)?
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] multiple html templates

[keith6014]

Hello,

I am building a site which will have the following pages, "/" (home), 
"/about", and "/data" which will show some data using DataTables 
. 

"/", "/about" will be simple so I am using templates/base.html but for 
/data I would still like to use tempaltes/data.html because I need to load 
DataTables code. To do this I am creating a seperate handler and I am not 
sure if this is the right way to do it.

So, is a seperate handler needed? and how inefficient is it?




var templates = template.Must(template.ParseFiles("templates/base.html", 
"templates/data.html")) 

func handler(w http.ResponseWriter, r *http.Request) { 
  
log.Printf(r.URL.Path[1:]) 
myVars := 
Variables{"Main", "My Heading #1"} 
err := 
templates.ExecuteTemplate(w, "base.html", myVars)   

  if err != nil {   

http.Error(w, err.Error(), http.StatusInternalServerError) 
  
return 
}
}

 
func staticHandler(w http.ResponseWriter, r *http.Request) {   


http.ServeFile(w, r, r.URL.Path[1:])   


}   

   


   
func dataHandler(w http.ResponseWriter, r *http.Request) { 


myVars := Variables{"About", "About page"} 


err := templates.ExecuteTemplate(w, "data.html", myVars)   


if err != nil { 

   
http.Error(w, err.Error(), http.StatusInternalServerError) 


return 


}   

   
}   

   
 
func main() {   

   
fmt.Printf("ok\n") 


var port string = ":9000"   

   
log.Printf("starting up on %s", port)   

  
http.HandleFunc("/", handler)   

   

Re: [go-nuts] Do Go Routines use continuations to allow yelding?

[Jesper Louis Andersen]

On Sun, Nov 27, 2016 at 5:47 PM Michael Jones 
wrote:

> Wait… *“If any of these fails to produce a correct result…”* Really? I
> believe that we want the correct result under any valid schedule, including
> the most perverse, and that anything less means a failure of the design of
> the underlying system. Perhaps I misunderstand what you imply.
>
>
>
Or perhaps I'm just bad at writing :)

What I meant is what you wrote: we want to try random schedules (under the
assumption that these random schedules are possible interleavings). The
code must behave the same, observationally, under any of these schedules.

In particular, pick any schedule and deem it the "deterministic one". If
the code is correct, any schedule should behave as the "determinstic one"
observationally.

The problem is that most optimized schedulers have a "common schedule
order" which they prefer. The other ways to schedule only shows up once the
system is under stress or load. And the reason we get another schedule is
often because some package in the system, totally unrelated to the code we
are scrutinizing, executes in the same process, but in another goroutine.
Thus, it pushes our code in ways which alters the schedule. Test cases,
which isolate packages, are not going to find anything amiss.

Also note that this has nothing to do with data races (which the race
detector finds). Usually the reason your code breaks down has to do with
some kind of causal dependency you didn't account for logically in your
code. It then leads to collapse in a relative or temporal way[0].

Luckily, there are many situations in which your system doesn't really have
the guarantees you think, but it will gracefully recover from the odd
events anyhow. In practice, we can often get away with a weaker consistency
than a linearizable consistency. And this is why many concurrent programs
can run, even though there are some schedules in them which have never been
tested or are slightly dubious from a correctness perspective.

[0] The funniest example is when you have three planets, A, X, and B where
X is in the middle between A and B. If a message is broadcast at exactly
the same point in time from the planets, the arrival is different on the
planets:

A sees A, X, and then B
B sees B, X and then A
X sees X, and then A+B at the same time (any interleaving is possible if we
force serialization)

You may be tempted to write a protocol which assumes these orderings, but
planets move, so you can't. You are forced to obey the laws of physics and
they say information flow is relative. The same happens inside a CPU,
though the window is measured in nanoseconds, not years.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Do Go Routines use continuations to allow yelding?

[Michael Jones]

Wait… “If any of these fails to produce a correct result…” Really? I believe 
that we want the correct result under any valid schedule, including the most 
perverse, and that anything less means a failure of the design of the 
underlying system. Perhaps I misunderstand what you imply.

 

From:  on behalf of Jesper Louis Andersen 

Date: Sunday, November 27, 2016 at 3:58 AM
To: Olivier El Mekki , golang-nuts 

Cc: 
Subject: Re: [go-nuts] Do Go Routines use continuations to allow yelding?

 

On Sat, Nov 26, 2016 at 8:32 PM Olivier El Mekki  wrote:

 

This could yield interesting properties, because this means that even using 
channels, maybe we could write concurrent code and still have a chance to have 
predictable and reproducible execution in testing environment. 

 

 

I think it is a bit more complex:

 

Having a deterministic execution of serial & sequential nature is good for 
establishing a base-point of what a given routine is going to do. The key point 
of a concurrent/parallel environment is that there should be no discrepancy 
between this base-point and any possible execution order of concurrent 
goroutines.

 

In short, you want to randomize the scheduler to run highly unlikely schedules. 
If any of these fails to produce a correct result, you want to know what order 
the scheduler executed the goroutines in. You also want the system to 
automatically degrade the concurrent schedule into a sequential one, so you can 
find the exact interleaving which is the culprit of the problem.

 

The reason this is possible has to do with linearizability. An external party, 
observing the sequential system will see a trace of result-events from the 
system in the test. A correctness criterion which is necessary but not 
sufficient is that the concurrent execution of the same program should produce 
a valid linearizable trace. That is, given a trace of observations, we must ask 
"is there any of the sequential executions which can give such a trace?". If 
yes, then the trace is valid. If no, we can report an error.

 

It isn't as far-fetched as one might think. For example, Kyle Kingsbury are 
using these ideas in his Jepsen tool, for testing the linearizability (and 
other) properties of distributed databases. In his setup, you record a trace of 
events from a distributed database. The database runs virtually in VMs and the 
network is forcefully cut via firewall rules. Afterwards, a tool, Knossos, 
tries to find errors in the traces by looking for invalid observations. It 
proves to be a very efficient way to test the PR-documentation of distributed 
databases and verify the claims of guarantees the databases have. Usually, 
vendors are trying to hand-wave themselves through impossibility results from 
distributed computing (such as the CAP theorem).

 

Another example is PULSE & Concuerror from the Erlang world. A program is 
compiled with instrumentation. This instrumentation inserts a control-point 
before each possible blocking request, so a separate scheduler can handle the 
actual interleaving of the program. Now, the properties of concurrent execution 
can be studied by forcing schedules in a certain order. The crucial trick in 
these systems is a function

 

    func split(seed int) (seedL int, seedR int)

    

which can split the current PRNG seed into two seeds, left and right. It is 
needed to simplify a schedule once a failing schedule is found. Imagine you 
were able to draw the sequence-diagram: you want it to be as small as possible 
since that is easier to understand.

 

Whenever you start up a possible subcomputation, you split the seed. Now, if 
you want to throw away the subcomputation, you throw away either seedL or 
seedR. This keeps the PRNG intact for the other half. If you had a standard 
linear seed (rather than a tree which the above essentially is) the problem is 
that you have to skip a lot of random values in the PRNG for the 
subcomputations you don't use. This allows you to ignore subcomputations which 
are not relevant to the current error, throw them away and continue as if they 
never happened. In particular, you can throw away operations from the trace by 
excising them out—simplifying things in the process.

 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Eliminating redundant map lookups

['Ian Cottrell' via golang-nuts]

Yes you will get two map operations happen, one lookup and one assign.

Wrapping the code into a (mildly modified for pretty output) runnable
example: https://play.golang.org/p/oGKvfS9ssH

In the basic case of a new key, there is no way to avoid doing both a
lookup and assign, but in the case where the key exists, you can avoid the
assign by adding another level of indirection, like this:
https://play.golang.org/p/Rsm1uT4tlP
You can see the only changes are the type of the likes map (from
map[string]People to map[string]*People), and the way it's modified in the
loop.
I don't recommend actually doing this in general though. The result is not
horrific, but it's definitely less clear, and most cases where the map
lookup is actually a significant factor, you have probably already picked
the wrong algorithm, so micro optimising it is not going to get you far. At
the very least you would want to see it actually improve a benchmark before
going with it.


On Sun, 27 Nov 2016 at 13:40  wrote:

I should clarify that this actually came up in practice and I have no clue
on how I can eliminate them

-- 
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Eliminating redundant map lookups

[Andrei Tudor Călin]

In general, a good way to answer questions like this is to inspect an 
assembly listing for the code. This is what I compiled:

$ cat foo.go
package foo

import (
"fmt"
)

type Person struct {
Name  string
Likes []string
}

func foo() {
var people []*Person
likes := make(map[string][]*Person)
for _, p := range people {
for _, l := range p.Likes {
likes[l] = append(likes[l], p) //This line
}
}
fmt.Println(likes)
}

Now for the assembly listing, as it appears on my machine (linux/amd64).
Everything but the critical section is elided, for brevity.

$ go tool compile -S foo.go
"".foo t=1 size=742 args=0x0 locals=0xd0
0x 0 (foo.go:12) TEXT "".foo(SB), $208-0
0x 0 (foo.go:12) MOVQ (TLS), CX
0x0009 9 (foo.go:12) LEAQ -80(SP), AX
0x000e 00014 (foo.go:12) CMPQ AX, 16(CX)
0x0012 00018 (foo.go:12) JLS 732
0x0018 00024 (foo.go:12) SUBQ $208, SP
0x001f 00031 (foo.go:12) MOVQ BP, 200(SP)
0x0027 00039 (foo.go:12) LEAQ 200(SP), BP
0x002f 00047 (foo.go:12) FUNCDATA $0, 
gclocals·3e27b3aa6b89137cce48b3379a2a6610(SB)
0x002f 00047 (foo.go:12) FUNCDATA $1, 
gclocals·c0d3101dde958787c02a66c7a040e782(SB)
0x002f 00047 (foo.go:14) LEAQ type.map[string][]*"".Person(SB), AX
0x0036 00054 (foo.go:14) MOVQ AX, (SP)
0x003a 00058 (foo.go:14) MOVQ $0, 8(SP)
0x0043 00067 (foo.go:14) MOVQ $0, 16(SP)
0x004c 00076 (foo.go:14) MOVQ $0, 24(SP)
0x0055 00085 (foo.go:14) PCDATA $0, $0
0x0055 00085 (foo.go:14) CALL runtime.makemap(SB)
0x005a 00090 (foo.go:14) MOVQ 32(SP), AX
0x005f 00095 (foo.go:14) MOVQ AX, "".likes+112(SP)
0x0064 00100 (foo.go:15) MOVQ $0, CX
0x0066 00102 (foo.go:13) MOVQ $0, DX
0x0068 00104 (foo.go:15) MOVQ CX, "".autotmp_19+96(SP)
0x006d 00109 (foo.go:15) MOVQ DX, "".autotmp_20+136(SP)
0x0075 00117 (foo.go:15) TESTQ CX, CX
0x0078 00120 (foo.go:15) JGE $0, 493
0x007e 00126 (foo.go:15) MOVQ (DX), BX
0x0081 00129 (foo.go:15) MOVQ BX, "".p+104(SP)
0x0086 00134 (foo.go:16) MOVQ 16(BX), SI
0x008a 00138 (foo.go:16) MOVQ 24(BX), DI
0x008e 00142 (foo.go:16) MOVQ DI, "".autotmp_21+88(SP)
0x0093 00147 (foo.go:15) MOVQ $0, R8
0x0096 00150 (foo.go:16) MOVQ R8, "".autotmp_22+80(SP)
0x009b 00155 (foo.go:16) MOVQ SI, "".autotmp_23+128(SP)
0x00a3 00163 (foo.go:16) CMPQ R8, DI
0x00a6 00166 (foo.go:16) JGE $0, 464
0x00ac 00172 (foo.go:16) MOVQ (SI), R9
0x00af 00175 (foo.go:16) MOVQ 8(SI), R10
0x00b3 00179 (foo.go:17) MOVQ R9, "".autotmp_2+144(SP)
0x00bb 00187 (foo.go:17) MOVQ R10, "".autotmp_2+152(SP)
0x00c3 00195 (foo.go:14) LEAQ type.map[string][]*"".Person(SB), R11
0x00ca 00202 (foo.go:17) MOVQ R11, (SP)
0x00ce 00206 (foo.go:17) MOVQ AX, 8(SP)
0x00d3 00211 (foo.go:17) MOVQ R9, 16(SP)
0x00d8 00216 (foo.go:17) MOVQ R10, 24(SP)
0x00dd 00221 (foo.go:17) PCDATA $0, $1
0x00dd 00221 (foo.go:17) CALL runtime.mapaccess1_faststr(SB)
0x00e2 00226 (foo.go:17) MOVQ 32(SP), AX
0x00e7 00231 (foo.go:17) MOVQ 8(AX), CX
0x00eb 00235 (foo.go:17) MOVQ CX, "".autotmp_24+72(SP)
0x00f0 00240 (foo.go:17) MOVQ 16(AX), DX
0x00f4 00244 (foo.go:17) MOVQ (AX), AX
0x00f7 00247 (foo.go:17) LEAQ 1(CX), BX
0x00fb 00251 (foo.go:17) CMPQ BX, DX
0x00fe 00254 (foo.go:17) JGT $0, 667
0x0104 00260 (foo.go:17) MOVQ AX, "".autotmp_25+120(SP)
0x0109 00265 (foo.go:17) MOVQ BX, "".autotmp_24+72(SP)
0x010e 00270 (foo.go:17) MOVQ DX, "".autotmp_26+64(SP)
0x0113 00275 (foo.go:17) LEAQ (AX)(CX*8), SI
0x0117 00279 (foo.go:17) MOVL runtime.writeBarrier(SB), DI
0x011d 00285 (foo.go:17) TESTB DIB, DIB
0x0120 00288 (foo.go:17) JNE $0, 623
0x0126 00294 (foo.go:17) MOVQ "".p+104(SP), SI
0x012b 00299 (foo.go:17) MOVQ SI, (AX)(CX*8)
0x012f 00303 (foo.go:17) MOVQ BX, "".autotmp_5+184(SP)
0x0137 00311 (foo.go:17) MOVQ DX, "".autotmp_5+192(SP)
0x013f 00319 (foo.go:17) MOVL runtime.writeBarrier(SB), CX
0x0145 00325 (foo.go:17) TESTB CL, CL
0x0147 00327 (foo.go:17) JNE $0, 591
0x014d 00333 (foo.go:17) MOVQ AX, "".autotmp_5+176(SP)
0x0155 00341 (foo.go:14) LEAQ type.map[string][]*"".Person(SB), AX
0x015c 00348 (foo.go:17) MOVQ AX, (SP)
0x0160 00352 (foo.go:17) MOVQ "".likes+112(SP), CX
0x0165 00357 (foo.go:17) MOVQ CX, 8(SP)
0x016a 00362 (foo.go:17) LEAQ "".autotmp_2+144(SP), DX
0x0172 00370 (foo.go:17) MOVQ DX, 16(SP)
0x0177 00375 (foo.go:17) LEAQ "".autotmp_5+176(SP), DX
0x017f 00383 (foo.go:17) MOVQ DX, 24(SP)
0x0184 00388 (foo.go:17) PCDATA $0, $3
0x0184 00388 (foo.go:17) CALL runtime.mapassign1(SB)
0x0189 00393 (foo.go:16) MOVQ "".autotmp_23+128(SP), R9
0x0191 00401 (foo.go:16) LEAQ 16(R9), SI
0x0195 00405 (foo.go:16) MOVQ "".autotmp_22+80(SP), R9
0x019a 00410 (foo.go:16) LEAQ 1(R9), R8
0x019e 00414 (foo.go:14) MOVQ "".likes+112(SP), AX
0x01a3 00419 (foo.go:15) MOVQ "".autotmp_19+96(SP), CX
0x01a8 00424 (foo.go:15) MOVQ "".autotmp_20+136(SP), DX
0x01b0 00432 (foo.go:15) MOVQ "".p+104(SP), BX
0x01b5 00437 (foo.go:16) MOVQ "".autotmp_21+88(SP), DI
0x01ba 00442 (foo.go:16) MOVQ R8, "".autotmp_22+80(SP)
0x01bf 00447 (foo.go:16) MOVQ SI, "".autotmp_23+128(SP)
0x01c7 00455 (foo.go:16) CMPQ R8, DI
0x01ca 00458 (foo.go:16) JLT $0, 172
0x01d0 00464 (foo.go:15) ADDQ $8, DX
0x01d4 00468 

[go-nuts] Re: stripping null in json string

[C Banning]

Not sure what you're looking for but here's a couple 
examples: https://play.golang.org/p/Rrg3kWkuGV

On Thursday, November 24, 2016 at 2:02:20 PM UTC-7, Seanchann Zhou wrote:
>
> hi:
>have json:
>
>   "test": {
>   "test1": null,
>   "test2": null,
>   "test3": null,
>   "test4": null,
>   "proto": "dhcp",
>   "test5": null
> }
>
>
>how to safety stripping null field from above json. thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Eliminating redundant map lookups

[stinkingmadgod]

I should clarify that this actually came up in practice and I have no clue 
on how I can eliminate them

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Eliminating redundant map lookups

[stinkingmadgod]

The following code is taken from https://blog.golang.org/go-maps-in-action.

type Person struct {Name  stringLikes []string}var 
people []*Person*likes := make(map[string][]*Person)*for _, p := range 
people {for _, l := range p.Likes {*likes[l] = 
append(likes[l], p)//This line*}}


Am I being paranoid in thinking the line with two likes[l] is actually doing 
two lookups ( two hashes, two probes ) and should be eliminated somehow?

Coming from a C++ background ( gophers have mercy :P ), things like this really 
bothers me.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Do Go Routines use continuations to allow yelding?

[Jesper Louis Andersen]

On Sat, Nov 26, 2016 at 8:32 PM Olivier El Mekki 
wrote:

>
> This could yield interesting properties, because this means that even
> using channels, maybe we could write concurrent code and still have a
> chance to have predictable and reproducible execution in testing
> environment.
>
>
>
I think it is a bit more complex:

Having a deterministic execution of serial & sequential nature is good for
establishing a base-point of what a given routine is going to do. The key
point of a concurrent/parallel environment is that there should be no
discrepancy between this base-point and any possible execution order of
concurrent goroutines.

In short, you want to randomize the scheduler to run highly unlikely
schedules. If any of these fails to produce a correct result, you want to
know what order the scheduler executed the goroutines in. You also want the
system to automatically degrade the concurrent schedule into a sequential
one, so you can find the exact interleaving which is the culprit of the
problem.

The reason this is possible has to do with linearizability. An external
party, observing the sequential system will see a trace of result-events
from the system in the test. A correctness criterion which is necessary but
not sufficient is that the concurrent execution of the same program should
produce a valid linearizable trace. That is, given a trace of observations,
we must ask "is there any of the sequential executions which can give such
a trace?". If yes, then the trace is valid. If no, we can report an error.

It isn't as far-fetched as one might think. For example, Kyle Kingsbury are
using these ideas in his Jepsen tool, for testing the linearizability (and
other) properties of distributed databases. In his setup, you record a
trace of events from a distributed database. The database runs virtually in
VMs and the network is forcefully cut via firewall rules. Afterwards, a
tool, Knossos, tries to find errors in the traces by looking for invalid
observations. It proves to be a very efficient way to test the
PR-documentation of distributed databases and verify the claims of
guarantees the databases have. Usually, vendors are trying to hand-wave
themselves through impossibility results from distributed computing (such
as the CAP theorem).

Another example is PULSE & Concuerror from the Erlang world. A program is
compiled with instrumentation. This instrumentation inserts a control-point
before each possible blocking request, so a separate scheduler can handle
the actual interleaving of the program. Now, the properties of concurrent
execution can be studied by forcing schedules in a certain order. The
crucial trick in these systems is a function

func split(seed int) (seedL int, seedR int)
which can split the current PRNG seed into two seeds, left and right. It is
needed to simplify a schedule once a failing schedule is found. Imagine you
were able to draw the sequence-diagram: you want it to be as small as
possible since that is easier to understand.

Whenever you start up a possible subcomputation, you split the seed. Now,
if you want to throw away the subcomputation, you throw away either seedL
or seedR. This keeps the PRNG intact for the other half. If you had a
standard linear seed (rather than a tree which the above essentially is)
the problem is that you have to skip a lot of random values in the PRNG for
the subcomputations you don't use. This allows you to ignore
subcomputations which are not relevant to the current error, throw them
away and continue as if they never happened. In particular, you can throw
away operations from the trace by excising them out—simplifying things in
the process.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Do Go Routines use continuations to allow yelding?

[Dave Cheney]

On Sun, Nov 27, 2016 at 10:25 PM, Jesper Louis Andersen
 wrote:
> On Fri, Nov 25, 2016 at 10:51 PM Dave Cheney  wrote:
>>
>>
>> Yes, goroutine switching occurs a known points; effectively where the
>> goroutine is blocked from proceeding; sending and receiving on channels (if
>> they are full/empty respectively), trying to acquire a locked mutex, and
>> performing IO. These operations happen frequently in a program that
>> interacts with others so things more or less work out.
>>
> I was of the impression Go checks on function calls nowadays. Since you have
> to check the stack anyway, you can always piggyback on that routine if you
> want the system to schedule out the goroutine.

Yes, that is correct, the runtime has its own goroutine called sysmon
which waits up to 10ms and then sets the stack size to the longest
running goroutine to a negative number. The next time the goroutine
executes a function call it will trap into the stack growth slow path
and be preempted.

>
> Only checking on points-of-communication could lead to worse latency because
> a non-communicating goroutine is then able to tie up a processing core
> indefinitely, and delay garbage collection.

Yup, the problem with a spinning goroutine blocking the STW safe point
has been known for several releases. The mechanism above appears to
limit the worst case to 20ms in my testing.

> Not checking inside for-loops puts the onus on the programmer to break such
> loops via function calls, but this is a far easier thing to handle from a
> programmer perspective when needed, and when not needed it could be argued
> it needlessly slows down the program.

David Chase has adding a check on the loop edge as an experiment, it
comes at about a 15% cost.


>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Do Go Routines use continuations to allow yelding?

[Jesper Louis Andersen]

On Fri, Nov 25, 2016 at 10:51 PM Dave Cheney  wrote:

>
> Yes, goroutine switching occurs a known points; effectively where the
> goroutine is blocked from proceeding; sending and receiving on channels (if
> they are full/empty respectively), trying to acquire a locked mutex, and
> performing IO. These operations happen frequently in a program that
> interacts with others so things more or less work out.
>
> I was of the impression Go checks on function calls nowadays. Since you
have to check the stack anyway, you can always piggyback on that routine if
you want the system to schedule out the goroutine.

Only checking on points-of-communication could lead to worse latency
because a non-communicating goroutine is then able to tie up a processing
core indefinitely, and delay garbage collection.

Not checking inside for-loops puts the onus on the programmer to break such
loops via function calls, but this is a far easier thing to handle from a
programmer perspective when needed, and when not needed it could be argued
it needlessly slows down the program.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] RDF "encoding"

[Johann Höchtl]

I am relatively new to RDF. I plan to use

https://github.com/knakk/rdf

to serialize data I read from CSV

Of course it fails to encode invalid IRIs like "example.com/A + B" where I 
read the information "A + B" from a CSV file.

Is there a standard to perform IRIs encoding? Would URL-encoding the read 
part be acceptable and interoperable?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.