Re: [go-nuts] Unmarshal Json

[Shawn Milochik]

Awesome! You're welcome.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Simple web crawler question. How to avoid crawling visited links?

[Michael Jones]

Aaron, this is the simplest change I could make to that program to show you
an example. It does what you want.

https://play.golang.org/p/Q8pu0l-Yvy

It shows what I was saying about replacing a string with a struct of string
and int.

Depth 1:

http://www.whitehouse.gov
https://www.whitehouse.gov/blog/2017/06/08/president-trumps-plan-rebuild-americas-infrastructure
:
https://www.whitehouse.gov/privacy
https://www.whitehouse.gov/copyright

On Tue, Sep 26, 2017 at 7:29 AM, Michael Jones 
wrote:

> To be sure, examples are a starting place.
>
> To crawl to a certain depth, you need to know the depth of each url. That
> means that the url being passed around stops being "just a string" and
> becomes a struct with information including at least, the url as a string
> and the link depth as an integer. Following a link means incrementing the
> depth, checking that, and also checking the presence of the string part in
> the map.
>
> To crawl to a certain distance from a website, in the sense of how many
> hops to other websites, needs similar processing on the hostname.
>
> To crawl with rate-limitation so as not to overburden a distant host's
> server(s), means to have multiple work queues for each hostname. That is
> the natural place to do rate limits, time windows, or other such processing.
>
> The real meaning of the example is to point out that the work queue -- a
> channel here -- can be written to by the (go)routines that read from it.
> This is a useful notion in this case.
>
> Good luck!
>
> On Mon, Sep 25, 2017 at 8:46 PM, Aaron  wrote:
>
>> Thank you Mike. I have read the book and done that exercise. The code in
>> the example will not crawl into certain depth and stop at certain depth of
>> the website. The requirements are a bit different. I can't just use the
>> approach in the example directly or more precisely I don't know how to use
>> it in my code.
>>
>> On Tuesday, September 26, 2017 at 6:18:43 AM UTC+8, Michael Jones wrote:
>>>
>>> The book, The Go Programming Language discusses the web crawl task at
>>> several points through the text. The simplest complete parallel version is:
>>>
>>> https://github.com/adonovan/gopl.io/blob/master/ch8/crawl3/findlinks.go
>>>
>>> which if you'll download and build works quite nicely:
>>>
>>> *$ crawl3 http://www.golang.org *
>>> http://www.golang.org
>>> http://www.google.com/intl/en/policies/privacy/
>>> https://golang.org/doc/tos.html
>>> https://golang.org/project/
>>> https://golang.org/pkg/
>>> https://golang.org/doc/
>>> http://play.golang.org/
>>> https://tour.golang.org/
>>> https://golang.org/LICENSE
>>> https://developers.google.com/site-policies#restrictions
>>> https://golang.org/dl/
>>> https://golang.org/blog/
>>> https://golang.org/help/
>>> https://golang.org/
>>> https://blog.golang.org/
>>> https://www.google.com/intl/en/privacy/privacy-policy.html
>>> https://www.google.com/intl/en/policies/terms/
>>> https://golang.org/LICENSE?m=text
>>> https://golang.org/pkg
>>> https://golang.org/doc/go_faq.html
>>> https://groups.google.com/group/golang-nuts
>>> https://blog.gopheracademy.com/gophers-slack-community/
>>> https://golang.org/wiki
>>> https://forum.golangbridge.org/
>>> irc:irc.freenode.net/go-nuts
>>> 2017/09/25 15:13:07 Get irc:irc.freenode.net/go-nuts: unsupported
>>> protocol scheme "irc"
>>> https://golang.org/doc/faq
>>> https://groups.google.com/group/golang-announce
>>> https://blog.golang.org
>>> https://twitter.com/golang
>>> :
>>>
>>> On Mon, Sep 25, 2017 at 7:46 AM, Michael Jones 
>>> wrote:
>>>
 i suggest that you first make it work in the simple way and then make
 it concurrent.

 however, one lock-free concurrent way to think of this is as follows...

 1. start with a list of urls (in code, on command line, etc.)
 2. spawn a go process that writes each of them to a channel of strings,
 perhaps called PENDING
 3. spawn a go process that reads a url string from work and if it is
 not in the map of already processed url's, writes it to a channel of
 strings, WORK, after adding the url to the map.
 4 spawn a set of go processes that read WORK, fetch the url, do
 whatever it it that you need to do, and for urls found there, writes them
 to PENDING

 this is enough. now as written you have the challenge to know when the
 workers are done and pending is empty. that's when you exit. there are
 other ways to do this, but the point is to state with emphasis what an
 earlier email said, which is to have the map in its own goroutine, the one
 that decides which urls should be processed.

 On Mon, Sep 25, 2017 at 5:35 AM, Aaron  wrote:

> I have come up with a fix, using Mutex. But I am not sure how to do it
> with channels.
>
> package main
>
> import (
> "fmt"
> "log"
> "net/http"
> 

Re: [go-nuts] Unmarshal Json

[Richard Masci]

Thank you very much I think that's going to work!

On Tue, Sep 26, 2017 at 5:22 PM, Richard Masci  wrote:

> Hi Shawn,
>
> Thanks for that... the JSON I gave stripped out a lot from the real JSON
> as I didn't want the company information to be out on the public internet.
>   The JSON is from querying a server via rest call, and the IP Addresses
> are other servers that have connected in to this one server to pull
> information from. So the JSON is not a static list but one that is subject
> to change where new IP connect in, and old disconnect.
>
> On Tue, Sep 26, 2017 at 3:55 PM, Shawn Milochik 
> wrote:
>
>> https://play.golang.org/p/5itvZlhOVr
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "golang-nuts" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>> pic/golang-nuts/Pss2cEqaiYs/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Unmarshal Json

[Richard Masci]

Hi Shawn,

Thanks for that... the JSON I gave stripped out a lot from the real JSON as
I didn't want the company information to be out on the public internet.
The JSON is from querying a server via rest call, and the IP Addresses are
other servers that have connected in to this one server to pull information
from. So the JSON is not a static list but one that is subject to change
where new IP connect in, and old disconnect.

On Tue, Sep 26, 2017 at 3:55 PM, Shawn Milochik 
wrote:

> https://play.golang.org/p/5itvZlhOVr
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/golang-nuts/Pss2cEqaiYs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Serialized HTTP requests with asynchronous responses

[Donovan Hide]

The nonce is intended just to stop replay attacks. It doesn't have to
increment with a delta of 1, typically you just use time.UnixNano(). The
API is not under my control, I just want to minimise the occurrence of
nonces being received out of order by the API.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Unmarshal Json

[Shawn Milochik]

https://play.golang.org/p/5itvZlhOVr

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

SV: [go-nuts] Unmarshal Json

[Michael Banzon]

Can it be parsed to a map[string]json.RawMessage (or a more specific type, if 
the value is a known type)?

-- 
Michael Banzon
https://michaelbanzon.com/

Fra: Rich
Sendt: 26. september 2017 20:23
Til: golang-nuts
Emne: [go-nuts] Unmarshal Json

I have a json I have to parse, and in that json one of the keys changes 
depending on the data being generated.  For example:

{
  "Servers": {
    "IPAddress=10.1.1.123": {
      "name": "DNS Server1",
      "type": "Apache Webserver"
    },
    "IPAddress=10.1.1.124": {
      "name": "DNS Server",
      "type": "Bind DNS"
    },
    "IPAddress=10.2.3.43": {
      "name": "Tomcat Server",
      "type": "Java App Server"
    }
  }
}

I have no idea what IPAddress is going to equal, so I'm having a hard time 
writing a struct for it.  Any idea how I can parse this JSON?

Thanks!
-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Unmarshal Json

[Rich]

I have a json I have to parse, and in that json one of the keys changes 
depending on the data being generated.  For example:

{
  "Servers": {
"IPAddress=10.1.1.123": {
  "name": "DNS Server1",
  "type": "Apache Webserver"
},
"IPAddress=10.1.1.124": {
  "name": "DNS Server",
  "type": "Bind DNS"
},
"IPAddress=10.2.3.43": {
  "name": "Tomcat Server",
  "type": "Java App Server"
}
  }
}

I have no idea what IPAddress is going to equal, so I'm having a hard time 
writing a struct for it.  Any idea how I can parse this JSON?

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Scope/Context in struct and interfaces design questions

[Denys Koch]

Hi all!

I need an gopher advice. What is more idiomatic and do you see any problems 
with my solutions?

Problem: I have a repository(persistence layer) for TodoLists. TodoLists 
usually belongs to a user. My system also have super-user/admin interface 
which can access all TodoLists. I can imagine something like this. But I 
think it is maybe not good, because mixing not user-scoped API and user 
scoped API

type TodoListID int

type TodoList struct {}

type TodoListRepository interface {

  Add(t *TodoList) error

  Update(t *TodoList) error

  Remove(id TodoListID) error

  All() ([]*TodoList, error)

  Find(id TodoListID) (* TodoList, error)

   // user scoped API

   TodoListsForUserID(id UserID) ([]*TodoList, error)
   AddTodo(t *Todo, u UserID) 
   RemoveTodo(t *Todo, u UserID)   


}



So I got this idea: Separate the interfaces

type TodoListRepository interface {

  Add(t *TodoList) error

  Update(t *TodoList) error

  Remove(id TodoListID) error

  All() ([]*TodoList, error)

  Find(id TodoListID) (* TodoList, error)
}
type UserTodoListsRepository interface {

   All() ([]*TodoList, error)
   AddTodo(t *Todo) 
   RemoveTodo(t *Todo) 
  

}

So I would define a struct wich implements UserTodoListsRepository 
interface and one struct for not user-scoped TodoListsRepository. Do I miss 
something? Bad or good design? 

Thanks all in advance!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Serialized HTTP requests with asynchronous responses

[Diego Medina]

Hi,

Probably not the answer you were expecting, but :

> The nonces must be received by the host in ascending order.

is going to fail sooner or later, no matter which trick/feature you use. A 
more robust solution would be to code the server to accept unordered 
requests and handle them gracefully.

To give you a concrete example or where such assumption/requirement fails:

1. first requests goes out
2. server starts reading the request including the nonce
3. second request goes out with a new nonce
4. the network from req 1 fails so the server did not finish reading the 
complete payload
5. req from step 3 finishes

Now the server **doesn't** have req 1. The higher the concurrency, the 
higher the chances of this issue from happening and being harder to handle 
from the client side.

Or , would something like this https://github.com/savaki/snowflake help you 
by creating unique IDs with *some* ordering?

Final question, is the nonce used anywhere as a cryptographic IV/Salt/etc? 
If so, I wouldn't use an ID that increments, but instead use something 
really random, like crypto/rand


Regards,

Diego


On Tuesday, September 26, 2017 at 1:40:38 AM UTC-4, Donovan wrote:
>
> Hi,
>
> I have an interesting problem where I need to make a rapid series of HTTP 
> requests to the same host, each with a nonce which increments with each 
> request and a signature of the request body including that nonce, but I 
> don't want to wait for the response before sending the next request. The 
> nonces must be received by the host in ascending order.
>
> I've tried this using standard library HTTP requests, each running in 
> separate goroutines with a time delay before releasing the next nonce, to 
> try and ensure that the requests are sent in order, but this fails 
> intermittently, possibly due to network delays. I've also looked at 
> PipelineClient with MaxConns set to 1, which seems closer to what I need, 
> but it would still require an asynchronous response to prevent blocking 
> between request dispatches.
>
> https://godoc.org/github.com/valyala/fasthttp#PipelineClient
>
> Has anyone battled a problem similar to this before? Any ideas much 
> appreciated!
>
> Cheers,
> Donovan.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Simple web crawler question. How to avoid crawling visited links?

[Michael Jones]

To be sure, examples are a starting place.

To crawl to a certain depth, you need to know the depth of each url. That
means that the url being passed around stops being "just a string" and
becomes a struct with information including at least, the url as a string
and the link depth as an integer. Following a link means incrementing the
depth, checking that, and also checking the presence of the string part in
the map.

To crawl to a certain distance from a website, in the sense of how many
hops to other websites, needs similar processing on the hostname.

To crawl with rate-limitation so as not to overburden a distant host's
server(s), means to have multiple work queues for each hostname. That is
the natural place to do rate limits, time windows, or other such processing.

The real meaning of the example is to point out that the work queue -- a
channel here -- can be written to by the (go)routines that read from it.
This is a useful notion in this case.

Good luck!

On Mon, Sep 25, 2017 at 8:46 PM, Aaron  wrote:

> Thank you Mike. I have read the book and done that exercise. The code in
> the example will not crawl into certain depth and stop at certain depth of
> the website. The requirements are a bit different. I can't just use the
> approach in the example directly or more precisely I don't know how to use
> it in my code.
>
> On Tuesday, September 26, 2017 at 6:18:43 AM UTC+8, Michael Jones wrote:
>>
>> The book, The Go Programming Language discusses the web crawl task at
>> several points through the text. The simplest complete parallel version is:
>>
>> https://github.com/adonovan/gopl.io/blob/master/ch8/crawl3/findlinks.go
>>
>> which if you'll download and build works quite nicely:
>>
>> *$ crawl3 http://www.golang.org *
>> http://www.golang.org
>> http://www.google.com/intl/en/policies/privacy/
>> https://golang.org/doc/tos.html
>> https://golang.org/project/
>> https://golang.org/pkg/
>> https://golang.org/doc/
>> http://play.golang.org/
>> https://tour.golang.org/
>> https://golang.org/LICENSE
>> https://developers.google.com/site-policies#restrictions
>> https://golang.org/dl/
>> https://golang.org/blog/
>> https://golang.org/help/
>> https://golang.org/
>> https://blog.golang.org/
>> https://www.google.com/intl/en/privacy/privacy-policy.html
>> https://www.google.com/intl/en/policies/terms/
>> https://golang.org/LICENSE?m=text
>> https://golang.org/pkg
>> https://golang.org/doc/go_faq.html
>> https://groups.google.com/group/golang-nuts
>> https://blog.gopheracademy.com/gophers-slack-community/
>> https://golang.org/wiki
>> https://forum.golangbridge.org/
>> irc:irc.freenode.net/go-nuts
>> 2017/09/25 15:13:07 Get irc:irc.freenode.net/go-nuts: unsupported
>> protocol scheme "irc"
>> https://golang.org/doc/faq
>> https://groups.google.com/group/golang-announce
>> https://blog.golang.org
>> https://twitter.com/golang
>> :
>>
>> On Mon, Sep 25, 2017 at 7:46 AM, Michael Jones 
>> wrote:
>>
>>> i suggest that you first make it work in the simple way and then make it
>>> concurrent.
>>>
>>> however, one lock-free concurrent way to think of this is as follows...
>>>
>>> 1. start with a list of urls (in code, on command line, etc.)
>>> 2. spawn a go process that writes each of them to a channel of strings,
>>> perhaps called PENDING
>>> 3. spawn a go process that reads a url string from work and if it is not
>>> in the map of already processed url's, writes it to a channel of strings,
>>> WORK, after adding the url to the map.
>>> 4 spawn a set of go processes that read WORK, fetch the url, do whatever
>>> it it that you need to do, and for urls found there, writes them to PENDING
>>>
>>> this is enough. now as written you have the challenge to know when the
>>> workers are done and pending is empty. that's when you exit. there are
>>> other ways to do this, but the point is to state with emphasis what an
>>> earlier email said, which is to have the map in its own goroutine, the one
>>> that decides which urls should be processed.
>>>
>>> On Mon, Sep 25, 2017 at 5:35 AM, Aaron  wrote:
>>>
 I have come up with a fix, using Mutex. But I am not sure how to do it
 with channels.

 package main

 import (
 "fmt"
 "log"
 "net/http"
 "os"
 "strings"
 "sync"

 "golang.org/x/net/html"
 )

 var lock = sync.RWMutex{}

 func main() {
 if len(os.Args) != 2 {
 fmt.Println("Usage: crawl [URL].")
 }

 url := os.Args[1]
 if !strings.HasPrefix(url, "http://;) {
 url = "http://; + url
 }

 n := 0

 for link := range newCrawl(url, 1) {
 n++
 fmt.Println(link)
 }

 fmt.Printf("Total links: %d\n", n)
 }

 func newCrawl(url string, num int) chan string {
 visited := make(map[string]bool)

Re: [go-nuts] Best practices for internal repository management

[Russ Cox]

On Fri, Sep 22, 2017 at 1:08 PM,  wrote:

> Hi all,
>
> I have a question regarding modern best practices for go repositories. As
> we are building out services in go, we still have the opportunity now to
> refactor the service code pretty dynamically. So now is a good time to
> check and see if we're growing the repositories in a reasonable way. Our
> version management system is git (github).
>
> There are two major routes I can see going down with regards to our
> organization's go code.
>
>1. All internal code is in a single repository, and any time a
>developer wants to update an external dependency (ex: logrus or whatever),
>they are responsible for making sure the new version works on the rest of
>the code base by resolving function signature changes. This is closer to
>the mono-repo workflow.
>2. All internal code resides in repositories which are split up into
>much smaller components. Each service service would be its own repository
>with its own build definitions, and the different "common libraries" would
>have their own repositories also. Each service lazily and independently
>updates their dependencies only when a dependency update is needed. This is
>closer to how the common Maven/SBT (java / scala) based artifact system
>works.
>
> Does anyone in the go community have some good suggestions or pitfalls
> regarding the above approaches? Or is there a different way that is
> suggested.
>

Today I think the best approach is still to use a single repository as in
your (1), with a vendoring tool such as dep (or glide, as Steven Hartland
suggested) to make a local copy of the code.

Shawn Milochik pointed out that a monorepo causes extra up-front
coordination to prevent or resolve merge conflicts, but that work usually
has to happen one way or the other. In #2 it is just taken on as debt, and
worse as debt that may not be obvious at the time. Front-loading the
coordination can help avoid taking on too much of this debt.

In the longer term we hope to provide better tooling around all of this,
but it's not ready yet.

Russ

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go 1.9 is released

[Igor Maznitsa]

if you use mvn-golang plugin  then 
since 2.1.6 version it can process missing SDK in the list and since 2.1.7 
it will be processing the list correctly because google returns not full 
list for just request but need some prefix in request

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go 1.9 is released

[alskiontheweb]

Hello,

We use a maven plugin to build a complex suite of Go based services. That 
plugin looks to the XML index provided 
by https://storage.googleapis.com/golang/ which currently does not include 
the 1.9 binaries even though they are available. Can someone look into why 
the 1.9 revs aren't indexed?

Thanks.

On Thursday, August 24, 2017 at 6:44:25 PM UTC-4, Chris Broadfoot wrote:
>
> Hello gophers,
>
> We just released Go 1.9.
>
> You can read the announcement blog post here:
>   https://blog.golang.org/go1.9
>
> You can download binary and source distributions from our download page:
>   https://golang.org/dl/
>
> To compile from source using a Git checkout, update to the release with 
> "git checkout go1.9" and build as usual.
>
> To find out what has changed, read the release notes:
>   https://golang.org/doc/go1.9
>
> Thanks to everyone who contributed to the release.
>
> Chris
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Bytconv where art thou

[Peter Waller]

On 25 September 2017 at 17:43, Amnon Baron Cohen  wrote:

> https://github.com/golang/go/issues/2632
>

Nice link, fun to see that all this discussion has happened before (and no
doubt, will happen again!).

Reading that thread and related threads, I get the impression that there is
not an open issue covering string([]byte) where the []byte can be proven to
not be mutated. At least not one that is cross referenced (or I may have
missed it with my quick scanning).

Russ did comment (https://github.com/golang/go/
issues/2632#issuecomment-252725945) that he would prefer to see the
compiler improved, but I didn't find an open issue which covers that. Can
anyone else find one? I think there are some valuable insights in the
linked issue, though it is closed because it proposes API duplication
rather than an underlying compiler fix.

For fun I did an AST search of string(thing of type []byte) in the Go
corpus. It comes a fair amount. Not sure though how to evaluate the
performance impact of fixing it. Would be nice if it were a simple matter
of running everyone's code to find out ;-). Anecdotally it appears in some
hot loops for me, and maybe it's interesting that other people have
discussed it quite a bit (quoting Bradfitz Oct 2015: "4 years and still a
problem": https://github.com/golang/go/issues/2632#issuecomment-144591243)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.