Re: [go-nuts] Locking goroutine to "main" thread

[Alex Buchanan]

Yep. My understanding of locking the OS thread from an init() was wrong. I 
think I have it working now. Thanks guys!

On Tue, Dec 17, 2019, at 9:37 AM, Ian Davis wrote:
> 
> 
> On Sat, 14 Dec 2019, at 7:51 PM, bucha...@gmail.com wrote:
>> If I understand correctly, that would prevent me from starting any other 
>> goroutines. The following program deadlocks:
> 
> I doesn't prevent you starting other goroutines but you need to ensure that 
> all calls to the OpenGL context are performed on the thread you initialised 
> it with (my understanding is that this is because OpenGL uses thread local 
> storage). Typically you can do this by sending closures containing OpenGL 
> calls via a channel to an executor running in a goroutine locked to the main 
> thread. There is/was an example of this in the gomobile codebase. I can share 
> more detail if you get stuck.
> 
> -- Ian
> 
> 
> 

> --
>  You received this message because you are subscribed to a topic in the 
> Google Groups "golang-nuts" group.
>  To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/golang-nuts/zAB8Si7EMVM/unsubscribe.
>  To unsubscribe from this group and all its topics, send an email to 
> golang-nuts+unsubscr...@googlegroups.com.
>  To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/0adcdc5a-dd2c-4506-9110-a3ecabc508b9%40www.fastmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a90ad587-495c-4edf-bbc1-82491f2e6238%40www.fastmail.com.

[go-nuts] multiple vendored x/net causing trouble (x/net/trace panic)

[Alex Buchanan]

This panic has caused me trouble many times over the last year:

panic: http: multiple registrations for /debug/requests

goroutine 1 [running]:
net/http.(*ServeMux).Handle(0x5c47ba0, 0x511cbc5, 0xf, 0x5223160, 0x515f730)
 /usr/local/go/src/net/http/server.go:2353 +0x239
net/http.(*ServeMux).HandleFunc(0x5c47ba0, 0x511cbc5, 0xf, 0x515f730)
 /usr/local/go/src/net/http/server.go:2368 +0x55
net/http.HandleFunc(0x511cbc5, 0xf, 0x515f730)
 /usr/local/go/src/net/http/server.go:2380 +0x4b
golang.org/x/net/trace.init.0()
 /Users/buchanae/src/golang.org/x/net/trace/trace.go:115 +0x42

This happens because more than one package is vendoring golang.org/x/net 
and importing x/net/trace, which has a global handler registration in an 
init() function. This has been discussed briefly here 
https://github.com/golang/go/issues/20478 and 
here https://github.com/golang/go/issues/24137

Unfortunately, we're using docker as a library (import 
github.com/docker/docker/client) and docker also vendors golang.org/x/net. 
The same is true for github.com/golang/gddo/httputil. We need to vendor 
golang.org/x/net in our root application for other dependencies, to avoid 
breaking changes such 
as https://github.com/golang/net/commit/cbb82b59bc5199680c5bf34109e12097095dca9b

I'm stuck. I have a PR with a useful feature which is blocked until I can 
find a way around this. The new feature uses a package which imports 
golang.org/x/net/trace.

I've read that "dep" will flatten vendor trees and solve this issue. I 
haven't tried it. Will "vgo" solve this issue as well?

Can't we get rid of the global registrations in the trace package? I must 
have fought this issue 10+ times in the last year.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] [ANN] github release notes builder, based on PR history

[Alex Buchanan]

I created a simple utility to build release notes based on github pull 
request history, if anyone finds it 
useful: https://github.com/buchanae/github-release-notes

Example results: https://github.com/ohsu-comp-bio/funnel/releases/tag/0.6.0

Cheers.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: How to lowercase all field names for MarshalJSON and UnmarshalJSON?

[Alex Buchanan]

Thanks, that does look helpful, although I'll need to evaluate the 
tradeoffs of not using the std lib. It's hard to believe a library could be 
substantially faster than the stdlib without giving something up.


On Monday, March 5, 2018 at 9:52:04 PM UTC-8, Tamás Gulácsi wrote:
>
> You can try github.com/json-iterator/go, it allows several naming 
> strategies (
> https://github.com/json-iterator/go/blob/master/extra/naming_strategy.go#L9), 
> even custom ones.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: How to lowercase all field names for MarshalJSON and UnmarshalJSON?

[Alex Buchanan]

I'm bummed there still isn't a good solution for this. It seems like such 
an easy and extremely common thing. In a decade of webdev, I've possibly 
never seen a JSON API with Golang style UpperCamelCase keys.

Is there a solution to this that I'm missing? 

Here's one project I'm working on where dozens of types need to have 
JSON/YAML formats: https://godoc.org/github.com/buchanae/cwl

Every single struct field needs a really obvious duplication of the field 
name in the tags, just in camel case. There are 3-4 variable name styles 
that would cover a majority of the data out there: camelCase, snake_case, 
etc. (Maybe I should do a code crawling exercise and get some numbers on 
how often tags are an obvious conversion of the field name...)

My documentation and code would rejoice if encoding/json added a callback 
for formatting the key name.

Can we just add a call to a callback 
here? https://golang.org/src/encoding/json/encode.go#L1110





On Monday, July 7, 2014 at 12:22:08 AM UTC-7, Dave Cheney wrote:
>
>
>
> On Saturday, 5 July 2014 18:36:06 UTC+10, i...@bodokaiser.io wrote:
>>
>> Hello,
>>
>> I would like to have all field names of a struct lower cased by default. 
>> At the moment I define the lowercased field name as json tag:
>>
>> type FooBar struct {
>> Id string `json:"id"`
>> }
>>
>> However with bson-, validation-, database- and xml-tags the space is 
>> getting very very tight.
>>
>
> Why is space tight ? Do you need to use a smaller font ?
>
> Seriously for moment, tags are the _correct_ solution for this. Apart from 
> a potential aesthetic issue, is there a specific reason why you don't want 
> to use them ?
>  
>
>>
>> Is there some hack how to implement the Unmarshaler Interface with just 
>> lowercasing all field names before proceeding?
>>
>> Bo
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: unused import and variables, necessary at compiler level? Experience can be improved?

[Alex Buchanan]

Clever workarounds!

These solutions don't handle the case of commenting out a function call, 
which is a common need IMO. Whatever the workaround, it would still be 
easier if done by vet, right?

On Wednesday, January 31, 2018 at 6:46:40 PM UTC-8, Alex Buchanan wrote:
>
> Unused variables and imports are strictly disallowed by the Go compiler. 
> Most of the time, I enjoy this benefit. These days, I've started to notice 
> how often things go dead and/or unused in my python code.
>
> But, sometimes, this behavior is incredibly annoying. If I'm trying to 
> debug something by commenting out lines, or if I'm hacking up an 
> experimental script (meant to run with `go run`), I find this checking is 
> very unwanted. For example, if I comment out one variable, it may lead to a 
> new set of variables being unused, which sometimes leads back to a package 
> being unused. It's not uncommon for this cycle to take 3-5 iterations, in 
> my experience.
>
> Quickly hacking up Go scripts would be improved by relaxing this 
> constraint, in my opinion.
>
> Is finding a better compromise at all interesting or even possible? Are 
> parts of the compiler written to depend on the fact that variables are 
> definitely used?
>
> My idea for a potential improvement (again, having zero knowledge of the 
> compiler requirements) is to move this checking to vet.
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] unused import and variables, necessary at compiler level? Experience can be improved?

[Alex Buchanan]

Unused variables and imports are strictly disallowed by the Go compiler. 
Most of the time, I enjoy this benefit. These days, I've started to notice 
how often things go dead and/or unused in my python code.

But, sometimes, this behavior is incredibly annoying. If I'm trying to 
debug something by commenting out lines, or if I'm hacking up an 
experimental script (meant to run with `go run`), I find this checking is 
very unwanted. For example, if I comment out one variable, it may lead to a 
new set of variables being unused, which sometimes leads back to a package 
being unused. It's not uncommon for this cycle to take 3-5 iterations, in 
my experience.

Quickly hacking up Go scripts would be improved by relaxing this 
constraint, in my opinion.

Is finding a better compromise at all interesting or even possible? Are 
parts of the compiler written to depend on the fact that variables are 
definitely used?

My idea for a potential improvement (again, having zero knowledge of the 
compiler requirements) is to move this checking to vet.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Best way to add retries to existing library using net/http.Client?

[Alex Buchanan]

I'm considering adding http client retries to an existing library, owned by 
someone else, in order to add resiliency to errors such as http 503. I'd 
like to keep the changes as minimal as possible, with maximum effect, as 
usual. I'm eyeing http.Client.Transport. Possibly I could use that to wrap 
the DefaultTransport, intercept responses, check the code against some 
rules, and retry on failure.

Am I on the right track here?

Thanks,
Alex 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: How to download a compressed S3 object ? example ( abc.zip )

[Alex Buchanan]

Here's some example s3 client code we use in Funnel: 
https://github.com/ohsu-comp-bio/funnel/blob/master/storage/s3.go

And Amazon's S3 client docs: 
https://godoc.org/github.com/aws/aws-sdk-go/service/s3#hdr-Download_Manager

Or, if you meant non-Amazon s3, we have some experience with minio: 
https://docs.minio.io/docs/golang-client-api-reference

On Tuesday, November 21, 2017 at 4:05:51 PM UTC-8, himadhar reddy wrote:
>
> How to download a compressed S3 object to local using golang? example ( 
> abc.zip )
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: context.C and context.BG

[Alex Buchanan]

So, context.Bg could be a value then? context.TODO could stay a function.

On Saturday, November 18, 2017 at 6:01:07 AM UTC-8, Sameer Ajmani wrote:
>
> Context.TODO and Background have different String values; see 
> emptyCtx.String.
> On Fri, Nov 17, 2017 at 9:25 PM me via golang-nuts <
> golan...@googlegroups.com > wrote:
>
>> I've found a hint. Apparently, both context.TODO and context.Background 
>> share a common implementation but they need distinct addresses.
>>
>> See https://golang.org/src/context/context.go#L168
>>
>> So they cannot be constants because then they would not have an address. 
>> They cannot be variables because then you could modify them. Therefore the 
>> only solution left is to make them a function that returns a variable.
>>
>> Then again, maybe you could make Background and TODO of an unexported 
>> type to prevent setting them. I think that was not chosen because it leaks 
>> the internals of the package.
>>
>> But I'm not sure why they need distinct addresses in the first place. I'm 
>> guessing its for debugging or just what makes sense? context.TODO and 
>> context.Background are two different contexts and should be treated as such.
>>
>>
>> On Friday, November 17, 2017 at 10:43:29 AM UTC-5, Alex Buchanan wrote:
>>>
>>> I don't have numbers, but context must be one of the most commonly used 
>>> imports/types in my code. It shows up many function signatures. We try to 
>>> use it for all cancelation.
>>>
>>> Would it make sense to add an alias "C" to the type "Context"? And maybe 
>>> "BG" for Background()? 
>>>
>>> Also, is there a specific reason Background is a function and not a 
>>> value?
>>>
>>> Cheers,
>>> Alex
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: test debugging woes

[Alex Buchanan]

Some ideas:

- I assume the code isn't public? If it is, you might share it
- Are you running `go test -v` for verbose logging?
- Try fmt.Printf
- Try adding printing lines at points down the stack leading up to the 
point you think is failing, to make sure the code is actually running.
- Put a log line before each return in said function

On Thursday, November 16, 2017 at 10:54:53 PM UTC-8, Tom Denton wrote:
>
> Hello!
>
> I'm debugging a strange test failure, and with the magic of printf 
> debugging (t.Logf) have been adding bits to the test itself to get a sense 
> of what's wrong. The bad bahaviour is happening deep in library code, in a 
> function which returns nil for one of six reasons... I threw some 
> log.Fatalf() statements in that code in my citc to see what breaks, but 
> it's not showing up in the test logs at all.  Probably I'm making bad life 
> decisions; what, go community, am I supposed to be doing here?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] context.C and context.BG

[Alex Buchanan]

I don't have numbers, but context must be one of the most commonly used 
imports/types in my code. It shows up many function signatures. We try to 
use it for all cancelation.

Would it make sense to add an alias "C" to the type "Context"? And maybe 
"BG" for Background()? 

Also, is there a specific reason Background is a function and not a value?

Cheers,
Alex

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] import a "main" package

[Alex Buchanan]

Interesting, thanks. Let me try to clarify my idea with some simple code:

// file lives at github.com/buchanae/foobar/foobar.go
package foobar

func Foobar() string {
  return "foobar"
}

func main() {
  print(Foobar())
}

This file exports both a function to be used as a library, and a main function 
which would be compiled into a command line utility. In order to use the 
library, you'd write `import "github.com/buchanae/foobar"; foobar.Foobar()`. In 
order to use the command line tool, you'd run `go get 
github.com/buchanae/foobar` and run `foobar` on the command line. There's no 
`import "main"` or package named main in this example. The compiler would need 
to look for a main function in the starting package and perhaps name symbols 
using the fully qualified package name. Also, packages in the "foobar" tree 
would not be importing the root, so there are no import cycles.

I'm not sure it's a good idea, but it was my instinct when I started writing 
Go, and feels like it would have simplified the organization and documentation 
of some projects which have a separate "go get " for the library vs the 
CLI command.

Hope that made sense. Probably too late to be writing about such things :)



On 10/24/17, 9:31 PM, "Ian Lance Taylor" <i...@golang.org> wrote:

>On Tue, Oct 24, 2017 at 8:14 PM, Alex Buchanan <buchanae.o...@gmail.com> wrote:
>>
>> Is there documentation (or other) explaining why the main package must have
>> name "main" and may not be imported? Is this an important simplification for
>> the compiler?
>
>The language spec says that the main package must have the name main.
>I guess we could use other mechanisms but that one seems reasonable to
>me.
>
>cmd/compile prohibiting `import "main"` is a moderately important
>simplification because cmd/compile uses the import path to set symbol
>names, and for the main package it uses "main".  If cmd/compile
>permitted `import "main"` then there would be two different packages
>with the same import path, which can't work.  So it would have to use
>something else for the imported "main", but what?
>
>Note that cmd/compile does not prohibit importing a package named
>"main" if the path used to import that package is not specifically
>"main".  But I think the go tool does prohibit that, and I think that
>is simply to be less surprising to most people.
>
>Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Any way to somehow include readable source code into compiled binaries?

[Alex Buchanan]

But, maybe a better solution is to faithfully record the current git (or 
other VCS) details you'd need to find the code which resulted in the 
binary? That way you have access to the source in its natural state.

On Thursday, October 19, 2017 at 11:49:39 AM UTC-7, Alex Buchanan wrote:
>
> We use go-bindata to bundle text files (html, js, css, yaml) into our 
> binary. I bet you could use it to bundle Go source.
>
> https://github.com/jteeuwen/go-bindata
>
> -Alex
>
> On Thursday, October 19, 2017 at 9:46:37 AM UTC-7, Samuel Lampa wrote:
>>
>> Hi Gophers,
>>
>> *Question:* Is there any way to include a readable verison of the source 
>> code of my program into the compiled Go binary, either in plain text, or 
>> the ability extract it somehow, with some go tooling or 3rd party tools?
>>
>> *Background:* The reason for asking is that we are using Go to write 
>> scientific workflows with scipipe <http://scipipe.org>. This means we 
>> can compile our scientific workflows into static binaries for easier 
>> deployment and protecting against undocumented code changes that code make 
>> us loose track of the provenance ("complete track record") of the steps ... 
>> and potentially lead to flawed science ("shudder").
>>
>> But with compiled binaries, we instead run into another problem of 
>> provenance: Allowing ourselves and others to inspect that a particular 
>> workflow binary actually does what it says.
>>
>> Thus, having a statically compiled binary with the source code included 
>> in some form, would be the best of two worlds.
>>
>> Cheers
>> // Samuel
>> PhD Student @ pharmb.io
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Any way to somehow include readable source code into compiled binaries?

[Alex Buchanan]

We use go-bindata to bundle text files (html, js, css, yaml) into our 
binary. I bet you could use it to bundle Go source.

https://github.com/jteeuwen/go-bindata

-Alex

On Thursday, October 19, 2017 at 9:46:37 AM UTC-7, Samuel Lampa wrote:
>
> Hi Gophers,
>
> *Question:* Is there any way to include a readable verison of the source 
> code of my program into the compiled Go binary, either in plain text, or 
> the ability extract it somehow, with some go tooling or 3rd party tools?
>
> *Background:* The reason for asking is that we are using Go to write 
> scientific workflows with scipipe . This means we can 
> compile our scientific workflows into static binaries for easier deployment 
> and protecting against undocumented code changes that code make us loose 
> track of the provenance ("complete track record") of the steps ... and 
> potentially lead to flawed science ("shudder").
>
> But with compiled binaries, we instead run into another problem of 
> provenance: Allowing ourselves and others to inspect that a particular 
> workflow binary actually does what it says.
>
> Thus, having a statically compiled binary with the source code included in 
> some form, would be the best of two worlds.
>
> Cheers
> // Samuel
> PhD Student @ pharmb.io
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: log. Sprintf

[Alex Buchanan]

You mean fmt.Fprintf right? Formatting a string without writing to a 
io.Writer (fmt.Sprintf) must be safe for concurrent use.

The "log" package should be safe for concurrent use, and has a Printf 
function:
https://godoc.org/log#Logger

"""
A Logger represents an active logging object that generates lines of output 
to an io.Writer. Each logging operation makes a single call to the Writer's 
Write method. A Logger can be used simultaneously from multiple goroutines; 
it guarantees to serialize access to the Writer.
"""

On Wednesday, October 18, 2017 at 12:18:10 PM UTC-7, David Renne wrote:
>
> I had a question that I cant seem to figure out.  I know fmt is a bad 
> package to use if you have any concurrent workers because it uses f.Write() 
> and is not thread safe apparently.
>
> But I like using Sprintf because I can format strings and return as a 
> string.  How can I implement something to use the log package in a manner 
> that I can have a concurrently safe log function which acts like Sprintf()?
>
> It's just so strange to me and I have always been curious of why there 
> isnt a log.Sprintf().
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Task scheduler as a library?

[Alex Buchanan]

>
>
> Having tried using things like gleam/glow (see 
> https://github.com/chrislusf/gleam) and dask (which has similar ideas but 
> written in python) it sounds like a FIFO work stealing type scheduler 
> wouldn't be a bad idea. In particular dask's approach is quite nicely put 
> together. Though one thing that I have noticed with a lot of these non-HPC 
> type schedulers is that they are great for doing embarrassingly parallel 
> type problems but fall apart reasonably quickly once you get into MPI 
> territory where you want to be careful about how you layout the 
> processes/network for your job, probably things to consider when writing a 
> scheduler.
>

Ya, Funnel currently has a really simple scheduler along these lines, but 
the minute it had two users, we realized we wanted something smarter (fair 
scheduling). Hadn't considered MPI yet, as I'm more interested in large 
batch jobs, but I can see people wanting it, and data locality, and worker 
admin, and, and, and pretty soon you've just built Kubernetes :)

Many projects have done this though, so it seems like a well understood 
problem, so it seems like something that could stand on its own as a 
library, built in Go and wrapped by other languages via C bindings. 

>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: concurrent write-only to map ok?

[Alex Buchanan]

Good point, simple examples are almost never enough. I guess I was hoping 
that we'd end up with 3-5 examples of a really simple case. Maybe I'll come 
up with a more complex example.

I'm loading under ~5K files in parallel and possibly making an HTTP request 
for each. I like the slice solution, but I need the files to be accessible 
later by path (hence the map key). This is a command line utility. Adding 
parallelism already cut the time down from 1-2 minutes to 5 seconds, so I'm 
not concerned about map contention. So, my example does kinda capture the 
core algorithm, but it doesn't explain it very well.


On Monday, October 16, 2017 at 9:42:18 AM UTC-7, Bryan Mills wrote:
>
> On Sunday, October 15, 2017 at 1:45:06 PM UTC-4, Alex Buchanan wrote:
>>
>> Show me the code! :)
>>
>> Here's mine: https://play.golang.org/p/ZwAlu5VuYr
>>
>
> The problem that sync.Map is intended to address is cache contention.
> Unfortunately, it doesn't currently address that problem well for stores 
> of disjoint keys (see https://golang.org/issue/21035).
>
> That said, if you're spawning a goroutine per write, you've already got 
> enough contention (on the scheduler) that any other synchronization 
> overhead is unlikely to matter.
>
> As you say: show me the code! The example you gave is simple enough that 
> it's not even clear why you need a map — a slice or channel would suffice.
> (Personally, I'd be inclined to use a slice: 
> https://play.golang.org/p/jpS06KFNbv)
>
> So a more realistic use case would help: what do you want to do with this 
> map when you're done with it?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Task scheduler as a library?

[Alex Buchanan]

Not a spanner at all. 

I think the Task Execution Schemas (TES) [1], which Funnel is based on, is 
a reinvention of DRMAA using technologies such as HTTP, REST, JSON, 
Protobuf. It's a pretty simple API and message type (Task) for create, get, 
list, cancel. But, admittedly, I don't know enough about DRMAA. I get a bit 
overwhelmed by its documentation, to be honest.

Funnel is an implementation of the TES spec. We'd like to keep it versatile 
for the reason you mentioned; many solutions end up feeling heavyweight and 
you get pigeon holed. We also think a lot about workflows, and the status 
quo is similar there. Funnel already supports many environments and 
schedulers: GCE, AWS, HTCondor, SGE, etc, etc. We're talking about adding 
Kubernetes. We're always thinking of ways to make it easier and more 
flexible, hence the thoughts about how far we should take the scheduler. If 
you need to run 10K tasks in a new GCE project on preemptible machines, how 
easy can we make that?

I encourage you to take a look at Funnel, let me know what you think. If 
it's lacking something that you need, I'd be interested in hearing about it.

Anywho, I'll stop ranting now. Thanks for the feedback!

-Alex

[1] https://github.com/ga4gh/task-execution-schemas


On Sunday, October 15, 2017 at 12:00:03 PM UTC-7, Jimmy Tang wrote:
>
> Not to throw a spanner into the works, but we have a similar problem in my 
> work environment of needing a scheduler to schedule distributed jobs, one 
> problem of writing a *nice* one for a given language is that you end up 
> being pigeon holed into one solution. We've been looking at using drmaa as 
> a way of accessing different schedulers in a more platform and language 
> agnostic way. It may be worth your while to take a look at the golang 
> bindings for drmaa so you aren't left reinventing the wheel. Maybe creating 
> a dumb scheduler for drmaa library might be the way to go?
>
> On Wednesday, 11 October 2017 20:14:46 UTC+1, Alex Buchanan wrote:
>>
>> Hey all,
>>
>> In Funnel (a distributed task toolkit) we're sort of dancing around 
>> having a full-on scheduler. We have a scheduler that has grown from 
>> development util, to prototype, to something we actually use, but it's 
>> missing many of the features you'd want in production. Mostly we aim to 
>> delegate scheduling to another application (SGE, Slurm, AWS Batch, 
>> Kubernetes, etc), but having a built-in ability to schedule tasks without 
>> extra infrastructure is undeniably attractive.
>>
>> Writing a scheduler is one of those things people warn you away from 
>> though. I wish there was a solid library we could embed, but I haven't 
>> found anything.
>>
>> I wanted to get some opinions from this community. Do you know of any 
>> scheduling libraries? Do you think having scheduling built in is a good 
>> idea? A bad idea? Should we keep chipping away at it? Would people be 
>> interested in a standalone scheduling library, or is this problem 
>> inherently too complex to be adequately captured in library form?
>>
>> Thanks!
>>
>> Funnel: https://github.com/ohsu-comp-bio/funnel
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: concurrent write-only to map ok?

[Alex Buchanan]

Show me the code! :)

Here's mine: https://play.golang.org/p/ZwAlu5VuYr

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] ld warnings with Go 1.9

[Alex Buchanan]

I tried Go 1.9.1 today, but got strange warnings from ld. Asking here 
before filing an issue as a sanity check in case my system is just borked 
somehow.

ld: warning: object file 
(/var/folders/kw/m6dz8snx7h33ppbkbllnqt48bkhsqx/T/go-link-991847961/00.o) 
was built for newer OSX version (10.11) than being linked (10.10)
ld: warning: object file 
(/var/folders/kw/m6dz8snx7h33ppbkbllnqt48bkhsqx/T/go-link-991847961/01.o) 
was built for newer OSX version (10.11) than being linked (10.10)
ld: warning: object file 
(/var/folders/kw/m6dz8snx7h33ppbkbllnqt48bkhsqx/T/go-link-991847961/03.o) 
was built for newer OSX version (10.11) than being linked (10.10)
ld: warning: object file 
(/var/folders/kw/m6dz8snx7h33ppbkbllnqt48bkhsqx/T/go-link-991847961/04.o) 
was built for newer OSX version (10.11) than being linked (10.10)
ld: warning: object file 
(/var/folders/kw/m6dz8snx7h33ppbkbllnqt48bkhsqx/T/go-link-991847961/05.o) 
was built for newer OSX version (10.11) than being linked (10.10)
ld: warning: object file 
(/var/folders/kw/m6dz8snx7h33ppbkbllnqt48bkhsqx/T/go-link-991847961/06.o) 
was built for newer OSX version (10.11) than being linked (10.10)

I tried installing via both the tarball and the installer. I'm on OSX 
10.11.6. I also tried removing those objects/directories. Downgrading 
(back) to 1.8 made the warnings go away.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: concurrent write-only to map ok?

[Alex Buchanan]

So many good answers! Lots of different ways to accomplish this. I'll say 
that in this specific case, which is does not require every ounce of 
performance, syncmap is by far the simplest and least amount of code, so 
readability wins on this one.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] concurrent write-only to map ok?

[Alex Buchanan]

Thanks for the quick help guys!

I ended up using https://godoc.org/golang.org/x/sync/syncmap

On Friday, October 13, 2017 at 11:22:44 AM UTC-7, Shawn Milochik wrote:
>
> On Fri, Oct 13, 2017 at 2:05 PM, Alex Buchanan <buchan...@gmail.com 
> > wrote:
>
>> Basically, I want to spawn a goroutine per object, objects have unique 
>> IDs, and I want each routine to write its results to a shared map. Nothing 
>> will be reading from the map while the goroutines are running.
>>
>> Is this safe?
>>
>>
> Whether this is technically safe *today* in the reference implementation, 
> it's better to *be* safe.
>
> I suggest using a waitgroup and a channel to collect the results in an 
> explicitly safe way. Perhaps something like this: 
> https://play.golang.org/p/1NdQbAW65k
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] concurrent write-only to map ok?

[Alex Buchanan]

Basically, I want to spawn a goroutine per object, objects have unique IDs, 
and I want each routine to write its results to a shared map. Nothing will 
be reading from the map while the goroutines are running.

Is this safe?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: What is the best way to represent functional options within factory methods?

[Alex Buchanan]

I'm not sure I understand the example, but have you seen the grpc library's 
use of options? For more type safety, you could define an interface such as 
type PluginOpt interface { pluginOpt() }.

https://godoc.org/google.golang.org/grpc#CallOption

On Wednesday, October 11, 2017 at 11:23:16 AM UTC-7, Frank Ruiz wrote:
>
> Greetings,
>
> Was hoping to solicit some feedback on utilizing functional options in 
> conjunction with factory methods.
>
> I currently have the following function:
>
> func ProcPlugin(opts …func(*procStruct){
> p := defaulProc
> for _, opt := range opts {
> opt()
> }
> return , nil
> }
>
> I am now trying to generalize things (as I will be creating more factory 
> functions), and I'm trying to determine if this is the most optimal way to 
> represent the logic above.
>
> type PluginFactory func(opts ...interface{}) (Plugin, error)
>
> func NewProcPlugin(opts ...interface{}) (Plugin, error) {
> p := defaulProc
> for _, opt := range opts {
> opt()
> }
> return , nil
> }
>
> Any feedback would be much appreciated. Please let me know if more context 
> is required.
>
> Thank you!
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Task scheduler as a library?

[Alex Buchanan]

Hey all,

In Funnel (a distributed task toolkit) we're sort of dancing around having 
a full-on scheduler. We have a scheduler that has grown from development 
util, to prototype, to something we actually use, but it's missing many of 
the features you'd want in production. Mostly we aim to delegate scheduling 
to another application (SGE, Slurm, AWS Batch, Kubernetes, etc), but having 
a built-in ability to schedule tasks without extra infrastructure is 
undeniably attractive.

Writing a scheduler is one of those things people warn you away from 
though. I wish there was a solid library we could embed, but I haven't 
found anything.

I wanted to get some opinions from this community. Do you know of any 
scheduling libraries? Do you think having scheduling built in is a good 
idea? A bad idea? Should we keep chipping away at it? Would people be 
interested in a standalone scheduling library, or is this problem 
inherently too complex to be adequately captured in library form?

Thanks!

Funnel: https://github.com/ohsu-comp-bio/funnel

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] [ANN] roger, an app configuration library experiment

[Alex Buchanan]

My weekend project for that past couple weeks has been 
roger: https://github.com/buchanae/roger

This is another take on application configuration management: flags, env 
vars, YAML, etc.

The core ideas are:
- define config as structs
- defaults are defined by functions which return instances of structs
- static code analysis and generation provides some extra metadata, pulling 
docs from code comments on struct fields
- able to dump YAML with docs and exclude default/empty values

Other than that, it's probably just a more buggy version of existing 
libraries (such as viper) with less features. I'll probably keep chipping 
away at it though.

It also:
- generates flags
- loads values from environment variables
- loads values from a YAML file
- coerces time.Duration and github.com/alecthomas/units and others 
via github.com/spf13/cast

More in the README

Thanks!
Alex

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] os/exec stdout bytes.Buffer and timeout behavior?

[Alex Buchanan]

Ok, that's a great explanation, thanks. Do you have a recommendation for how to 
exit early? Would StdoutPipe() help? I tried setting Setpgid to true, but 
probably os/exec doesn't pay attention to process groups.

Alternatively, I can configure my shell scripts to close stdout/err on exit? 
Not pretty, but might work.

My best approach so far is to use time.After and panic, instead of context. 
This code is being used in a test, so panic is somewhat acceptable.

Thanks again.




On 9/29/17, 1:26 PM, "Ian Lance Taylor" <i...@golang.org> wrote:

>On Fri, Sep 29, 2017 at 12:33 PM, Alex Buchanan <buchanae.o...@gmail.com> 
>wrote:
>>
>> package main
>>
>> import (
>> "bytes"
>>   "context"
>>   "log"
>>   "os/exec"
>>   "time"
>> )
>>
>> func main() {
>> var stdout bytes.Buffer
>>
>>   ctx, cancel := context.WithTimeout(context.Background(), time.Second)
>>   defer cancel()
>>
>>   cmd := exec.CommandContext(ctx, "/bin/sh", "-c", "sleep 10; echo foo")
>>   cmd.Stdout = 
>>
>>   err := cmd.Run()
>>   log.Printf("%s, %s, '%s'", err, ctx.Err(), stdout.String())
>> }
>>
>>
>> This runs for 10 seconds, but I was expecting 1 second. Can someone help me
>> understand what is happening? I think it has something to do with the stdout
>> bytes.Buffer?
>
>Yes.  The problem is that when the context passed to CommandContext
>expires, it kills the process started by the command, but does not
>kill any subprocesses that that command may have started.  So when the
>context expires the /bin/sh is killed, but the sleep subprocess is
>still running.  Because you use a bytes.Buffer, the program has
>created a pipe to capture the standard output of the command.  After
>the process dies, cmd.Run is waiting for that pipe to be closed to
>make sure that it gathers all the data.  But the pipe is held open by
>the sleep subprocess.  It is only after that subprocess exits that the
>pipe is closed and your program continues.
>
>Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] os/exec stdout bytes.Buffer and timeout behavior?

[Alex Buchanan]

package main

import (
"bytes"
  "context"
  "log"
  "os/exec"
  "time"
)

func main() {
var stdout bytes.Buffer

  ctx, cancel := context.WithTimeout(context.Background(), time.Second)
  defer cancel()

  cmd := exec.CommandContext(ctx, "/bin/sh", "-c", "sleep 10; echo foo")
  cmd.Stdout = 

  err := cmd.Run()
  log.Printf("%s, %s, '%s'", err, ctx.Err(), stdout.String())
}


This runs for 10 seconds, but I was expecting 1 second. Can someone help me 
understand what is happening? I think it has something to do with the 
stdout bytes.Buffer?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Docs for vendored packages?

[Alex Buchanan]

Does anyone have a strategy for accessing the docs (godoc) of vendored 
dependencies?

I was a little surprised to find that godoc decided not to show vendored 
packages, here:
https://github.com/golang/go/issues/13929

This also applies more generally than vendored deps. In the past, I have 
found myself wishing I had easy access to docs of older go versions.

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.