Re: [go-nuts] detecting cyclic references

[twp...@gmail.com]

> My use case is that I want to pretty print in-memory objects for debug 
purposes during testing, and one of the features would be to point out such 
cycles as this.

Consider using an existing library for this, for example:
  https://github.com/sanity-io/litter
  https://github.com/davecgh/go-spew

On Wednesday, November 11, 2020 at 1:59:18 PM UTC+1 arpad@gmail.com 
wrote:

> `l[0] = l` would be fine for me, indeed. Though I am not sure I understand 
> the suggested solution. Notice that the type of the slice is 
> `[]interface{}`. This...
>
> ll := l[0].([]interface{})
> println(&l == &ll)
>
> ...would print `false`.
>
> I think the main challenge is that l and ll, or l[0] for that matter, are 
> not the same values. I tried with reflection, too, but couldn't figure a 
> way.
>
> PS: I am quite aware how unearthy this problem is, and how it should not 
> ever happen in case of actual programs. My use case is that I want to 
> pretty print in-memory objects for debug purposes during testing, and one 
> of the features would be to point out such cycles as this.
>
>
> On Wednesday, November 11, 2020 at 8:13:55 AM UTC+1 
> axel.wa...@googlemail.com wrote:
>
>> The question was about detecting cyclic references. For that, the `l[0] = 
>> l` case is enough.
>>
>> On Wed, Nov 11, 2020 at 2:31 AM jake...@gmail.com  
>> wrote:
>>
>>> FYI, that does detect the simple case of l[0] = l, but not more 
>>> complicated circular cases like l[1] = l[1:]
>>>
>>> On Tuesday, November 10, 2020 at 2:38:26 AM UTC-5 
>>> axel.wa...@googlemail.com wrote:
>>>
 If the slice is empty, it doesn't reference anything.
 If it is not empty, &x[0] can be used to identify the slice 
 (potentially also using len/cap, if it's interesting).

 On Tue, Nov 10, 2020 at 4:11 AM arpad ryszka  
 wrote:

> Hi,
>
> is there a way to detect the cyclic reference in the following example 
> somehow? Either via reflection or by other means? My understanding is 
> that 
> slices cannot be compared with == (unless to nil), or used as keys in 
> maps. 
> Is there a different way?
>
> l := []interface{}{"foo"}
> l[0] = l
> fmt.Println(l)
>
> or here it is as a playground link: 
> https://play.golang.org/p/T0qZlF8m-vi
>
> Cheers,
> Arpad
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/b2cb6b5e-febc-407f-b5b3-d9ca196ce68bn%40googlegroups.com
>  
> 
> .
>
 -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to golang-nuts...@googlegroups.com.
>>>
>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/250107e9-f688-4205-ae52-728221eb2e4cn%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/cc25d4f0-51fa-4851-989b-fab4527bc709n%40googlegroups.com.

[go-nuts] [ANN] [Geospatial] New set of Go bindings for GEOS

[twp...@gmail.com]

GEOS is the standard geometry library that underpins PostGIS and much of 
the open source geospatial world.

https://github.com/twpayne/go-geos is a new set of Go bindings for GEOS 
that make it easy to use GEOS geometries in your Go applications. Features 
include:

   - Fluent Go API.
   - Low-level Context, CoordSeq, Geom, and PrepGeom types provide access 
   to all GEOS methods.
   - High-level geometry.Geometry type implements all GEOS functionality 
   and many standard Go interfaces, 
   including database/sql/driver.Valuer and database/sql.Scanner (WKB) for 
   PostGIS database integration 
   and encoding/json.Marshaler and encoding/json.Unmarshaler (GeoJSON). See 
   the PostGIS example 
    for a 
   demonstration of the use of these interfaces.
   - Concurrency-safe. go-geos uses GEOS's threadsafe *_r functions under 
   the hood, with locking to ensure safety, even when used across multiple 
   goroutines. For best performance, use one geos.Context per goroutine.
   - Caching of geometry properties to avoid cgo overhead.
   - Optimized GeoJSON encoder.
   - Automatic finalization of GEOS objects.

Check it out at https://github.com/twpayne/go-geos.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4d74baa9-2e56-43b8-a0e3-0ffcec4f34e1n%40googlegroups.com.

Re: [go-nuts] Getting cross-package test coverage data?

[twp...@gmail.com]

Very, very, late response to this, but I was missing the `-coverpkg=./...` 
argument to `go test`. /facepalm

On Thursday, May 14, 2020 at 10:46:43 AM UTC+2 ppi...@gmail.com wrote:

> Not sure if a tool is required. 
> It can be done with a couple of commands. 
> An elaborate example (of coverage across different types of tests, etc) is 
> provided at: 
> https://www.elastic.co/blog/code-coverage-for-your-golang-system-tests
>
> On Thu, May 14, 2020 at 5:50 AM Tom Payne  wrote:
>
>> What's the current state of getting combined cross-package test coverage 
>> data? Specifically:
>>
>>- I have a lib directory containing my library code, which has some 
>>tests, but not many.
>>- I have a cmd directory containing my main code and many tests that 
>>exercises the code in lib .
>>- I want to know what code in lib is covered by my tests in cmd (and, 
>>more specifically, which code in lib is *not* covered by the tests so 
>>I can add more tests).
>>
>> My current understanding is I need to use a third-party tool to merge 
>> coverage, like https://github.com/ory/go-acc.
>>
>> Is this correct?
>>
>> Many thanks,
>> Tom
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/3e24588a-005e-4072-80fe-ff7a912bd368%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Pankaj Pipada
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8774259b-71d3-40d7-88f5-9b510bb12ff6n%40googlegroups.com.

Re: [go-nuts] Performance of syscall calls vs. CGO

[twp...@gmail.com]

Two tips from writing github.com/twpayne/go-geos (Go bindings for the 
popular GEOS geometry library), where I also encountered performance 
problems with cgo when calling small C functions:

1. Make each cgo call do more by moving common combinations of functions in 
to C, i.e. use a higher-level API.

Example: this function 
 moves a for 
loop over an element lookup function from Go to C. In this particular case, 
this saves hundreds or thousands of cgo calls.

2. Eliminate cgo calls where possible.

Example: this function 
 copies 
commonly-used data from C into a Go struct (i.e. caches them in Go) so cgo 
calls are not required are not required to retrieve those properties.

Regards,
Tom

On Friday, January 21, 2022 at 11:06:12 AM UTC+1 Sean wrote:

> Hello Ian,
>
> Thank you for the answer.
>
> I'm currently using Cgo heavily for a game. But microseconds don't 
> matter to me. The calls must not exceed milliseconds, and it all depends 
> on the C api I wrote.
> I'm currently in testing and development, there doesn't seem to be a 
> problem now but I don't know what to do if Cgo in production will be a 
> problem for me.
> Dropt this wonderful std and switching to C++ will be a nightmare for me.
>
> I'm trying some untested things in the Golang world.
>
> Hopefully, CFFI performance will eventually get on the Golang team's 
> radar and improve.
>
> On 20/01/2022 23:25, Ian Lance Taylor wrote:
> > On Thu, Jan 20, 2022 at 2:54 AM Sean  wrote:
> >> I know CGO is not performing well in Golang at the moment.
> > That is true, but we should quantify it. The last time I tested it, a
> > cgo call took about 10 times as long as an ordinary function call. So
> > that is pretty bad if you are calling a tiny function, but it doesn't
> > really matter if you are calling a function that does I/O.
> >
> >> If we use a dll will this performance issue decrease?
> > I don't know but I don't think so.
> >
> >> If I do the cgo calls with syscall, will there be no performance 
> improvements?
> > I haven't measured but I don't see why using the syscall package would
> > be any faster.
> >
> >> When I think about it, Golang always has to make syscall calls on the 
> os it's running on. For example, in Windows, the net package has to talk to 
> the socket api of Windows.
> >> My assumptions are that syscall should be better than Cgo.
> >> If syscall calls are no different from Cgo, how does Golang build this 
> performance in the core library?
> > The Go runtime has two advantages. First, it can make the call
> > directly in the system ABI rather than having to translate from the Go
> > ABI to the system ABI. Second, it can know that certain system calls
> > can never block, and can use that to optimize the way that they are
> > handled.
> >
> > Also, of course, system calls by their nature tend to be a bit slow,
> > so the overhead is less important. As mentioned above, the overhead
> > of a cgo call is most important when calling a small C function. Most
> > system calls are not small.
> >
> > Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/dceda1be-e47a-4da2-b80c-a6fcefe7a769n%40googlegroups.com.

[go-nuts] Re: BPF target

[twp...@gmail.com]

https://tinygo.org/ might be a more suitable initial project for compiling 
Go to eBPF. A lot is already in place including an LLVM backend and a 
simplified Go runtime suitable for small systems.

On Thursday, February 24, 2022 at 6:41:00 PM UTC+1 stalke...@protonmail.ch 
wrote:

> With the Solana project running with the old and perfectly adequate BPF 
> bytecode and execution engines already inside the linux kernel, it seems 
> that this might be a good target to add to Go's existing targets. 
>
> I mean, yes, I am aware of the severe limitations, no loops, etc, but to 
> be able to deploy Solana apps end to end with a Go backed frontend and Go 
> language for the programs... 
>
> I'm not expecting a strong enthusiastic response about the subject, of 
> course, but I think there could be a big bonus for Go adoption if it became 
> an option for this very interesting, very fast blockchain platform.
>
> It does not have to even permit most of the basics, maps, slices, they 
> just simply don't need to be used. It might also go along with efforts  
> such as tinygo, rather than on the mainline project. 
>
> As I spent a long time developing my skills with Go towards "Layer 1" 
> blockchain systems, including also some time inside a Geth fork, I feel 
> like Go is slipping a little with keeping relevant to this very active area 
> of software development.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/77d61e0b-e53e-4c02-9417-bafbd0676b76n%40googlegroups.com.

Re: [go-nuts] Improving on unit test styles

[twp...@gmail.com]

To identify failing test cases in a table of tests where you don't have 
individual names you can use its index:

for i, testCase := range testCases {
t.Run(strconv.Itoa(i), func(t *testing.T) {
// ...

For debugging an individual test case, just skip over all but the failing 
test case:

for i, testCase := range testCases {
if i != 5 { // 5 is the index of the failing test, remove if 
statement before committing
continue
}
t.Run(strconv.Itoa(i), func(t *testing.T) {
// ...

It's a quick cheap hack, but occasionally useful.

On Monday, February 28, 2022 at 9:48:52 PM UTC+1 rog wrote:

>
>
> On Fri, 25 Feb 2022, 18:46 'Markus Zimmermann' via golang-nuts, <
> golan...@googlegroups.com> wrote:
>
>> Hi Gophers!
>>
>> We were unhappy with the common unit test styles and we think we found a 
>> style that has clear advantages. An in-depth comparison can be found here 
>> https://symflower.com/en/company/blog/2022/better-table-driven-testing.
>>
>> We also added support for maintaining such tests in our VS Code extension 
>> https://marketplace.visualstudio.com/items?itemName=symflower.symflower. 
>> You can either use the command or the context menu item to maintain tests. 
>> Here is a short introduction video 
>> https://www.youtube.com/watch?v=mgWLY9DDyDE&ab_channel=Symflower
>>
>> There are some changes necessary to have better stack traces for 
>> github.com/stretchr/testify because "t.Run" calls the test function from 
>> another location. We are in the process of upstreaming them. Until then you 
>> can find them in our fork at https://github.com/symflower/testify.
>>
>> Would appreciate your feedback on the style and extension. Would be also 
>> interesting to hear other approaches and conventions that could help others 
>> to write better tests.
>>
>> Cheers,
>> Markus
>>
>
> One disadvantage of your approach: the table can't be reused for different 
> tests. Not uncommonly I've found that it's useful to be able to plug the 
> same table into more than one testing function - after all, the test data 
> is an abstract description of some property and there can be more than one 
> way of testing that property.
>
> I usually do something half way between the "standard" approach and your 
> suggestion. I have a struct with a "testName" field ("name" is too easily 
> confused with input data IMHO), and I'll use T.Run to run it as a subtest.
>
> I've found that the lack of stack trace is much less of a problem if you 
> use a test name that isn't mangled by the testing package - in other words: 
> don't use spaces! That way a simple text search in your editor will usually 
> take you straight to the relevant table entry.
>
>   cheers,
> rog.
>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/4c52a6b6-6762-442c-b9b8-82b8f50a732dn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b2476893-2744-48e3-a990-a8f721c77ef3n%40googlegroups.com.

[go-nuts] [ANN] go-xmlstruct: generate Go structs from multiple XML documents

[twp...@gmail.com]

There are a few existing XML document-to-Go struct generators, but none of 
them had all of the following features:

* Taking multiple XML documents as input.
* Generating correct field types.
* Customizable field names.
* Handling optional and repeated elements.
* Ignoring whitespace.

If you need to handle XML documents in Go then check out 
https://github.com/twpayne/go-xmlstruct.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6d3544b0-1d83-4cb5-9b03-b1ef03aeac7en%40googlegroups.com.

[go-nuts] Does Go optimize math.Float64frombits(binary.LittleEndian.Uint64[...])) to a single instruction?

[twp...@gmail.com]

tl;dr: on 64-bit little-endian machines, does

x := math.Float64frombits(binary.LittleEndian.Uint64(byteSlice[:8]))

get optimized to a single move instruction from byteSlice[:8] to x?


Background info:

I'm currently writing a bunch of code that unmarshals binary data. A small 
example is this that unmarshals eight float64s 
,
 
and a longer example is this that unmarshals an arbitrary number of float64s 
.
 
In theory, when the endianness of the binary data matches the endianness of 
the architecture, these should become simple memory copies. Does Go 1.19 do 
this?

Although this is a performance question, performance is not a concern in my 
use case. This is just an idle "can Go do this yet?" question and I'm 
curious about the answer.


Thanks for any insight,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e60948d1-8907-47ee-bdc5-1f344a21b5e2n%40googlegroups.com.

Re: [go-nuts] go tool dist: FAILED: not a Git repo; must put a VERSION file in $GOROOT

[twp...@gmail.com]

For info, this is broken in Fedora 39, released a day or two ago:

$ docker pull fedora:latest
$ docker run -it fedora:latest /bin/bash
[root@01330dfaac82 /]# dnf update && dnf install -y golang
...
[root@01330dfaac82 /]# go version
go version go1.21.3 linux/amd64
[root@01330dfaac82 /]# go tool dist list
go tool dist: FAILED: not a Git repo; must put a VERSION file in $GOROOT

On Saturday, August 27, 2022 at 9:21:15 PM UTC+2 Ian Lance Taylor wrote:

> On Sat, Aug 27, 2022 at 12:06 PM Lonnie TC  wrote:
> >
> > I hope that everyone is doing well today.
> > I am working on a Golang project to port go-pmem over to MSYS2 using 
> MinGW 64:
> >
> > https://github.com/jerrinsg/go-pmem
> >
> > In trying to build the patched Go distro toolchain, I am getting some 
> strange error in the build that I do not know how to handle and am hoping 
> that someone can answer this for me.
> >
> > go tool dist: FAILED: not a Git repo; must put a VERSION file in $GOROOT
> >
> > any help would be truly appreciate.
>
> I don't know what go-pmem is, but a Go distribution should have a file
> $GOROOT/VERSION that contains the version number of the release. If
> you download the sources of, say, the Go 1.19 release, you will see a
> VERSION file. For development purposes, if that VERSION file does not
> exist, the Go build will pull the version from the git repo. You seem
> to have Go sources that are not in a git repo and also do not have a
> VERSION file. That setup is not expected or supported. The simple
> fix is to create your own VERSION file.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6bcfe005-1b23-48a1-ae9d-9859943a3e9dn%40googlegroups.com.

Re: [go-nuts] go tool dist: FAILED: not a Git repo; must put a VERSION file in $GOROOT

[twp...@gmail.com]

Reported to Fedora at https://bugzilla.redhat.com/show_bug.cgi?id=2248782

On Thursday, November 9, 2023 at 1:00:55 AM UTC+1 twp...@gmail.com wrote:

> For info, this is broken in Fedora 39, released a day or two ago:
>
> $ docker pull fedora:latest
> $ docker run -it fedora:latest /bin/bash
> [root@01330dfaac82 /]# dnf update && dnf install -y golang
> ...
> [root@01330dfaac82 /]# go version
> go version go1.21.3 linux/amd64
> [root@01330dfaac82 /]# go tool dist list
>
> go tool dist: FAILED: not a Git repo; must put a VERSION file in $GOROOT
>
> On Saturday, August 27, 2022 at 9:21:15 PM UTC+2 Ian Lance Taylor wrote:
>
>> On Sat, Aug 27, 2022 at 12:06 PM Lonnie TC  
>> wrote: 
>> > 
>> > I hope that everyone is doing well today. 
>> > I am working on a Golang project to port go-pmem over to MSYS2 using 
>> MinGW 64: 
>> > 
>> > https://github.com/jerrinsg/go-pmem 
>> > 
>> > In trying to build the patched Go distro toolchain, I am getting some 
>> strange error in the build that I do not know how to handle and am hoping 
>> that someone can answer this for me. 
>> > 
>> > go tool dist: FAILED: not a Git repo; must put a VERSION file in 
>> $GOROOT 
>> > 
>> > any help would be truly appreciate. 
>>
>> I don't know what go-pmem is, but a Go distribution should have a file 
>> $GOROOT/VERSION that contains the version number of the release. If 
>> you download the sources of, say, the Go 1.19 release, you will see a 
>> VERSION file. For development purposes, if that VERSION file does not 
>> exist, the Go build will pull the version from the git repo. You seem 
>> to have Go sources that are not in a git repo and also do not have a 
>> VERSION file. That setup is not expected or supported. The simple 
>> fix is to create your own VERSION file. 
>>
>> Ian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e19c7e0e-11cd-4f1c-9856-cd0645b54697n%40googlegroups.com.

[go-nuts] rsc.io/script: How to contribute?

[twp...@gmail.com]

Internally, Go has a very nice package for integration testing 
.

Roger Peppe  has been doing a fantastic 
service by making it public and maintaining it as a community project at 
github.com/rogpeppe/go-internal/testscript 
, and 
Roger's package has been extremely valuable for my project 
 and many others.

Over a quarter of a year ago, Russ Cox  made a 
semi-official 
release of the code  which includes 
several improvements (e.g. better modularity), but also removing critical 
functionality  required to make 
rsc.io/script usable by other projects.

Appeals to the author about how to contribute 
 have been met with silence. 
Fundamental issues like installation failures 
 have had no response.

Does rsc.io/switch have a future? If so, how can we contribute? This is a 
fantastically useful piece of software for the Go community and we will all 
be better off if it succeeds.

Regards,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/45323826-e1b8-4cc0-95ac-37decea58738n%40googlegroups.com.

[go-nuts] Re: rsc.io/script: How to contribute?

[twp...@gmail.com]

*rsc.io/script

On Wednesday, March 6, 2024 at 10:46:23 PM UTC+1 twp...@gmail.com wrote:

> Internally, Go has a very nice package for integration testing 
> <https://github.com/golang/go/tree/master/src/cmd/go/internal/script>.
>
> Roger Peppe <https://github.com/rogpeppe> has been doing a fantastic 
> service by making it public and maintaining it as a community project at 
> github.com/rogpeppe/go-internal/testscript 
> <https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript>, and 
> Roger's package has been extremely valuable for my project 
> <https://github.com/twpayne/chezmoi> and many others.
>
> Over a quarter of a year ago, Russ Cox <https://research.swtch.com/> made 
> a semi-official release of the code <https://pkg.go.dev/rsc.io/script> which 
> includes several improvements (e.g. better modularity), but also removing 
> critical 
> functionality <https://github.com/rsc/script/issues/5> required to make 
> rsc.io/script usable by other projects.
>
> Appeals to the author about how to contribute 
> <https://github.com/rsc/script/issues/4> have been met with silence. 
> Fundamental issues like installation failures 
> <https://github.com/rsc/script/issues/3> have had no response.
>
> Does rsc.io/switch have a future? If so, how can we contribute? This is a 
> fantastically useful piece of software for the Go community and we will all 
> be better off if it succeeds.
>
> Regards,
> Tom
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b1aa4202-8a2f-448e-b94f-4fc45f0ab401n%40googlegroups.com.

[go-nuts] Re: rsc.io/script: How to contribute?

[twp...@gmail.com]

Thanks, but I'm already heavily invested in testscript with over 10K lines 
of scripts and a lot of custom configuration, and commander-cli/commander 
clearly lacks functionality I need.

On Thursday, March 7, 2024 at 2:47:03 AM UTC+1 Ajay Kidave wrote:

> I have had a good experience using 
> https://github.com/commander-cli/commander for testing CLI commands. It 
> uses a yaml file for test config. I generally avoid yaml but the commander 
> config format is easy to work with. Never used script, the state saving 
> between commands feature might be missing in commander-cli.
>
> On Wednesday, March 6, 2024 at 1:52:57 PM UTC-8 twp...@gmail.com wrote:
>
>> *rsc.io/script
>>
>> On Wednesday, March 6, 2024 at 10:46:23 PM UTC+1 twp...@gmail.com wrote:
>>
>>> Internally, Go has a very nice package for integration testing 
>>> <https://github.com/golang/go/tree/master/src/cmd/go/internal/script>.
>>>
>>> Roger Peppe <https://github.com/rogpeppe> has been doing a fantastic 
>>> service by making it public and maintaining it as a community project at 
>>> github.com/rogpeppe/go-internal/testscript 
>>> <https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript>, and 
>>> Roger's package has been extremely valuable for my project 
>>> <https://github.com/twpayne/chezmoi> and many others.
>>>
>>> Over a quarter of a year ago, Russ Cox <https://research.swtch.com/> 
>>> made a semi-official release of the code 
>>> <https://pkg.go.dev/rsc.io/script> which includes several improvements 
>>> (e.g. better modularity), but also removing critical functionality 
>>> <https://github.com/rsc/script/issues/5> required to make rsc.io/script 
>>> usable by other projects.
>>>
>>> Appeals to the author about how to contribute 
>>> <https://github.com/rsc/script/issues/4> have been met with silence. 
>>> Fundamental issues like installation failures 
>>> <https://github.com/rsc/script/issues/3> have had no response.
>>>
>>> Does rsc.io/switch have a future? If so, how can we contribute? This is 
>>> a fantastically useful piece of software for the Go community and we will 
>>> all be better off if it succeeds.
>>>
>>> Regards,
>>> Tom
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f6f92dd8-9033-43a0-9348-02c039c7fe90n%40googlegroups.com.

Re: [go-nuts] Re: rsc.io/script: How to contribute?

[twp...@gmail.com]

github.com/rogpeppe/go-internal/testscript is absolutely wonderful. I've 
been using it successfully for several years and am the #4 contributor to 
the project. My only complaint would be that it sometimes takes a few 
months for changes to be merged and the release cycle is infrequent, 
meaning that I have to maintain my own workarounds for missing 
functionality longer than I would like. There also some long-term 
unresolved issues like the ability to "not care" about the exit code of a 
command <https://github.com/rogpeppe/go-internal/issues/93>, which has been 
open for nearly four years.

testscript is a fork from an earlier version of Go's 
src/cmd/go/internal/script, and since then script has been re-written to be 
more modular. rsc.io/script is a semi-official release of 
src/cmd/go/internal/script as it's from one of the original Go developers. 
This puts the future of testscript into doubt: will people switch to 
rsc.io/script or not? I've also raised this with the testscript developers 
<https://github.com/rogpeppe/go-internal/issues/238>.

I can continue to use testscript, but if the feeling is that the world is 
moving to the more modern rsc.io/script, then I'm keen to contribute to 
rsc.io/script to make it useable by other projects. But first, I need to 
know how to contribute to rsc.io/script...

On Thursday, March 7, 2024 at 3:18:51 PM UTC+1 Eli Bendersky wrote:

> On Thu, Mar 7, 2024 at 5:44 AM twp...@gmail.com  wrote:
>
>> Thanks, but I'm already heavily invested in testscript with over 10K 
>> lines of scripts and a lot of custom configuration, and 
>> commander-cli/commander clearly lacks functionality I need.
>>
>
> OOC, is there anything wrong with 
> github.com/rogpeppe/go-internal/testscript 
> <https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript>?
>
> I've recently used it successfully on two diverse projects; it works well, 
> it still appears to be maintained, and has good documentation for 
> out-of-stdlib use cases.
>
>
>
>
>  
>
>>
>> On Thursday, March 7, 2024 at 2:47:03 AM UTC+1 Ajay Kidave wrote:
>>
>>> I have had a good experience using 
>>> https://github.com/commander-cli/commander for testing CLI commands. It 
>>> uses a yaml file for test config. I generally avoid yaml but the commander 
>>> config format is easy to work with. Never used script, the state saving 
>>> between commands feature might be missing in commander-cli.
>>>
>>> On Wednesday, March 6, 2024 at 1:52:57 PM UTC-8 twp...@gmail.com wrote:
>>>
>>>> *rsc.io/script
>>>>
>>>> On Wednesday, March 6, 2024 at 10:46:23 PM UTC+1 twp...@gmail.com 
>>>> wrote:
>>>>
>>>>> Internally, Go has a very nice package for integration testing 
>>>>> <https://github.com/golang/go/tree/master/src/cmd/go/internal/script>.
>>>>>
>>>>> Roger Peppe <https://github.com/rogpeppe> has been doing a fantastic 
>>>>> service by making it public and maintaining it as a community project at 
>>>>> github.com/rogpeppe/go-internal/testscript 
>>>>> <https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript>, and 
>>>>> Roger's package has been extremely valuable for my project 
>>>>> <https://github.com/twpayne/chezmoi> and many others.
>>>>>
>>>>> Over a quarter of a year ago, Russ Cox <https://research.swtch.com/> 
>>>>> made a semi-official release of the code 
>>>>> <https://pkg.go.dev/rsc.io/script> which includes several 
>>>>> improvements (e.g. better modularity), but also removing critical 
>>>>> functionality <https://github.com/rsc/script/issues/5> required to 
>>>>> make rsc.io/script usable by other projects.
>>>>>
>>>>> Appeals to the author about how to contribute 
>>>>> <https://github.com/rsc/script/issues/4> have been met with silence. 
>>>>> Fundamental issues like installation failures 
>>>>> <https://github.com/rsc/script/issues/3> have had no response.
>>>>>
>>>>> Does rsc.io/switch have a future? If so, how can we contribute? This 
>>>>> is a fantastically useful piece of software for the Go community and we 
>>>>> will all be better off if it succeeds.
>>>>>
>>>>> Regards,
>>>>> Tom
>>>>>
>>>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/f6f92dd8-9033-43a0-9348-02c039c7fe90n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/golang-nuts/f6f92dd8-9033-43a0-9348-02c039c7fe90n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/964e1f1e-ac19-4930-bdbf-9494b2519c8en%40googlegroups.com.

Re: [go-nuts] rsc.io/script: How to contribute?

[twp...@gmail.com]

Many thanks for the response Russ. It's clear I should stick with 
testscript for now.

Regards,
Tom

On Friday, March 8, 2024 at 3:04:10 AM UTC+1 Russ Cox wrote:

> Hi all,
>
> I published rsc.io/script for people to play with, to support my testing 
> talk . For now it is on a back burner 
> - I get so much GitHub mail that I cannot keep up with all the 
> notifications, so I hadn't seen the activity there. Perhaps at some point 
> we will decide to publish this code in an officially supported location and 
> change cmd/go to use that copy instead of cmd/go/internal/script, but that 
> would require finalizing the API to the point where we are comfortable 
> supporting it forever, and there are other projects that are more pressing 
> (for example, range over functions and iterator libraries). I am reluctant 
> to change rsc.io/script to add new features and so on, because that just 
> creates a divergence that will have to be resolved when we do want cmd/go 
> to use this hypothetical official version. 
>
> I'm glad people find it useful and I apologize that there is not time to 
> do something more with it myself. That happens in open source.
>
> Best,
> Russ
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1f4cccaa-9e58-42ec-9348-771c84c48634n%40googlegroups.com.

[go-nuts] pkg.go.dev: redirect URL for latest major version or big red warning for old version

[twp...@gmail.com]

Is there a way to get an unchanging URL that always redirects to the latest 
major version of a package on pkg.go.dev?

Right now, if I go to
  https://pkg.go.dev/$PACKAGE_NAME
I get the documentation for either v0 or v1 of a package, which is out of 
date if there's a more recent major version.

What I would like is a URL that given $PACKAGE_NAME always redirects to the 
latest major version of that package. For example, a URL something like
  https://pkg.go.dev/$PACKAGE_NAME@latest
which would redirect to
  https://pkg.go.dev/$PACKAGE_NAME/v2
if the latest major version of $PACKAGE_NAME is v2.

The reasons for this request are:
1. I'm usually using the latest version of the package, and so I want to 
see the docs for the most recent version.
2. If users stumble across documentation for an old major version by 
default then there's a risk that they'll use or evaluate the old major 
version by default. Usually there's a good reason why users should not be 
using the old major version.

Right now, pkg.go.dev displays a tiny black text on a grey background line 
at the top of  the page that says something like:
  The highest tagged major version is v61.
See https://pkg.go.dev/github.com/google/go-github for an example. This is 
extremely easy to miss. Alternatively, this banner should be in big red 
letters - if you're using this version of the module then you're probably 
doing something wrong.

If there's support for this new URL and/or the banner usability 
improvement, then I'd be happy to contribute a fix.

Thoughts?

Regards,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/94837840-dfb7-4c51-9db4-ab184fd8aec1n%40googlegroups.com.

Re: [go-nuts] xml to json, parsing xml

[twp...@gmail.com]

You can parse XML and JSON quickly with these tools:

https://github.com/twpayne/go-xmlstruct
https://github.com/twpayne/go-jsonstruct

They generate Go code that parses all the example XML and JSON files that 
you throw at them.

Regards,
Tom

On Tuesday, April 23, 2024 at 9:17:33 PM UTC+2 Don Caldwell wrote:

> Oops, I didn't look carefully at the json output of my little program. It 
> does sometimes emit arrays. For example:
>
> go build xmlparse.go
>
> ./xmlparse -url 'https://news.google.com/atom'
>
>
> This produces what, for some part, represents acceptable json. The 
> exceptions are the atom entries that would still require, I think, another 
> pass to fix up the go structure unless someone applies a priori knowledge 
> of the xml structure when building it.
>
>
> D
>
>
>
> D
>
>
> D
>
> On Tue, Apr 23, 2024 at 1:18 PM Don Caldwell  wrote:
>
>> I agree. The link that I sent demonstrates one very simple way. Mapping 
>> XML with repetitive tag sets to JSON arrays, though, would take a bit of 
>> work meaning, I think, at least two passes.
>>
>> D
>>
>> On Tue, Apr 23, 2024, 13:04 robert engels  wrote:
>>
>>> I don’t think that is true. There are multiple ways to model XML into 
>>> json.
>>>
>>> See this http://badgerfish.ning.com for one of them.
>>>
>>> On Apr 23, 2024, at 11:43 AM, burak serdar  wrote:
>>>
>>> In general, you cannot convert xml to json. They have incompatible
>>> models. XML elements are ordered similar to a JSON array, but in many
>>> cases you want to map XML elements to JSON objects, which are
>>> unordered name-value collections. Also, there is no JSON equivalent of
>>> an XML attribute.
>>>
>>> If you want to work with XML, either use xml marshaling, or find a
>>> third-party DOM library.
>>>
>>> On Tue, Apr 23, 2024 at 10:29 AM Don Caldwell  wrote:
>>>
>>>
>>> Disclaimer - I am very new to golang.
>>> I puzzled about this for a few days. After some struggle, I got a little 
>>> program working that parses
>>> arbitrary xml into a structure that json can understand. You can find it 
>>> here:
>>> https://github.com/dfwcnj/gxmldecode
>>>
>>> On Thursday, October 7, 2021 at 10:06:30 AM UTC-4 RS wrote:
>>>
>>>
>>> What is the best approach to convert xml to json?
>>> However I need above all to parse a very complex xml to see if related 
>>> properties are existing.
>>> I thought maybe converting it to Json maybe make it easier.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/90c0dd22-2d81-4393-b534-651a2376f386n%40googlegroups.com
>>> .
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/CAMV2RqowOgbOnmGxsWegOuJ-_crQcNhsjj1Gxk3pAXhBmtNK5Q%40mail.gmail.com
>>> .
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5a2d51fa-92e3-47b7-9f62-5412f0e07685n%40googlegroups.com.

[go-nuts] Re: Alternative text/template function libraries to github.com/Masterminds/sprig?

[twp...@gmail.com]

Just following up on this old message, there now seems to be at least:

https://github.com/go-sprout/sprout
https://github.com/chezmoi/templatefuncs

Regards,
Tom

On Monday, January 9, 2023 at 8:53:41 PM UTC+1 Tom Payne wrote:

> github.com/Masterminds/sprig is a popular library of template functions, 
> used by some popular projects, e.g. Kubernetes Helm.
>
> Unfortunately, Masterminds/sprig also has a number of inherent flaws:
> 1. The order of arguments to many of its functions is incompatible with 
> text/template's pipeline syntax.
> 2. Many of its functions do not handle strings, []bytes, variable numbers 
> of arguments, and other argument variations in a sensible way.
> 3. It has, at the time of writing, 78 open issues 
> , most of them unaddressed, 
> and has seen only minor maintenance activity over the last few years.
> 4. Its function names do not follow Go's naming conventions.
>
> #1 and #2 cannot be fixed in a backwards-compatible way. #3 means that 
> fixes aren't practically accepted anyway. #4 just means that templates 
> using Masterminds/sprig are ugly to Go developer eyes.
>
> Instead, I want a library of template functions that works well with 
> text/template, has decent ergonomics, follows Go's naming conventions, and 
> is actively maintained. I don't care about backwards compatibility with 
> sprig as this is impossible to achieve anyway.
>
> Before I start such a project, are there any existing good existing 
> alternatives to Masterminds/sprig?
>
> Many thanks,
> Tom
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/af98d89e-2aa7-4758-9597-0a4192574e64n%40googlegroups.com.

[go-nuts] Re: Advancing the container/set design?

[twp...@gmail.com]

+1 on this.

At the moment, every module that needs a set implementation ends up 
creating its own, either using map[T]struct{} or map[T]bool. Having a set 
implementation in the standard library would significantly increase 
operability between modules, even if the implementation is trivial.



On Monday, July 1, 2024 at 7:43:19 PM UTC+2 Frederik Zipp wrote:

> Some time ago, Ian started a discussion about a potential proposal for a 
> container/set package: https://github.com/golang/go/discussions/47331
>
> The main point of uncertainty was iterating over elements. Now that 
> iteration is solved in Go 1.23, is it perhaps time to advance this design 
> and turn it into a proposal for the 1.24 release cycle?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7730dc45-fc43-4ac5-a9b3-2c8ec7647dddn%40googlegroups.com.

[go-nuts] Re: Advice on Using Pure Go with eBPF

[twp...@gmail.com]

For eBPF support in Go, see https://github.com/cilium/ebpf.

AFAIK at the moment you can't compile Go to eBPF. The most promising 
approach is probably to adapt TinyGo to generate eBPF. There's an issue 
that's been open for three years with no 
activity: https://github.com/tinygo-org/tinygo/issues/2017.

Regards,
Tom

On Wednesday, August 14, 2024 at 12:55:30 AM UTC+2 Sharon Mafgaoker wrote:

> Hey Everyone,
>
> Nice to e-meet you all! I'm new here and kindly asking for some advice on 
> a topic I'm exploring.
>
> While watching Rob Pike's videos about Go, I noticed he doesn't recommend 
> using CGO due to potential issues when changing OS/architecture.
>
> I have an idea to read incoming IPs, perform background checks, and 
> validate them by harnessing the power of concurrency. To analyze packets 
> faster, I want to use eBPF. However, I haven't found a way to load Go code 
> directly into eBPF, and it seems like CGO is the only way.
>
> Does anyone have advice on how to use pure Go with eBPF? Any insights or 
> alternative approaches would be greatly appreciated.
>
> Thank you!
>
> Best regards,
> Sharon
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0fd558f3-28aa-45d4-a205-462e350da1c0n%40googlegroups.com.

Re: [go-nuts] How to define struct with defined properties that is also a map

[twp...@gmail.com]

Also, quick plug for
  https://github.com/twpayne/go-jsonstruct
which will generate type-safe Go structs for you for all fields. Just throw 
a bunch of JIRA JSON responses at it.

Regards,
Tom

On Friday, September 13, 2024 at 12:39:41 AM UTC+2 burak serdar wrote:

> The current json implementation in the stdlib does not support a
> catchall like that. Also, you cannot embed a map in a struct.
>
> You can replace JIRAFields with a map[string]any. If there are keys
> with known structures, you can do a map[string]json.RawMessage, and
> then you can do json.Unmarshal(fields[knownKey],&knownStruct) for
> those keys whose structure you know.
>
> Alternatively, you can use a map[string]any, and use the mapstructure
> library to unmarshal a map[string]any to a known struct.
>
> On Thu, Sep 12, 2024 at 4:30 PM David Karr  wrote:
> >
> > This is probably basic, but I'm working on some code that unmarshals 
> JIRA tickets (just for some context). Examining the Json output, there is a 
> big "fields" object that mostly contains custom fields, but also some 
> defined fields. Right now, I defined "Fields" as a "map[string]any", which 
> is appropriate for the custom fields. However, I would really like to 
> define something like this:
> >
> > type JIRAFields struct {
> > map[string]any
> > Parent JIRAParent
> > }
> > type JIRAParent struct {
> > ...
> > }
> >
> > So I could reference a custom field like "Fields[key]", but the parent 
> field with "Fields.Parent".
> >
> > I'm certain the syntax I have so far isn't correct, as it flags the 
> "map" saying "}" was expected.
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to golang-nuts...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAA5t8VqfpZ7og6Zxobcy388Zbyj%2BnTGc-J0gW_LBj9ngrErg4w%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a4f561e7-5838-4698-93c1-403750c3fc32n%40googlegroups.com.

[go-nuts] Re: ANN: Basic test impact analysis (benchmark shows avg 29% reduction in test execution time atm)

[twp...@gmail.com]

How does this interact with Go's existing test result cache, and 
specifically the invalidation of that cache?

For example,  go test tracks which external files are accessed by tests and 
will invalidate the test results cache when those files are modified.

Will symflower test-runner also re-run tests when files accessed by tests 
are modified, even if the affected tests are not touched by the recent code 
changes?

Regards,
Tom

On Thursday, September 12, 2024 at 4:48:03 PM UTC+2 Markus Zimmermann wrote:

>
> [image: with-go-test-with-test-runner.png]
>
> We implemented a basic test impact analysis that identifies and then 
> executes affected tests for a change, e.g. for a range of commits. Our 
> benchmark for the repositories we looked at shows an average 29% reduction 
> in test execution time. I am very much looking forward to hearing how your 
> repositories perform!
>
> Details of the benchmark and how the analysis/command works can be found 
> here: https://symflower.com/en/company/blog/2024/test-impact-analysis/. 
> The approach right now is to query a diff using Git and then check which Go 
> packages are affected. In the next iterations, we will bring this analysis 
> down to individual function and test-case level. The eventual goal is to 
> use our symbolic execution engine to allow for even deeper granularity.
>
> At one point i like to have it as a drop-in command for `go` but right now 
> execution looks like this (for all changes of the last commit to now):
>
> ```
> symflower test-runner --commit-from HEAD~ -- go test -v
> ```
>
> Looking forward to your feedback!
>
> Cheers,
> Markus
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/be6062e6-09dc-4e18-bce9-2c3cef419d8en%40googlegroups.com.

[go-nuts] Re: ANN: Basic test impact analysis (benchmark shows avg 29% reduction in test execution time atm)

[twp...@gmail.com]

Constructively, here's a great starting point for understanding go test's 
cache:
  
https://github.com/golang/go/blob/f117d1c9b5951ab2456c1e512ac0423fcf3d7ada/src/cmd/go/internal/test/test.go#L1750-L1773

Regards,
Tom

On Friday, September 13, 2024 at 3:44:37 PM UTC+2 twp...@gmail.com wrote:

> How does this interact with Go's existing test result cache, and 
> specifically the invalidation of that cache?
>
> For example,  go test tracks which external files are accessed by tests 
> and will invalidate the test results cache when those files are modified.
>
> Will symflower test-runner also re-run tests when files accessed by tests 
> are modified, even if the affected tests are not touched by the recent code 
> changes?
>
> Regards,
> Tom
>
> On Thursday, September 12, 2024 at 4:48:03 PM UTC+2 Markus Zimmermann 
> wrote:
>
>>
>> [image: with-go-test-with-test-runner.png]
>>
>> We implemented a basic test impact analysis that identifies and then 
>> executes affected tests for a change, e.g. for a range of commits. Our 
>> benchmark for the repositories we looked at shows an average 29% reduction 
>> in test execution time. I am very much looking forward to hearing how your 
>> repositories perform!
>>
>> Details of the benchmark and how the analysis/command works can be found 
>> here: https://symflower.com/en/company/blog/2024/test-impact-analysis/. 
>> The approach right now is to query a diff using Git and then check which Go 
>> packages are affected. In the next iterations, we will bring this analysis 
>> down to individual function and test-case level. The eventual goal is to 
>> use our symbolic execution engine to allow for even deeper granularity.
>>
>> At one point i like to have it as a drop-in command for `go` but right 
>> now execution looks like this (for all changes of the last commit to now):
>>
>> ```
>> symflower test-runner --commit-from HEAD~ -- go test -v
>> ```
>>
>> Looking forward to your feedback!
>>
>> Cheers,
>> Markus
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/415b0fc1-9707-4afc-a414-c3ec47b5ef1en%40googlegroups.com.

[go-nuts] go test cache: include umask as a test input for caching?

[twp...@gmail.com]

tl;dr: umask  is a system-wide global 
that affects go tests and should be considered when validating go test's 
cache.


Motivation:

I'm the author of a popular open source  
project that writes files to the user's home directory and cares about 
getting exact file permissions correct. The file permissions depend on the 
the current umask setting. As umask is a process global variable, it cannot 
be set for individual tests, and so separate tests are needed for different 
umasks 

.

Currently changing the umask does not invalidate go test's cache, so I get 
incorrect test results if I change the umask and re-run go test.


Suggested solution:

Include umask as an op in go test's id calculation 

.


Next steps:

* Is this something that the Go project would consider?
* If so, I would be happy to submit a CL.


Thanks,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/80655581-661e-46a5-baaa-9a1ba7a8b319n%40googlegroups.com.

Re: [go-nuts] go test cache: include umask as a test input for caching?

[twp...@gmail.com]

> Personally, I would approach this kind of thing by writing a test that
sets the umask to various different values and invokes subtests with
each umask value.

I would love to do this but umask is a process global (as far as I 
understand it) and so can't be set within subtests.

On Saturday, September 14, 2024 at 12:33:19 AM UTC+2 Ian Lance Taylor wrote:

> On Fri, Sep 13, 2024 at 3:03 PM twp...@gmail.com  wrote:
> >
> > tl;dr: umask is a system-wide global that affects go tests and should be 
> considered when validating go test's cache.
> >
> >
> > Motivation:
> >
> > I'm the author of a popular open source project that writes files to the 
> user's home directory and cares about getting exact file permissions 
> correct. The file permissions depend on the the current umask setting. As 
> umask is a process global variable, it cannot be set for individual tests, 
> and so separate tests are needed for different umasks.
> >
> > Currently changing the umask does not invalidate go test's cache, so I 
> get incorrect test results if I change the umask and re-run go test.
> >
> >
> > Suggested solution:
> >
> > Include umask as an op in go test's id calculation.
> >
> >
> > Next steps:
> >
> > * Is this something that the Go project would consider?
> > * If so, I would be happy to submit a CL.
>
> Personally, I would approach this kind of thing by writing a test that
> sets the umask to various different values and invokes subtests with
> each umask value. That way the test is independent of the external
> environment.
>
> In general our approach has been that if your test intrinsically
> depends on the external environment, then you should run it with
> -test.run=1 to disable the test cache.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5c112417-db5a-46a4-b4ef-0146af38dfc2n%40googlegroups.com.

Re: [go-nuts] go test cache: include umask as a test input for caching?

[twp...@gmail.com]

umask cannot be set in subtests for two reasons:
1. It is a process-wide global variable stored by the operating system. 
Changing the umask in a test changes the umask for the entire process, i.e. 
it changes the umask for all tests.
2. It is not possible to set and restore the umask atomically. This makes 
it inherently racy for concurrent programs.

To learn more about umask, and why it is special, please read the man page 
<https://man7.org/linux/man-pages/man2/umask.2.html>.
On Monday, September 16, 2024 at 5:34:29 PM UTC+2 Jason Phillips wrote:

> Why can't it be set within subtests? Note that subtests (like regular 
> tests) aren't run in parallel unless you explicitly call t.Parallel().
>
> On Friday, September 13, 2024 at 6:35:15 PM UTC-4 twp...@gmail.com wrote:
>
>> > Personally, I would approach this kind of thing by writing a test that
>> sets the umask to various different values and invokes subtests with
>> each umask value.
>>
>> I would love to do this but umask is a process global (as far as I 
>> understand it) and so can't be set within subtests.
>>
>> On Saturday, September 14, 2024 at 12:33:19 AM UTC+2 Ian Lance Taylor 
>> wrote:
>>
>>> On Fri, Sep 13, 2024 at 3:03 PM twp...@gmail.com  
>>> wrote: 
>>> > 
>>> > tl;dr: umask is a system-wide global that affects go tests and should 
>>> be considered when validating go test's cache. 
>>> > 
>>> > 
>>> > Motivation: 
>>> > 
>>> > I'm the author of a popular open source project that writes files to 
>>> the user's home directory and cares about getting exact file permissions 
>>> correct. The file permissions depend on the the current umask setting. As 
>>> umask is a process global variable, it cannot be set for individual tests, 
>>> and so separate tests are needed for different umasks. 
>>> > 
>>> > Currently changing the umask does not invalidate go test's cache, so I 
>>> get incorrect test results if I change the umask and re-run go test. 
>>> > 
>>> > 
>>> > Suggested solution: 
>>> > 
>>> > Include umask as an op in go test's id calculation. 
>>> > 
>>> > 
>>> > Next steps: 
>>> > 
>>> > * Is this something that the Go project would consider? 
>>> > * If so, I would be happy to submit a CL. 
>>>
>>> Personally, I would approach this kind of thing by writing a test that 
>>> sets the umask to various different values and invokes subtests with 
>>> each umask value. That way the test is independent of the external 
>>> environment. 
>>>
>>> In general our approach has been that if your test intrinsically 
>>> depends on the external environment, then you should run it with 
>>> -test.run=1 to disable the test cache. 
>>>
>>> Ian 
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c9dcf1bb-25ae-4e58-8714-66325077c2d1n%40googlegroups.com.

Re: [go-nuts] go test cache: include umask as a test input for caching?

[twp...@gmail.com]

> I interpret that to mean you are changing the umask before running `go 
test`. That is, you might run `go test` with different umask values and 
don't want the cached results to ignore the umask value. As did Ian if I 
correctly understood their reply. The discussion then evolved into how a 
specific unit test can safely change the umask without affecting other unit 
tests. Running unit tests with different umask values is a different 
problem from how to write unit tests that modify the umask value.

That is correct. Unfortunately there seems to be a lack of knowledge in 
some cases of what umask is and how it behaves, which is why the discussion 
got sidetracked.

> At this point it is unclear, at least to me, what problem you are trying 
to solve.

I want go test to invalidate its cache when the umask is changed, i.e. I 
want the following sequence of commands to work correctly:

$ umask 002
$ go test ./...
$ umask 022
$ go test ./... # should not use cache
$ go test ./... # should use cache

Currently, the go test cache does not record the umask, so it incorrectly 
re-uses the cached test results in the second invocation of go test. It 
should instead invalidate the cache.

Note that go test does record the external files used by the tests and the 
external environment variables used by the tests, and so it correctly 
invalidates the cache when any of them change. I would like the cache to 
also be invalidated when the external umask changes. Alternately put, go 
test already has special case code for external changes to files and 
environment variables, and I would like extend that to external umask 
changes too.

Hope this makes my request clearer.

I'd be delighted to submit at CL to do this (it's probably 10-20 lines of 
code with almost no performance penalty [1]), but I want to check that it 
has a reasonable chance to being accepted before I start the work.

[1] The performance cost is two very fast syscalls during startup.
On Tuesday, September 17, 2024 at 10:49:12 AM UTC+2 Kurtis Rader wrote:

> You wrote earlier:
>
> > Currently changing the umask does not invalidate go test's cache, so I 
> get incorrect test results if I change the umask and re-run go test
>
> I interpret that to mean you are changing the umask before running `go 
> test`. That is, you might run `go test` with different umask values and 
> don't want the cached results to ignore the umask value. As did Ian if I 
> correctly understood their reply. The discussion then evolved into how a 
> specific unit test can safely change the umask without affecting other unit 
> tests. Running unit tests with different umask values is a different 
> problem from how to write unit tests that modify the umask value. At this 
> point it is unclear, at least to me, what problem you are trying to solve.
>
> On Tue, Sep 17, 2024 at 1:33 AM twp...@gmail.com  wrote:
>
>> umask cannot be set in subtests for two reasons:
>> 1. It is a process-wide global variable stored by the operating system. 
>> Changing the umask in a test changes the umask for the entire process, i.e. 
>> it changes the umask for all tests.
>> 2. It is not possible to set and restore the umask atomically. This makes 
>> it inherently racy for concurrent programs.
>>
>> To learn more about umask, and why it is special, please read the man 
>> page <https://man7.org/linux/man-pages/man2/umask.2.html>.
>> On Monday, September 16, 2024 at 5:34:29 PM UTC+2 Jason Phillips wrote:
>>
>>> Why can't it be set within subtests? Note that subtests (like regular 
>>> tests) aren't run in parallel unless you explicitly call t.Parallel().
>>>
>>> On Friday, September 13, 2024 at 6:35:15 PM UTC-4 twp...@gmail.com 
>>> wrote:
>>>
>>>> > Personally, I would approach this kind of thing by writing a test that
>>>> sets the umask to various different values and invokes subtests with
>>>> each umask value.
>>>>
>>>> I would love to do this but umask is a process global (as far as I 
>>>> understand it) and so can't be set within subtests.
>>>>
>>>> On Saturday, September 14, 2024 at 12:33:19 AM UTC+2 Ian Lance Taylor 
>>>> wrote:
>>>>
>>>>> On Fri, Sep 13, 2024 at 3:03 PM twp...@gmail.com  
>>>>> wrote: 
>>>>> > 
>>>>> > tl;dr: umask is a system-wide global that affects go tests and 
>>>>> should be considered when validating go test's cache. 
>>>>> > 
>>>>> > 
>>>>> > Motivation: 
>>>>> > 
>>>>> > I'm the author of a popular open source project that writes file

Re: [go-nuts] go test cache: include umask as a test input for caching?

[twp...@gmail.com]

> Should work, AIUI, even with the issues you mention. Or is there 
something else going on that I'm unaware of?

The problem is not running the tests. The problem is that the test cache is 
not invalidated correctly.

On Tuesday, September 17, 2024 at 11:52:35 AM UTC+2 Axel Wagner wrote:

On Tue, 17 Sept 2024 at 10:33, twp...@gmail.com  wrote:

umask cannot be set in subtests for two reasons:
1. It is a process-wide global variable stored by the operating system. 
Changing the umask in a test changes the umask for the entire process, i.e. 
it changes the umask for all tests.
2. It is not possible to set and restore the umask atomically. This makes 
it inherently racy for concurrent programs.


I might be misunderstanding something, but neither of these seems to 
actually be a problem. Tests are not run concurrently, unless explicitly 
requested by calling `t.Parallel()`. The same goes for sub tests, I 
believe. So, simply doing

func TestOne(t *testing.T) {
defer syscall.Umask(syscall.Umask(0))
// test goes here
}
func TestTwo(t *testing.T) {
defer syscall.Umask(syscall.Umask(0777))
// test goes here
}



To learn more about umask, and why it is special, please read the man page 
<https://man7.org/linux/man-pages/man2/umask.2.html>.
On Monday, September 16, 2024 at 5:34:29 PM UTC+2 Jason Phillips wrote:

Why can't it be set within subtests? Note that subtests (like regular 
tests) aren't run in parallel unless you explicitly call t.Parallel().

On Friday, September 13, 2024 at 6:35:15 PM UTC-4 twp...@gmail.com wrote:

> Personally, I would approach this kind of thing by writing a test that
sets the umask to various different values and invokes subtests with
each umask value.

I would love to do this but umask is a process global (as far as I 
understand it) and so can't be set within subtests.

On Saturday, September 14, 2024 at 12:33:19 AM UTC+2 Ian Lance Taylor wrote:

On Fri, Sep 13, 2024 at 3:03 PM twp...@gmail.com  wrote: 
> 
> tl;dr: umask is a system-wide global that affects go tests and should be 
considered when validating go test's cache. 
> 
> 
> Motivation: 
> 
> I'm the author of a popular open source project that writes files to the 
user's home directory and cares about getting exact file permissions 
correct. The file permissions depend on the the current umask setting. As 
umask is a process global variable, it cannot be set for individual tests, 
and so separate tests are needed for different umasks. 
> 
> Currently changing the umask does not invalidate go test's cache, so I 
get incorrect test results if I change the umask and re-run go test. 
> 
> 
> Suggested solution: 
> 
> Include umask as an op in go test's id calculation. 
> 
> 
> Next steps: 
> 
> * Is this something that the Go project would consider? 
> * If so, I would be happy to submit a CL. 

Personally, I would approach this kind of thing by writing a test that 
sets the umask to various different values and invokes subtests with 
each umask value. That way the test is independent of the external 
environment. 

In general our approach has been that if your test intrinsically 
depends on the external environment, then you should run it with 
-test.run=1 to disable the test cache. 

Ian 

-- 

You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c9dcf1bb-25ae-4e58-8714-66325077c2d1n%40googlegroups.com
 
<https://groups.google.com/d/msgid/golang-nuts/c9dcf1bb-25ae-4e58-8714-66325077c2d1n%40googlegroups.com?utm_medium=email&utm_source=footer>
.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/abe6db22-3ca3-4be7-8fe2-539ffc44689dn%40googlegroups.com.

Re: [go-nuts] go test cache: include umask as a test input for caching?

[twp...@gmail.com]

On Tuesday, September 17, 2024 at 12:44:42 PM UTC+2 Stephen Illingworth 
wrote:

If the only problem is that the cache is getting in the way then you can 
run the tests such that test cache is not used. The idiomatic way is to use 
-count=1 on the test command line

I would do this:

go test -count=1 

And then:

go test 


I am aware that I can work around go test cache's incorrect behavior by 
disabling the cache. This is annoying for several reasons:
* I have to manually maintain a list of tests that are affected by the 
umask.
* I have to run go test twice.
* I don't get the speed-ups of caching when the caching should work, which 
slows down my development cycle.

Note that all tests that write files are affected by umask, even if many 
people don't realize this. To demonstrate this, find a project that calls 
testing.T.TempDir() and run:

$ umask 777
$ go test ./... -count=1

The odds are that the tests will fail.

However, I want to make go test cache's behavior more correct for a case 
where it is currently incorrect for no measurable performance penalty. What 
are the reasons for *not* doing this?


On Tuesday 17 September 2024 at 11:25:40 UTC+1 twp...@gmail.com wrote:

> Should work, AIUI, even with the issues you mention. Or is there 
something else going on that I'm unaware of?

The problem is not running the tests. The problem is that the test cache is 
not invalidated correctly.

On Tuesday, September 17, 2024 at 11:52:35 AM UTC+2 Axel Wagner wrote:

On Tue, 17 Sept 2024 at 10:33, twp...@gmail.com  wrote:

umask cannot be set in subtests for two reasons:
1. It is a process-wide global variable stored by the operating system. 
Changing the umask in a test changes the umask for the entire process, i.e. 
it changes the umask for all tests.
2. It is not possible to set and restore the umask atomically. This makes 
it inherently racy for concurrent programs.


I might be misunderstanding something, but neither of these seems to 
actually be a problem. Tests are not run concurrently, unless explicitly 
requested by calling `t.Parallel()`. The same goes for sub tests, I 
believe. So, simply doing

func TestOne(t *testing.T) {
defer syscall.Umask(syscall.Umask(0))
// test goes here
}
func TestTwo(t *testing.T) {
defer syscall.Umask(syscall.Umask(0777))
// test goes here
}


To learn more about umask, and why it is special, please read the man page 
<https://man7.org/linux/man-pages/man2/umask.2.html>.
On Monday, September 16, 2024 at 5:34:29 PM UTC+2 Jason Phillips wrote:

Why can't it be set within subtests? Note that subtests (like regular 
tests) aren't run in parallel unless you explicitly call t.Parallel().

On Friday, September 13, 2024 at 6:35:15 PM UTC-4 twp...@gmail.com wrote:

> Personally, I would approach this kind of thing by writing a test that
sets the umask to various different values and invokes subtests with
each umask value.

I would love to do this but umask is a process global (as far as I 
understand it) and so can't be set within subtests.

On Saturday, September 14, 2024 at 12:33:19 AM UTC+2 Ian Lance Taylor wrote:

On Fri, Sep 13, 2024 at 3:03 PM twp...@gmail.com  wrote: 
> 
> tl;dr: umask is a system-wide global that affects go tests and should be 
considered when validating go test's cache. 
> 
> 
> Motivation: 
> 
> I'm the author of a popular open source project that writes files to the 
user's home directory and cares about getting exact file permissions 
correct. The file permissions depend on the the current umask setting. As 
umask is a process global variable, it cannot be set for individual tests, 
and so separate tests are needed for different umasks. 
> 
> Currently changing the umask does not invalidate go test's cache, so I 
get incorrect test results if I change the umask and re-run go test. 
> 
> 
> Suggested solution: 
> 
> Include umask as an op in go test's id calculation. 
> 
> 
> Next steps: 
> 
> * Is this something that the Go project would consider? 
> * If so, I would be happy to submit a CL. 

Personally, I would approach this kind of thing by writing a test that 
sets the umask to various different values and invokes subtests with 
each umask value. That way the test is independent of the external 
environment. 

In general our approach has been that if your test intrinsically 
depends on the external environment, then you should run it with 
-test.run=1 to disable the test cache. 

Ian 

-- 

You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c9dcf1bb-25ae-4e58-8714-66325077c2d1n%40googlegroups.com
 
<https://groups.google.com/d/msgid/gol

[go-nuts] Re: ANN: Priority channels: re-order values sent over a channel

[twp...@gmail.com]

Thanks Jason, this is great feedback.

I've added a context argument in https://github.com/twpayne/go-heap/pull/6.

Regards,
Tom

On Sunday, October 6, 2024 at 9:58:43 PM UTC+2 Jason E. Aten wrote:

> Hi Tom,
>
> This is an interesting project.
>
> Per your request for feedback: on quick inspection, I would certainly want 
> also a way to close down the priority channels too. 
>
> Some sort of Close() method that would stop all running goroutines from 
> behind-the-scenes, so they do not leak, would be important for any long 
> running process.
>
> Maybe even use of a context.Context to effect the shutdown?
>
> You'll quickly realize that every go channel "raw" operation (not in a 
> select) needs to be wrapped in a select{} in order to not be deadlocked 
> when the time to shutdown arrives (usually on a channel itself). So for 
> instance,
>
> https://github.com/twpayne/go-heap/blob/v0.0.2/prioritychannel.go#L89 
> which says
>
> for value := range heap.All() {
> outCh <- value
> }
>
> would never do. One would need to allow for shutdown, something like:
>
> for value := range heap.All() {
>select {
> case outCh <- value:
> case <- 
> shutdownRequestedCh:
>return // and 
> presumably run other cleanup in a defer at the top of this func
> }
> }
>
> ...and repeat such fixes for all other "raw" channel sends and receives.
>
> Regards,
> Jason
>
>
>
>
> On Sunday, October 6, 2024 at 1:42:14 AM UTC+1 twp...@gmail.com wrote:
>
>> Have you ever wanted a channel that partially sorts the values being 
>> passed through it? For example, you have some kind of work queue and you 
>> want to do the most important work first. Well, now you can!
>>
>> github.com/twpayne/go-heap.PriorityChannel 
>> <https://pkg.go.dev/github.com/twpayne/go-heap#PriorityChannel> 
>> implements this: give it a channel and a comparison function and you get 
>> back a new channel that returns values in priority order until the original 
>> channel is closed.
>>
>> Only want the highest priority value from the last N? Then use 
>> github.com/twpayne/go-heap.BufferedPriorityChannel 
>> <https://pkg.go.dev/github.com/twpayne/go-heap#BufferedPriorityChannel>.
>>
>> Code is at 
>> https://github.com/twpayne/go-heap/blob/main/prioritychannel.go. It's a 
>> fun combination of generics, iterators, and channels.
>>
>> Feedback welcome!
>>
>> Regards,
>> Tom
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/cf08ad9f-c47d-46b3-8653-7caf80ac31dcn%40googlegroups.com.

[go-nuts] ANN: Priority channels: re-order values sent over a channel

[twp...@gmail.com]

Have you ever wanted a channel that partially sorts the values being passed 
through it? For example, you have some kind of work queue and you want to 
do the most important work first. Well, now you can!

github.com/twpayne/go-heap.PriorityChannel 
 implements 
this: give it a channel and a comparison function and you get back a new 
channel that returns values in priority order until the original channel is 
closed.

Only want the highest priority value from the last N? Then use 
github.com/twpayne/go-heap.BufferedPriorityChannel 
.

Code is at https://github.com/twpayne/go-heap/blob/main/prioritychannel.go. 
It's a fun combination of generics, iterators, and channels.

Feedback welcome!

Regards,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/95ad14df-81aa-48c7-b3a3-658bd747f82cn%40googlegroups.com.

Re: [go-nuts] How to Implement Server-Sent Events in Go

[twp...@gmail.com]

I've had good success with Go/Server-Sent Events/HTMX/templ for 
cross-platform tools. Quick overview:

* Go runs a webserver listening on localhost and opens the user's browser.
* Front end is HTMX  with HTML fragments served using 
the templ  HTML templating engine.
* Updates to the front end are pushed with SSE.
* Everything is distributed as a single self-contained binary, with assets 
bundled with go:embed.
* Cross-platform support (Linux/Windows/macOS) is trivial thanks to Go's 
cross-compilation.

My own SSE library that makes SSE as easy to use as Go channels: 
https://github.com/twpayne/go-sse 

There's also https://github.com/romshark/templier which supports this kind 
of application, but I've not used it yet.

Regards,
Tom


On Thursday, October 24, 2024 at 3:01:06 AM UTC+2 Robert Engels wrote:

> Not sure. I worked with SSE last year and got our needs WS was much easier 
> to deal with - for a few of the reasons cited in the article and some 
> others. 
>
> On Oct 23, 2024, at 7:24 PM, Sharon Mafgaoker  wrote:
>
> 
>
> Hi Robert
> Thank you for sharing with us.
> From 2020 still relevant?
>
> Sharon Mafgaoker – Senior Solutions Architect 
>
> M. 050 995 99 16 | sha...@cloud5.co.il
>
>
>
>
> On Wed, 23 Oct 2024 at 17:53 robert engels  wrote:
>
>> This might be helpful 
>> https://dev.to/miketalbot/server-sent-events-are-still-not-production-ready-after-a-decade-a-lesson-for-me-a-warning-for-you-2gie
>>
>> We had an SSE and abandoned it for WS - but we were high volume.
>>
>> On Oct 23, 2024, at 9:48 AM, robert engels  wrote:
>>
>> A lot depends on the volume - you need back pressure support for high 
>> volume events over the internet - and the built-in ping/pong of WS makes 
>> this straightforward. I think it is pretty hard to do back pressure if 
>> unidirectional as well (need side endpoints, etc.).
>>
>> On Oct 23, 2024, at 9:44 AM, 'Zane Attahri' via golang-nuts <
>> golan...@googlegroups.com> wrote:
>>
>> Second this, and go ever further. 
>>
>> If you don’t need bi-directional communication, SSE are almost always the 
>> better choice.  Simpler to implement, standard, and easier to consume by 
>> non-browser clients.
>>
>> On Oct 23, 2024, at 10:37 AM, Brian Hatfield  wrote:
>>
>> I don't think it's quite so binary. Websockets are a lot more complex, 
>> require more sophisticated endpoints and load balancing. SSE has fine 
>> browser support in 2024, minus the ability to set auth headers. I think for 
>> cases with unidirectional communication, SSE is a choice worth evaluating
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/2A0061EB-36AA-472E-8B9D-81BBD0F779E0%40attahri.com
>> .
>>
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/797096F1-5219-40BD-A382-AF79B435EAAB%40ix.netcom.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/27067e2d-c9cf-4eed-87a6-10ff97f91823n%40googlegroups.com.

[go-nuts] Re: Should os.WriteFile call Sync()?

[twp...@gmail.com]

See also: https://github.com/google/renameio

On Friday, April 18, 2025 at 1:31:52 PM UTC+2 Brian Candler wrote:

> On Thursday, 17 April 2025 at 00:10:08 UTC+1 Karel Bílek wrote:
>
> I was quite surprised recently that, at least on linux, file.Close() does 
> not guarantee file.Sync(); in some edge-cases, files can be not put 
> properly to filesystem.
>
> If the data integrity is critical, it seems better to call file.Sync() 
> before file.Close().
>
>
> It depends what you mean by "data integrity". If you want to be reasonably 
> sure[^1] that the data has been persisted to disk *before you continue with 
> anything else*, e.g. in case the power is pulled out later, then yes, you 
> need to fsync the file [^2].
>
> However, of course, someone could pull the power plug *before* your 
> program gets to the point of calling fsync() and/or close().  Therefore, 
> it's really a question of how your application recovers from errors when it 
> next starts, and/or how it communicates with other applications. For 
> example, say that its next step is to send a reply saying "yes I got your 
> request, you don't need to worry about it any more", then maybe the 
> contract says that the other party is entitled to assume that the message 
> has been persisted safely when it receives that message. Therefore, you 
> should persist to disk before sending the reply.
>
> Theodore Ts'o explains this very nicely:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54
> https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/
>
> https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/
>
> In particular, people were assuming that if you write and close a file 
> (without syncing), followed by an atomic rename, the filesystem would 
> guarantee that *either* the old file *or* the new file would be persisted 
> to disk. Because of delayed allocation this was a false assumption - but it 
> is so ingrained in many pieces of code that a workaround was put into ext4 
> so that people get the behaviour they "expect".
>
> Aside: sometimes all that you need for data integrity is sequencing, which 
> can be enforced by write barriers, without having to wait for things to 
> complete (since the write barrier passes down the stack even through 
> deferred writes).
>
> For example, suppose your application did the following:
> - write chunk A
> - barrier
> - write chunk B
>
> The power could be pulled out *at any point*, even in the middle of a 
> write. On restart you will have to deal with these situations:
> - corrupt or incomplete A only
> - complete A, corrupt or incomplete B
> - complete A, complete B
>
> But with a write barrier, you will never see:
> - corrupt or incomplete A, corrupt or incomplete B
> - corrupt or incomplete A, complete B
>
> [^1] if the device lies, e.g. it says the data has been put in persistent 
> storage but it's only in non-battery-backed RAM, then all bets are off.
>
> [^2] it's also essential to check the return code from fsync():
> https://wiki.postgresql.org/wiki/Fsync_Errors
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/e85a36d7-9f01-402a-b8b2-bb0ec13d7054n%40googlegroups.com.

[go-nuts] Re: go mod:Force dependency to use newer major version of a module?

[twp...@gmail.com]

Thank you David and Joe for you replies.

To close this issue for now: the conclusion is that there is no current 
solution.

Regards,
Tom


On Monday, April 14, 2025 at 3:37:15 PM UTC+2 twp...@gmail.com wrote:

> Hello,
>
> My project uses github.com/google/go-github/v71 and one of its 
> dependencies uses an earlier major version of the same module, 
> github.com/google/go-github/v61. Consequently, the compiled binary 
> contains both major versions of go-github, which bloats the binary by about 
> 2MB.
>
> Is there a way to use only the most recent major version?
>
> I tried (in go.mod):
>
>   replace github.com/google/go-github/v61 v61.0.0 => 
> github.com/google/go-github/v71 v71.0.0
>
> but running go mod tidy then gives the error:
>
>   go: github.com/google/go-github/v...@v71.0.0 
> <http://github.com/google/go-github/v71@v71.0.0> used for two different 
> module paths (github.com/google/go-github/v61 and 
> github.com/google/go-github/v71)
>
> For the full code of the attempt, see this draft PR 
> <https://github.com/twpayne/chezmoi/pull/4403>.
>
> Background info:
> * github.com/google/go-github seems to bump its major version every 
> couple of weeks, even when it does not break backwards compatibility.
> * I can use the same old version (v61) in my project as in the dependency, 
> but I'd like to keep my project's dependencies up to date.
> * I could submit a PR to the dependency to update it to use the latest 
> major version of github.com/google/go-github, but because 
> google/go-github bumps major versions so often, this is not a sustainable 
> long term solution.
>
> So, is there an incantation that I can add to my go.mod to force a 
> dependency to use a newer major version of a module?
>
> Thank you for any insight,
> Tom
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/8d923e41-bc25-4f21-adae-e0559c7378d8n%40googlegroups.com.

[go-nuts] go mod:Force dependency to use newer major version of a module?

[twp...@gmail.com]

Hello,

My project uses github.com/google/go-github/v71 and one of its dependencies 
uses an earlier major version of the same module, 
github.com/google/go-github/v61. Consequently, the compiled binary contains 
both major versions of go-github, which bloats the binary by about 2MB.

Is there a way to use only the most recent major version?

I tried (in go.mod):

  replace github.com/google/go-github/v61 v61.0.0 => 
github.com/google/go-github/v71 v71.0.0

but running go mod tidy then gives the error:

  go: github.com/google/go-github/v71@v71.0.0 used for two different module 
paths (github.com/google/go-github/v61 and github.com/google/go-github/v71)

For the full code of the attempt, see this draft PR 
.

Background info:
* github.com/google/go-github seems to bump its major version every couple 
of weeks, even when it does not break backwards compatibility.
* I can use the same old version (v61) in my project as in the dependency, 
but I'd like to keep my project's dependencies up to date.
* I could submit a PR to the dependency to update it to use the latest 
major version of github.com/google/go-github, but because google/go-github 
bumps major versions so often, this is not a sustainable long term solution.

So, is there an incantation that I can add to my go.mod to force a 
dependency to use a newer major version of a module?

Thank you for any insight,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/200df86c-c042-42aa-80ed-a6a6f885d02cn%40googlegroups.com.

[go-nuts] golang.org/x/crypto/x509roots/fallback: high, unskippable, init cost

[twp...@gmail.com]

tl;dr importing golang.org/x/crypto/x509roots/fallback adds ~8ms to the 
startup time of every program or library that imports it. I would like this 
cost to be zero and would be happy to contribute a fix.

Running the following program:

package main
import _ "golang.org/x/crypto/x509roots/fallback"
func main() {}

with the command:

$ go build -o tmp main.go
$ GODEBUG=inittrace=1 ./tmp |& awk '{print $5, $6, $2}' | sort -n | 
tail -n 3
0.043 ms internal/godebug
0.052 ms runtime
8.8 ms golang.org/x/crypto/x509roots/fallback

shows that golang.org/x/crypto/x509roots/fallback adds an 9ms start-up cost 
on a fast CPU (AMD Ryzen 8700G). This cost is paid whether or not the 
fallback X.509 certificates are used or not.

>From looking at the code 
, 
it seems that the CPU is cost is due to parsing about 150 certificates in 
.PEM format in the package's unskippable init function.

The obvious, easy, backwards-compatible fix would be to parse the .PEM 
files while executing the template that generates bundle.go, instead of 
parsing them every time at startup.

Would you accept a CL to fix this?

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/d9258786-72c2-47ee-bad3-de6934ff5f80n%40googlegroups.com.

[go-nuts] Re: "gopkg.in/yaml.v2" package silently misparses file that is misformatted

[twp...@gmail.com]

I've recently switched to https://github.com/goccy/go-yaml and am very 
happy with it so far. I've not tried it on your use case.

Regards,
Tom

On Friday, February 28, 2025 at 12:34:29 AM UTC+1 David Karr wrote:

> I wrote some code to load a yaml file and do some work with the resulting 
> data.  I'm using the "gopkg.in/yaml.v2" package for it.  This has been 
> working fine for properly formatted YAML.  However, today I discovered that 
> a slightly misformatted YAML file is being happily loaded by this code, 
> without throwing any error, but also making sort of odd decisions on what 
> data to actually load, although seeing what it did I suppose that's 
> debatable.
>
> In my suspect yaml file, I have something like this:
>
> stuff:
>   keys:
> - key1
>   - key2
>   - key3
>
> Note the incorrect indentation for "key2" and "key3".
>
> When I load this with code like this:
>
> err = yaml.Unmarshal(configFile, &config)
> if err != nil {
> log.Fatalf("Failed to parse configuration file: %v", err)
> }
>
> This unexpectedly does NOT fail. However, it produces a "keys" list with 
> only one entry, with the following value:
>
> key1 - key2 - key3
>
> I can sort of see why it would make that decision. Is the lesson here that 
> YAML is intended to be easily readable, but not easily writable? I see that 
> there are some command-line tools for consistent formatting of YAML, but I 
> need this done in code, but I think I'd rather just fail if the formatting 
> is inconsistent. Is there any kind of a "strict yaml parser" that will 
> notice things like this?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/640a43c6-8f1c-41ca-99f4-efa498164fe0n%40googlegroups.com.

[go-nuts] runtime.AddCleanup and C struct hierarchies: cleanup order?

[twp...@gmail.com]

The documentation for runtime.AddCleanup 
 says:

> There is no specified order in which cleanups will run.

Given the following types:

type Parent struct {
parentResource int
}

type Child struct {
parent *Parent
childResource int
}

and the following code:

parentResource := 0
parent := &Parent{
parentResource: parentResource,
}
runtime.AddCleanup(parent, func (int) {}, parentResource)
childResource := 1
child := &Child{
parent: parent,
childResource: childResource
}
runtime.AddCleanup(child, func(int) {}, childResource)

is it guaranteed that the cleanup for childResource will run before the 
cleanup for parentResource?

Notes:
* I know that the structs here are small enough to be "tiny", but Child 
contains a pointer to Parent so the cleanups should run.
* The example here is for a specific case where the resources are allocated 
by a C library, the child resource is "owned" by the parent, and so it's 
important that the child resource cleanup runs before the parent resource 
cleanup.

Many thanks for any insight!

Tom

The actual code I'm worried about is 
here: 
https://github.com/twpayne/go-geos/pull/183/commits/8ea732a7c65f873d48526cb1574e3c137dd66f74

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/38a31657-5073-46c1-8726-8cec39652dd8n%40googlegroups.com.

Re: [go-nuts] runtime.AddCleanup and C struct hierarchies: cleanup order?

[twp...@gmail.com]

Thank you for the super-fast and authoritative response!

OK, I will work on an alternative solution for cleaning up hierarchies in 
bottom-up order and will report back here if/when I find a good solution.

Regards,
Tom

On Wednesday, February 19, 2025 at 10:20:22 PM UTC+1 Ian Lance Taylor wrote:

> On Wed, Feb 19, 2025 at 1:12 PM twp...@gmail.com  wrote:
> >
> > The documentation for runtime.AddCleanup says:
> >
> > > There is no specified order in which cleanups will run.
> >
> > Given the following types:
> >
> > type Parent struct {
> > parentResource int
> > }
> >
> > type Child struct {
> > parent *Parent
> > childResource int
> > }
> >
> > and the following code:
> >
> > parentResource := 0
> > parent := &Parent{
> > parentResource: parentResource,
> > }
> > runtime.AddCleanup(parent, func (int) {}, parentResource)
> > childResource := 1
> > child := &Child{
> > parent: parent,
> > childResource: childResource
> > }
> > runtime.AddCleanup(child, func(int) {}, childResource)
> >
> > is it guaranteed that the cleanup for childResource will run before the 
> cleanup for parentResource?
>
> No. There is no specified order in which cleanups will run.
>
> The fact that child points to parent doesn't really matter here. They
> can both be collected as garbage in the same GC pass, and that GC pass
> may schedule both cleanups. There is no guarantee as to which cleanup
> will be scheduler first or which will run first.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/7ecad685-5ca6-4db6-b09d-bdce8f60d7can%40googlegroups.com.

[go-nuts] Re: golang.org/x/crypto/x509roots/fallback: high, unskippable, init cost

[twp...@gmail.com]

Gentle ping on this.

If this is not wanted, then say. Otherwise, I will create an issue on 
https://github.com/golang/go to reach out to the relevant folk directly.

Regards,
Tom

On Wednesday, April 23, 2025 at 1:11:22 AM UTC+2 twp...@gmail.com wrote:

> tl;dr importing golang.org/x/crypto/x509roots/fallback adds ~8ms to the 
> startup time of every program or library that imports it. I would like this 
> cost to be zero and would be happy to contribute a fix.
>
> Running the following program:
>
> package main
> import _ "golang.org/x/crypto/x509roots/fallback"
> func main() {}
>
> with the command:
>
> $ go build -o tmp main.go
> $ GODEBUG=inittrace=1 ./tmp |& awk '{print $5, $6, $2}' | sort -n | 
> tail -n 3
> 0.043 ms internal/godebug
> 0.052 ms runtime
> 8.8 ms golang.org/x/crypto/x509roots/fallback
>
> shows that golang.org/x/crypto/x509roots/fallback adds an 9ms start-up 
> cost on a fast CPU (AMD Ryzen 8700G). This cost is paid whether or not the 
> fallback X.509 certificates are used or not.
>
> From looking at the code 
> <https://go.googlesource.com/crypto/+/refs/tags/v0.37.0/x509roots/fallback/>, 
> it seems that the CPU is cost is due to parsing about 150 certificates in 
> .PEM format in the package's unskippable init function.
>
> The obvious, easy, backwards-compatible fix would be to parse the .PEM 
> files while executing the template that generates bundle.go, instead of 
> parsing them every time at startup.
>
> Would you accept a CL to fix this?
>
> Tom
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/a9694f82-1dcd-4eb3-8c5f-874c990363a4n%40googlegroups.com.