Re: [go-nuts] Change slice type
On Thu, Nov 28, 2019 at 3:16 AM Michael Jones wrote: > > The general rule -- if there can be a general rule for risk behavior -- is > that CPUs like having addresses aligned on an integer multiple of the data > element size. So: > > access-as-byte data may be on any address (address&(1-1)==0), > access as 2-byte data on a multiple of two address (address&(2-1)==0), > access as 4-byte data on a multiple of four address (address&(4-1)==0), > access as 8-byte data on a multiple of eight address (address&(8-1)==0), > access as 16-byte data on a multiple of sixteen address (address&(16-1)==0), It's a bit more complicated for struct fields on 386, some 8 byte data are 4 byte aligned. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAA40n-U7tcWxq65pOGZhk2oLfgR2z3fMpK1Z2un65tmjv9%3DQaw%40mail.gmail.com.
Re: [go-nuts] Change slice type
The general rule -- if there can be a general rule for risk behavior -- is
that CPUs like having addresses aligned on an integer multiple of the data
element size. So:
access-as-byte data may be on any address (address&(1-1)==0),
access as 2-byte data on a multiple of two address (address&(2-1)==0),
access as 4-byte data on a multiple of four address (address&(4-1)==0),
access as 8-byte data on a multiple of eight address (address&(8-1)==0),
access as 16-byte data on a multiple of sixteen address
(address&(16-1)==0),
:
(How this might extend to "SIMD vector registers" of various kinds depends
on the machine--is it based on the individual component's size, the
aggregate size, etc.) There may be an upper limit to this logic of
alignment restriction, so maybe 32-bits or 64-bits is always sufficient.
Based on this thinking, the safest way to abuse Go's implicit type safety
is to tell Go's allocator that you want something with large data elements
(uint64, say) and then use unsafe to view this as 8x as many bytes or 4x as
many uint16s.
this what Ian meant.
On Wed, Nov 27, 2019 at 9:26 AM wrote:
> Hi Ian, thanks for the help... quick question about alignment because i
> may be doing something wrong but it shows me that it doesn't matter for the
> slice type, everything is alignment the same way. Can you comment on this?
>
> On 32 bit it'll always return 4 and for 64 bit - 8... even for char or
> boolean slices
> https://play.golang.org/p/wLbt41sISft
>
> Another question, i create []byte slice, then i create []int slice, in the
> middle of []byte by changing
>
> arr := [10]byte{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
> size := len(arr)
> p := uintptr(unsafe.Pointer(&arr))
>
> var data []int
> sh := (*reflect.SliceHeader)(unsafe.Pointer(&data))
> sh.Data = p + some_alignment_offset_
> sh.Len = size
> sh.Cap = size
>
> Now data points at memory used by arr, when arr will become unreachable...
> will underlying memory be collected by GC? Should i call some function to
> inform GC to not collect the memory?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/7430d6a7-909e-41fd-a292-5de2be9f9733%40googlegroups.com
>
> .
>
--
*Michael T. jonesmichael.jo...@gmail.com *
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CALoEmQwkP%3D%2Bfi8%2BRNQz-YtE5CM9WX4LnfZijfv37N63RxzmzSQ%40mail.gmail.com.
Re: [go-nuts] Change slice type
On Wed, Nov 27, 2019 at 9:26 AM wrote:
>
> Hi Ian, thanks for the help... quick question about alignment because i may
> be doing something wrong but it shows me that it doesn't matter for the slice
> type, everything is alignment the same way. Can you comment on this?
All slices have the same alignment requirement, but slices have an
underlying array, and the alignment requirements of the underlying
array are not the same.
https://play.golang.org/p/s6uFbk-WLkw
> Another question, i create []byte slice, then i create []int slice, in the
> middle of []byte by changing
>
> arr := [10]byte{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
> size := len(arr)
> p := uintptr(unsafe.Pointer(&arr))
That doesn't look right. That will get you the address of the slice.
You need the address of the underlying array. You need to write
p := uintptr(unsafe.Pointer(&arr[0]))
But that is wrong for another reason. This conversion from
unsafe.Pointer to uintptr is not one of the patterns permitted at
https://golang.org/pkg/unsafe. This approach will not work
> var data []int
> sh := (*reflect.SliceHeader)(unsafe.Pointer(&data))
> sh.Data = p + some_alignment_offset_
> sh.Len = size
> sh.Cap = size
>
> Now data points at memory used by arr, when arr will become unreachable...
> will underlying memory be collected by GC? Should i call some function to
> inform GC to not collect the memory?
That is not a problem. Just because arr is unreachable does not mean
that arr's underlying array is unreachable. You could write
p1 := &arr[1]
and then arr could disappear but p1 would still be valid.
Anyhow you need to write this more like (completely untested)
(*[1<<28]int)(unsafe.Pointer((uintptr(unsafe.Pointer(&arr[0])) +
7)&^7)[:size:size]
except that that doesn't handle any alignment issues.
Ian
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUA1LHAzwvyGmfdxrZYBMoXCywX6JANuAp%2BR_5xLJwV4g%40mail.gmail.com.
[go-nuts] Symlink warnings in 'go mod' tool output
I'm experimenting with converting a large source tree into a Go module. This went pretty smoothly. However, I'm wondering why 'go mod tidy' and 'go mod why' print warnings about symlinks. These look like: $ go mod tidy warning: ignoring symlink /path/to/some/symlink/dir ... Where a warning line is printed for every symlinked directory in the repo. These warnings make these 'go mod' commands unpleasant to use (you have to scroll through all the junk to see the relevant bits, if any, at the bottom). None of the Go code is in symlinked directories; these all exist for certain projects in other languages. For example, one use of symlinks is as a cheap way to use certain shared JS/CSS assets in multiple sub-projects. Am I holding this wrong, somehow? Can we remove the prints from the Go tool? Or maybe it could only print the warning if there are .go files somewhere in the symlinked dir? If someone can confirm that I'm using the tools as intended, I'll file a bug for further discussion. Thanks! Caleb -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAGeFq%2BkwZBxB_05%2B2Ca%3DHK7TMXVHN%3DG4X7UKbTcUHNpUPqT_Pg%40mail.gmail.com.
[go-nuts] Re: Unable to lock versions when converting to go.mod
I tried blowing away my entire ~/go tree to be sure. Starting from a fresh checkout (outside the ~/go tree), and a fresh checkout and go mod init ..., here's what the build does: $ go build ./... go: downloading github.com/go-kit/kit v0.5.1-0.20170917202734-0d313fb5fb3a *go: downloading github.com/prometheus/common v0.0.0-20170908161822-2f17f4a9d485* go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.5 go: finding github.com/prometheus/client_golang v1.2.1 go: downloading github.com/prometheus/client_golang v1.2.1 go: finding github.com/aperum/nrpe latest *go: extracting github.com/prometheus/common v0.0.0-20170908161822-2f17f4a9d485* go: downloading github.com/pkg/errors v0.8.1-0.20170910134614-2b3a18b5f0fb go: extracting gopkg.in/alecthomas/kingpin.v2 v2.2.5 go: downloading github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf go: downloading github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc go: downloading github.com/aperum/nrpe v0.0.0-20170524093721-53d9ca02dfca go: extracting github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf go: extracting github.com/pkg/errors v0.8.1-0.20170910134614-2b3a18b5f0fb go: extracting github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc go: extracting github.com/prometheus/client_golang v1.2.1 go: extracting github.com/go-kit/kit v0.5.1-0.20170917202734-0d313fb5fb3a go: extracting github.com/aperum/nrpe v0.0.0-20170524093721-53d9ca02dfca go: downloading github.com/go-stack/stack v1.6.0 go: downloading github.com/go-logfmt/logfmt v0.3.0 go: extracting github.com/go-logfmt/logfmt v0.3.0 go: extracting github.com/go-stack/stack v1.6.0 go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6 go: downloading github.com/go-kit/kit v0.9.0 go: downloading github.com/prometheus/procfs v0.0.5 go: downloading github.com/golang/protobuf v1.3.2 go: downloading github.com/cespare/xxhash/v2 v2.1.0 *go: downloading github.com/prometheus/common v0.7.0* go: downloading github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4 go: extracting gopkg.in/alecthomas/kingpin.v2 v2.2.6 go: extracting github.com/prometheus/procfs v0.0.5 go: extracting github.com/cespare/xxhash/v2 v2.1.0 go: extracting github.com/go-kit/kit v0.9.0 go: extracting github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4 go: downloading github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4 go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 *go: extracting github.com/prometheus/common v0.7.0* go: downloading github.com/beorn7/perks v1.0.1 go: extracting github.com/golang/protobuf v1.3.2 go: downloading github.com/pkg/errors v0.8.1 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1 go: downloading github.com/go-logfmt/logfmt v0.4.0 go: extracting github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4 go: extracting github.com/go-logfmt/logfmt v0.4.0 go: extracting github.com/matttproud/golang_protobuf_extensions v1.0.1 go: extracting github.com/pkg/errors v0.8.1 go: extracting github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 go: extracting github.com/beorn7/perks v1.0.1 go: finding github.com/go-kit/kit v0.9.0 *go: finding github.com/prometheus/common v0.7.0* go: finding gopkg.in/alecthomas/kingpin.v2 v2.2.6 go: finding github.com/go-logfmt/logfmt v0.4.0 go: finding github.com/pkg/errors v0.8.1 go: finding github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4 go: finding github.com/beorn7/perks v1.0.1 go: finding github.com/cespare/xxhash/v2 v2.1.0 go: finding github.com/golang/protobuf v1.3.2 go: finding github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 go: finding github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4 go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1 go: finding github.com/prometheus/procfs v0.0.5 # github.com/RobustPerception/nrpe_exporter ./nrpe_exporter.go:124:37: cannot use &allowedLevel (type *promlog.AllowedLevel) as type *promlog.Config in argument to flag.AddFlags ./nrpe_exporter.go:128:23: cannot use allowedLevel (type promlog.AllowedLevel) as type *promlog.Config in argument to promlog.New So it seems to be downloading both the old versions of packages as they were in the initial go.mod, *and* the newer ones. Here is how go.mod changes: --- go.mod.orig 2019-11-27 22:43:06.884601059 + +++ go.mod 2019-11-27 22:43:20.733075672 + @@ -3,13 +3,15 @@ go 1.13 require ( - github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc - github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf - github.com/go-kit/kit v0.5.1-0.20170917202734-0d313fb5fb3a - github.com/go-logfmt/logfmt v0.3.0 - github.com/go-stack/stack v1.6.0 + github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 + github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4 + github.com/aperum/nrpe v0.0.0-20170524093721-53d9ca02dfca + github.com/go-kit/kit v0.9.0 + g
[go-nuts] Unable to lock versions when converting to go.mod
I am having a problem with go.mod. I'm using go 1.13.4 and you can easily replicate what I'm seeing. I am trying to convert https://github.com/RobustPerception/nrpe_exporter from vendor/vendor.json to go.mod. Here's how I start: $ git clone https://github.com/RobustPerception/nrpe_exporter $ cd nrpe_exporter $ go mod init github.com/RobustPerception/nrpe_exporter go: creating new go.mod: module github.com/RobustPerception/nrpe_exporter go: copying requirements from vendor/vendor.json go: converting vendor/vendor.json: stat github.com/go-kit/log@: unknown revision $ cat go.mod module github.com/RobustPerception/nrpe_exporter go 1.13 require ( github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf github.com/go-kit/kit v0.5.1-0.20170917202734-0d313fb5fb3a github.com/go-logfmt/logfmt v0.3.0 github.com/go-stack/stack v1.6.0 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515 github.com/pkg/errors v0.8.1-0.20170910134614-2b3a18b5f0fb *github.com/prometheus/common v0.0.0-20170908161822-2f17f4a9d485* gopkg.in/alecthomas/kingpin.v2 v2.2.5 ) That looks fine up to this point. Note in particular how a specific old version of prometheus/common from 2017 has been selected - highlighted. However, the next step is to type "go build ./...", and as soon as I do that, the dependencies are changed to point to newer versions of those libraries. $ go build ./... go: finding github.com/aperum/nrpe latest # github.com/RobustPerception/nrpe_exporter ./nrpe_exporter.go:124:37: cannot use &allowedLevel (type *promlog.AllowedLevel) as type *promlog.Config in argument to flag.AddFlags ./nrpe_exporter.go:128:23: cannot use allowedLevel (type promlog.AllowedLevel) as type *promlog.Config in argument to promlog.New $ cat go.mod module github.com/RobustPerception/nrpe_exporter go 1.13 require ( github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4 github.com/aperum/nrpe v0.0.0-20170524093721-53d9ca02dfca github.com/go-kit/kit v0.9.0 github.com/go-logfmt/logfmt v0.4.0 github.com/go-stack/stack v1.8.0 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515 github.com/pkg/errors v0.8.1 github.com/prometheus/client_golang v1.2.1 *github.com/prometheus/common v0.7.0* gopkg.in/alecthomas/kingpin.v2 v2.2.6 ) You can see that go.mod has been updated to point to the newest version of prometheus/common - and that is why the build is failing (since the promlog API in prometheus/common has changed). I could update the code to make use of the newer dependencies. But really I'm confused: surely the point of go.mod / go.sum is to be able to refer to specific versions of libraries you depend on, so that you *don't* get caught out by this sort of problem? Obviously I'm missing something here, so I'd be very grateful if someone could enlighten me. What would I need to do to have "go build" use the dependencies listed in go.mod, rather than updating them? (I tried -mod=readonly, but then it just fails saying it needs to update them) Many thanks, Brian. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5f27912d-2f31-45ad-b7fa-b74c8ad7f29a%40googlegroups.com.
Re: [go-nuts] Change slice type
Hi Ian, thanks for the help... quick question about alignment because i may
be doing something wrong but it shows me that it doesn't matter for the
slice type, everything is alignment the same way. Can you comment on this?
On 32 bit it'll always return 4 and for 64 bit - 8... even for char or
boolean slices
https://play.golang.org/p/wLbt41sISft
Another question, i create []byte slice, then i create []int slice, in the
middle of []byte by changing
arr := [10]byte{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
size := len(arr)
p := uintptr(unsafe.Pointer(&arr))
var data []int
sh := (*reflect.SliceHeader)(unsafe.Pointer(&data))
sh.Data = p + some_alignment_offset_
sh.Len = size
sh.Cap = size
Now data points at memory used by arr, when arr will become unreachable...
will underlying memory be collected by GC? Should i call some function to
inform GC to not collect the memory?
Thanks
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/7430d6a7-909e-41fd-a292-5de2be9f9733%40googlegroups.com.
[go-nuts] Re: Is anyone aware of a blocking ring buffer implementation?
https://github.com/glycerine/rbuf very hardened. Implements Reader and Writer. You would have to add your desired blocking strategy. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/88a0325b-9225-4753-89a3-0f2e56d4cb68%40googlegroups.com.
