Re: [go-nuts] SetSafe default value in mgo session

[Serhat Şevki Dinçer]

There is the difference of (not) calling 
https://docs.mongodb.com/v3.0/reference/method/db.getLastError/

This is possibly a better place to ask this:
https://groups.google.com/forum/m/#!forum/mgo-users

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] GC problem

[Serhat Şevki Dinçer]

Hi,
I have a gc.go:

package main

import (
"fmt"
"runtime/debug"
)

const N = 3e8

func f(x int) {
ls := make([]uint64, N)
for i := N - 1; i >= 0; i-- {
ls[i] = uint64(i)
}
fmt.Println(ls[x])
}

func main() {
debug.SetGCPercent(10)
f(8)
f(9)
}

1st call suscessfully returns, I have enough ram on my laptop, but the 2nd 
call fails with "fatal error: runtime: out of memory"
The output is in gc.txt. So what's happening ?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
8
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x4a726c, 0x16)
/Program/go/src/runtime/panic.go:596 +0x95
runtime.sysMap(0xc4af1e, 0x8f0e, 0x0, 0x515a98)
/Program/go/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0x4fd9c0, 0x8f0e, 0x27)
/Program/go/src/runtime/malloc.go:428 +0x374
runtime.(*mheap).grow(0x4fd9c0, 0x47869, 0x0)
/Program/go/src/runtime/mheap.go:774 +0x62
runtime.(*mheap).allocSpanLocked(0x4fd9c0, 0x47869, 0x1)
/Program/go/src/runtime/mheap.go:678 +0x44f
runtime.(*mheap).alloc_m(0x4fd9c0, 0x47869, 0x1, 0x)
/Program/go/src/runtime/mheap.go:562 +0xe2
runtime.(*mheap).alloc.func1()
/Program/go/src/runtime/mheap.go:627 +0x4b
runtime.systemstack(0x7ffc67c18f20)
/Program/go/src/runtime/asm_amd64.s:343 +0xab
runtime.(*mheap).alloc(0x4fd9c0, 0x47869, 0x101, 0x42e8c1)
/Program/go/src/runtime/mheap.go:628 +0xa0
runtime.largeAlloc(0x8f0d1800, 0xc42001c001, 0xc421a0)
/Program/go/src/runtime/malloc.go:795 +0x93
runtime.mallocgc.func1()
/Program/go/src/runtime/malloc.go:690 +0x3e
runtime.systemstack(0x4fab00)
/Program/go/src/runtime/asm_amd64.s:327 +0x79
runtime.mstart()
/Program/go/src/runtime/proc.go:1132

goroutine 1 [running]:
runtime.systemstack_switch()
/Program/go/src/runtime/asm_amd64.s:281 fp=0xc420028628 sp=0xc420028620
runtime.mallocgc(0x8f0d1800, 0x489da0, 0xc420028701, 0x1)
/Program/go/src/runtime/malloc.go:691 +0x930 fp=0xc4200286c8 
sp=0xc420028628
runtime.makeslice(0x489da0, 0x11e1a300, 0x11e1a300, 0x2, 0x0, 0x0)
/Program/go/src/runtime/slice.go:54 +0x7b fp=0xc420028718 
sp=0xc4200286c8
main.f(0x9)
/Dev/go/src/tools/gc.go:11 +0x43 fp=0xc420028768 sp=0xc420028718
main.main()
/Dev/go/src/tools/gc.go:21 +0x44 fp=0xc420028788 sp=0xc420028768
runtime.main()
/Program/go/src/runtime/proc.go:185 +0x20a fp=0xc4200287e0 
sp=0xc420028788
runtime.goexit()
/Program/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4200287e8 
sp=0xc4200287e0
exit status 2

[go-nuts] Shared stdlib questions

[Serhat Şevki Dinçer]

Hi,
On Manjaro 64-bit with go 1.9.3 I am doing:
go install -gcflags '-B -s' -ldflags '-s -w' -buildmode shared std

I get two warnings many times:
go/src/unicode/casetables.go:17:11: redundant type: CaseRange
go/src/vendor/golang_org/x/net/http2/hpack/tables.go:131:13: redundant 
type: HeaderField
What does that mean?

Also I want to run go tests against that shared stdlib I just built (25 
mb). Neither of these seem to be what I need:
go test std
go test -linkshared std
How do I run go tests against libstd.so?

Thanks..

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] sync/atomic suggestion

[Serhat Şevki Dinçer]

Hi,

I would like to update a counter up to a certain limit atomically, without 
using a channel or mutex. What do you think of the following (equivalent) 
functions for sync/atomic?

func CompareAndInc(*addr, Max) bool {

  if *addr < Max {
*addr++
return true
  }
  return false
}

func CompareAndDec(*addr, Min) bool {

  if *addr > Min {
*addr--
return true
  }
  return false
}

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Channel Feature Suggestion

[Serhat Şevki Dinçer]

Hi,

Say you have a buffered channel
ch := make(chan int, 10)

Would it make sense and be easy to add the following directives to the Go 
language core?

waitempty(ch)
blocks caller until ch buffer is empty

waitfull(ch)
blocks caller until ch buffer is full

This means there are 3 types of goroutines related to a channel:
- senders
- receivers
- observers

There are nice possibilities with these, for example wait groups, resource 
usage alerts etc.

What do you think?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Channel Feature Suggestion

[Serhat Şevki Dinçer]

Hi rog,

As -> and <- are 'not racy', because they are 'core' language features, 
because runtime handles them, because they are guaranteed to 'work', I am 
asking how easy and useful to implement them as 'core' language features.

Thank you..

On Thursday, February 21, 2019 at 9:04:22 PM UTC+3, rog wrote:
>
> As others have said, such operations are inherently racy - when you 
> receive the message that a channel is empty, it may not be empty any more. 
> For that reason, it's not a good idea to add them as primitives to the 
> language.
>
> That said, it's not too hard to implement them yourself. We are talking 
> about buffered channels here, so an extra item being buffered should be OK. 
> So you can insert a goroutine in between sender and receiver that signals 
> when it can't receive on a channel (it's empty) or when it can't send on a 
> channel (it's full).
>
> Here's some code that implements those operations:
>
> https://play.golang.org/p/2xyZnUI0imX
>
> Note that there are many other possible variants. For example, you might 
> want to send on the notification channel only when the channel has been 
> idle for some time.
>
>   cheers,
> rog.
>
>
> On Thu, 21 Feb 2019 at 13:38, Serhat Şevki Dinçer  > wrote:
>
>> Hi,
>>
>> Say you have a buffered channel
>> ch := make(chan int, 10)
>>
>> Would it make sense and be easy to add the following directives to the Go 
>> language core?
>>
>> waitempty(ch)
>> blocks caller until ch buffer is empty
>>
>> waitfull(ch)
>> blocks caller until ch buffer is full
>>
>> This means there are 3 types of goroutines related to a channel:
>> - senders
>> - receivers
>> - observers
>>
>> There are nice possibilities with these, for example wait groups, 
>> resource usage alerts etc.
>>
>> What do you think?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Channel Feature Suggestion

[Serhat Şevki Dinçer]

btw waitempty(ch) does not return any value, and it does not (have to) 
guarantee that ch stays empty when it returns..

On Thursday, February 21, 2019 at 11:10:45 PM UTC+3, Serhat Şevki Dinçer 
wrote:
>
> Burak, I think you dont get the potential of the suggesred directives.
>
> You can be interested in waitepty(ch) for example when:
> - you have direct control of who is pushing and pulling on it, and you 
> know what an empty buffer means
> buffer could be "things to do or process", and you have a very 
> easy-to-setup observer to alert some routine that:
> - you are out of job, or
> - you have too many to process
>
> Also what you suggest as raciness is irrelevant: 
>
> for { 
>   x := <- ch { 
> // here, do you have a guarantee that you can put x back to the 
> channel ?? 
>   } 
> } 
>
> On Thursday, February 21, 2019 at 10:57:43 PM UTC+3, Burak Serdar wrote:
>>
>> You can implement waitempty(ch) without a race condition. However, any 
>> scenario in which you use waitempty(ch) would have a race condition, 
>> because the moment it returns there is no guarantee on the state of 
>> the channel. So in a piece of code like this: 
>>
>> waitempty(ch) 
>> post 
>>
>>
>> the only guarantee you have is that when 'post' is running, ch was 
>> empty at least once since waitempty was called. 
>>
>> This code: 
>>
>> for { 
>>   if waitempty(ch) { 
>> ... 
>>   } 
>> } 
>>
>> is racy, because when you go into the if block, you have no idea how 
>> many times ch was empty.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Channel Feature Suggestion

[Serhat Şevki Dinçer]

On Thursday, February 21, 2019 at 11:21:11 PM UTC+3, Jan Mercl wrote:
>
> But then calling it or not has the exact same semantics. So what's it even 
> good for?
>

Let's take your case Jan:

func consumer() {
select { 
case x := <-ch:
// received, consume x
default:
// empty ch, sleep or poll
}
}   

func producer() {
// produce x
select {
case ch <- x:
// send successful, on to a new x
default:
// full ch, sleep or poll, try to resend
}
}

With the directives, you can do the following 'without disturbing the 
processing pipeline' :

func consumer() {
for {
x := <-ch
// consume x
}
}   

func producer() {
for {
// produce x
ch <- x
}
}

func observer1() {
for {
waitempty(ch)
// not enough producers, say, by some rate or counter, last empty 
buffer time, empty buffer occorance etc.
// create anotherproducer
   go producer() // could be up to a max number of producers
}
}

func observer2() {
for {
waitfull(ch)
// not enough consumers, say, by some rate or counter etc.
// create another one
go consumer() // could be up to a max number of consumers
}
}

I think that will be much more efficient, because you dont 'disturb the 
processing pipeline'.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Channel Feature Suggestion

[Serhat Şevki Dinçer]

Another use case is wait groups with Max number of goroutines allowed:

ch:= make(chan bool, 10)

func waiter() {
  ch <-true
  go worker()

  // when empty, all jobs are finished
  waitempty(ch)
}

func worker() {
  // do work

  Select {
  ch <- true:
go worker() // try to handover some jobs
  default:
// max goroutine limit
// do them yourself
  } 

  // do remaining jobs

  // make way
  <- ch
  return
}

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go 1.12.1 and Go 1.11.6 are released

[Serhat Şevki Dinçer]

Hi,

I see a regression on speed with sorty  
tests (go test -short -gcflags '-B -s' -ldflags '-s -w') on my Intel Core 
i5-4210M laptop (also sortutil became faster, zermelo float became much 
slower):

with *go 1.12.1*

Sorting uint32
sortutil took 18.64s
zermelo took 10.92s
sorty-2 took 17.10s
sorty-3 took 14.22s
sorty-4 took 12.36s
sorty-5 took 12.10s

Sorting float32
sortutil took 18.03s
zermelo took 14.57s
sorty-2 took 19.27s
sorty-3 took 15.82s
sorty-4 took 13.93s
sorty-5 took 13.90s

with *go 1.11.6* (consistent with 1.11.5)

Sorting uint32
sortutil took 25.18s
zermelo took 10.93s
sorty-2 took 15.85s
sorty-3 took 13.05s
sorty-4 took 11.27s
sorty-5 took 11.05s

Sorting float32
sortutil took 23.69s
zermelo took 8.89s
sorty-2 took 19.25s
sorty-3 took 15.42s
sorty-4 took 13.19s
sorty-5 took 13.09s


On Friday, March 15, 2019 at 12:13:07 AM UTC+3, Katie Hockman wrote:
>
> Hello gophers,
>
> We have just released Go versions 1.12.1 and 1.11.6, minor point releases.
>
> These releases include fixes to cgo, the compiler, the go command,
>
> and the fmt, net/smtp, os, path/filepath, sync, and template packages.
>
> View the release notes for more information:
> https://golang.org/doc/devel/release.html#go1.12.minor
>
> You can download binary and source distributions from the Go web site:
> https://golang.org/dl/
>
> To compile from source using a Git clone, update to the release with
> "git checkout go1.12.1" and build as usual.
>
> Thanks to everyone who contributed to the release.
>
> Cheers,
> Katie for the Go team
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] sorty tuning on non-X86 platforms

[Serhat Şevki Dinçer]

Hi,

If you have access to non-x86 platforms (ppc, mips, etc), I would like to 
know tuned parameter values and performance info of sorty 
 on those platforms.
Readme file has info on how to run the tests. I would appreciate if you 
paste test outputs here.

Thanks a lot..

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] A Measure of Hash Function Collisions

[Serhat Şevki Dinçer]

On Tuesday, February 5, 2019 at 12:29:32 AM UTC+3, Michael Jones wrote:

> I recently did just this in an effort (successful!) to make a well-known 
> simple hash function be its best with minor single CPU cycle changes. 
>

yes I am told 15 is not the best shift amount, how about this?
x = x<<27 ^ x>>37

the upper limit of length sum could be 9, because hashes of 9-byte valid 
utf8 strings are less than 255^9, on average ~247 per uin64 value, so one 
such uint64 value most likely will be zero which is the hash of empty 
string. so you get a length sum of 9. can you bring it down? or can we 
"design" a hash to guarantee 9?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: [ANN] sorts - parallel quicksort and radix sort

[Serhat Şevki Dinçer]

Hi,

I needed a concurrent sort for a huge integer list, wrote a simple one and 
found out about yours. Mine (sorty) does not:
- implement sort.Interface
- limit number of goroutines
- implement all common types, just uint64

Here is the go test output (sorting random 2^27 uint64s) on my laptop:

sort.Slice took 33.147487459s
sorty took 6.984276576s
sortutil took 13.109776227s
PASS
ok  sorty64.500s

Here is the code: https://github.com/jfcg/sorty
Is the speed difference because of a lack of interface or unlimited number 
of goroutines? Do you think this is a good approach?

Thanks..

Note: to use it, assign to sorty.Ar and then call sorty.Sort() :P

On Sunday, May 3, 2015 at 11:37:52 PM UTC+3, Randall Farmer wrote:
>
> https://github.com/twotwotwo/sorts/ provides parallel radix- and 
> quicksorts for faster sorting of large datasets. (This is the stable 
> replacement for the radixsort.test package I posted some weeks back.)
>
> On the right sort of task, it can help a lot: sorting 11m English strings, 
> it saved 38% wall time on one core, 59% on two cores, and and 80% on eight 
> cores. On the stdlib's synthetic Benchmark1e6, ns/op was 71% lower on one 
> core and 80% lower on two. (Random-ish data like Benchmark1e6's is kind of 
> a best-case scenario for radix sort, so YMMV there.) More timings are at 
> https://docs.google.com/spreadsheets/d/1GkXMLXQ7oW5Bp0qwyYw0IiQElIq8B-IvNEYE_RPCTvA/edit
>  
> .
>
> The package has 100% test coverage (when tested with -cpu 2) and checks 
> its output is sorted after each run. There's no unsafe or cgo or asm. 
> Parallel/radix sorts aren't used on smaller collections where they don't 
> help. To limit or turn off concurrency in general, set sorts.MaxProcs. The 
> sorts/sortutil package has shortcuts for common slice types like stdlib 
> sort's (e.g., Ints(data []int)).
>
> Mostly, you should stick to stdlib sort, which is fast, standard, and 
> flexible, with a cleaner API. But I'm sure some folks out there have legit 
> use cases where package sorts's gains are worth the hassle, and this is for 
> them.
>
> Docs are at http://godoc.org/github.com/twotwotwo/sorts . E-mail me or 
> holler on Twitter (@rf) if you're using it, or just to say hey.
>
> Best,
> Randall
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] A Measure of Hash Function Collisions

[Serhat Şevki Dinçer]

On Saturday, February 9, 2019 at 5:23:14 PM UTC+3, Serhat Şevki Dinçer 
wrote:
>
> On Fri, Feb 8, 2019 at 7:42 PM Michael Jones wrote: 
> > clustering: 
> > https://www.cs.cornell.edu/courses/cs3110/2014fa/lectures/13/lec13.html 
> > 
> > careful hash functions often treat short inputs specially. 
> > 
> > iterated shift-xor alone is weak in expanding the "changed bits(s)" 
> signal, at least by comparison to a) large prime multiply, b) good s-boxes, 
> c) introduction of keyed material. 
> Hm, thanks. I would like to try a particular version of this prime 
> multiplication idea wih utf8 strings. 
> I wrote for loops to find collisions (attached). It takes 3 seconds 
> and finds no collision for strings with length < 4. 
> The next step (including length=4, commented in the code) will take 
> 13+ min and 32+ gb ram, so I appreciate if anyone with sufficient RAM 
> could try that and report the result ;P 
> Thanks.. 
>

I got access to a server with 64 gb ram. On it, using a Go map did not 
work, so I used a list and sorted it for identifying collisions.
It turned out both prime and shift versions (attached) of the simple hash 
turned out to be really good for small inputs. No collisions for 
7_918_845_952 inputs.
This required 59 GiB of ram. For a 64-bit cryptographic hash output, the 
probability of a collision for that many inputs is about %82.

What I am curious about next is
- How further can this test be taken ? When are the first collisions for 
these simple hashes with the given code ?
- What are their "minimum sum of colliding inputs lengths" ?
- Is there a (smooth) trade-off between being cryptographic / 
good-bit-mixing *and* being nice on small inputs / high minimum sum of 
colliding inputs lengths ? Assume that we dont treat small inputs 
differently, and there is one general algorithm for all inputs..
- Can a cryptographic hash function have high minimum sum of colliding 
inputs lengths, or be nice on small inputs ?

Thanks..

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


txt2int2.go
Description: Binary data

Re: [go-nuts] A Measure of Hash Function Collisions

[Serhat Şevki Dinçer]

On Tuesday, February 12, 2019 at 9:51:17 PM UTC+3, Michael Jones wrote:
>
> Serhat, some ideas for you...
> https://play.golang.org/p/7QPy5wa-9eO
>

Interestingly I found out input iteration order does matter :) 15,33 shift 
version yields an amazing number of collisions (7_910_886_368 collisions 
over 7_918_845_952 inputs) when fed with a spesific 6-byte sequence 
(attached).
rotate-27 version interestingly gave no collisions for this case. prime 
version also had no collisions. prime version with ^= and -= instead of += 
also gave no collision.

I want to identify a collision on the prime version (any variant ^= -= +=) 
with small inputs, say sum of lengths <= 12. Does anyone have any idea how 
to identify such inputs?
So far prime version seems to be the most promising (in terms of high 
minimum sum of collding inputs)..

Thanks..

Note: Thanks Damian, it looks like an advanced test suite for cryto hashes, 
though text input tests seems limited.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


txt2int3.go
Description: Binary data

[go-nuts] A Measure of Hash Function Collisions

[Serhat Şevki Dinçer]

Hi,

What is the minimum sum of input lengths len(s1)+len(s2) such that:

- s1, s2 are distinct utf8 strings
- Txt2int(s1) = Txt2int(s2)

for the below simple hash function ?

func Txt2int(s string) uint64 {
x := uint64(len(s))
for i := len(s) - 1; i >= 0; i-- {
x = x<<15 ^ x>>49
x += uint64(s[i])
}
return x
}

In general, how good is "minimum sum of colliding input lengths" as a measure 
of collision-resistance for a (not necessarily cryptographic) hash function ?

Thanks..

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Clarification on unsafe conversion between string <-> []byte

[Serhat Şevki Dinçer]

Hi,

I wrote a string utility library  with many 
tests to ensure the conversion assumptions are correct. It could be helpful.

Cheers..

On Wednesday, September 18, 2019 at 12:42:31 PM UTC+3, Francis wrote:
>
> I am looking at the correct way to convert from a byte slice to a string 
> and back with no allocations. All very unsafe.
>
> I think these two cases are fairly symmetrical. So to simplify the 
> discussion below I will only talk about converting from a string to []byte.
>
> func StringToBytes(s string) (b []byte)
>
> From what I have read it is currently not clear how to perform this 
> correctly.
>
> When I say correctly I mean that the function returns a `[]byte` which 
> contains all of and only the bytes in the string and never confuses the 
> garbage collector. We fully expect that the `[]byte` returned will contain 
> the same underlying memory as the string and modifying its contents will 
> modify the string, making the string dangerously mutable. We are 
> comfortable with the dangerously mutable string.
>
> Following the directions in unsafe you _might_ think that this would be a 
> good solution.
>
> func StringToBytes(s string) []byte {
> return *(*[]byte)(unsafe.Pointer({
> Data: (*(*reflect.StringHeader)(unsafe.Pointer())).Data,
> Len:  len(s),
> Cap:  len(s),
> }))
> }
>
> The line
>
> Data: (*(*reflect.StringHeader)(unsafe.Pointer())).Data,
>
> here is a really carefully commented example of this approach from github
>
> seems to satisfy unsafe  rule 5 about 
> converting uintptr to and from unsafe.Pointer in a singe expression.
>
> However, it clearly violates rule 6 which states `...SliceHeader and 
> StringHeader are only valid when interpreting the content of an actual 
> slice or string value.`. The `[]byte` we are returning here is built from a 
> `` and not based on an existing `[]byte`.
>
> So we can switch to
>
> func StringToBytes(s string) (b []byte) {
> stringHeader := (*reflect.StringHeader)(unsafe.Pointer())
> sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer())
> sliceHeader.Data = stringHeader.Data
> sliceHeader.Len = stringHeader.Len
> sliceHeader.Cap = stringHeader.Len
> return b
> }
>
> Now we are using an existing []byte to build `sliceHeader` which is good. 
> But we end up with a new problem. sliceHeader.Data and stringHeader.Data 
> are both uintptr. So by creating them in one expression and then writing 
> them in another expression we violate the rule that `uintptr cannot be 
> stored in variable`.
>
> There is a possible sense that we are protected because both of our 
> `uinptr`s are actually real pointers inside a real string and []byte. This 
> seems to be indicated by the line `In this usage hdr.Data is really an 
> alternate way to refer to the underlying pointer in the string header, not 
> a uintptr variable itself.`
>
> This feels very unclear to me.
>
> In particular the code example in the unsafe package
>
> var s string
> hdr := (*reflect.StringHeader)(unsafe.Pointer()) // case 1
> hdr.Data = uintptr(unsafe.Pointer(p))  // case 6 (this case)
> hdr.Len = n
>
>
> is not the same as the case we are dealing with here. Specifically in the 
> unsafe package documentation we are writing from a uintpr stored in a 
> separate variable to another uinptr. They are probably very similar in 
> practice, but it isn't obvious and in my experience subtly incorrect code 
> often comes from relying on vague understandings of important documents.
>
> If we assume that our uinptrs are safe because they are backed by real 
> pointers then there is another issue with our string being garbage 
> collected.
>
> func StringToBytes(s string) (b []byte) {
> stringHeader := (*reflect.StringHeader)(unsafe.Pointer())
> // Our string is no longer referenced anywhere and could 
> potentially be garbage collected
> sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer())
> sliceHeader.Data = stringHeader.Data
> sliceHeader.Len = stringHeader.Len
> sliceHeader.Cap = stringHeader.Len
> return b
> }
>
>
> There is a discussion where this potential problem is raised
>
> https://github.com/golang/go/issues/25484
>
> we also see this issue mentioned in
>
> https://groups.google.com/forum/#!msg/golang-nuts/dcjzJy-bSpw/tcZYBzQqAQAJ
>
> The solution of 
>
> func StringToBytes(s string) (b []byte) {
> stringHeader := (*reflect.StringHeader)(unsafe.Pointer())
> // Our string is no longer referenced anywhere and could 
> potentially be garbage collected
> sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer())
> sliceHeader.Data = stringHeader.Data
> sliceHeader.Len = stringHeader.Len
> sliceHeader.Cap = stringHeader.Len
>  runtime.KeepAlive()
> return b
> }
>
>
> is proposed. 

[go-nuts] Re: GOMAXPROCS > num of CPU

[Serhat Şevki Dinçer]

In runtime it says:

The GOMAXPROCS variable limits the number of operating system threads that 
can execute user-level Go code simultaneously. There is no limit to the 
number of threads that can be blocked in system calls on behalf of Go code; 
those do not count against the GOMAXPROCS limit. This package's GOMAXPROCS 
function queries and changes the limit.

So normally it does not make sense to increase it beyond available physical 
"HW threads" (I think this is what you meant with cpu) count (since blocked 
threads do not count towards this limit).
As long as "active thread accounting" is accurate in the OS, I dont see any 
reason to set it higher. I think it is easy to test >HWthreads effects with 
a concurrent cpu-intensive job.

On Wednesday, December 4, 2019 at 8:24:13 PM UTC+3, Vincent Blanchon wrote:
>
> Hello,
>
>
> I've read on GitHub (
> https://github.com/golang/go/issues/20303#issuecomment-329418911) "there 
> are good reasons" to set GOMAXPROCS to > num CPU.
> Just out of curiosity, I want to know if someone has an example of it or 
> any good reason to set it up more than the number of CPUs?
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7246cd08-f340-4120-875d-7fca069ac8f3%40googlegroups.com.

[go-nuts] module dependency & testing

[Serhat Şevki Dinçer]

Hi,

Say you have a project with:
- hard dependencies
- some extra dependencies only used during testing, not needed otherwise

Are they treated equally? When you switch to go modules, does the second 
group become hard dependencies? It seems like they do.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f640cd0f-c569-4f19-9add-d63e53875bc3%40googlegroups.com.

[go-nuts] [ANN] sorty

[Serhat Şevki Dinçer]

Hi everyone,

I've been improving type-specific concurrent sorting library sorty 
 for a while and the results seem really 
competitive on different data types for such a small library (effective 
code is less than 200 lines). It also implements sort.Interface, though no 
stable sorting or worst case guarantees yet.

Let me know what you think. I am especially interested on sorting timings 
on non-x86 platforms.

Thanks..

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d555d564-2669-44a7-b41a-cd40cec9aeb8%40googlegroups.com.

[go-nuts] Re: Clarification on unsafe conversion between string <-> []byte

[Serhat Şevki Dinçer]

Turns out it did not really need uintptr so I switched to unsafe.Pointer. 
Also added more tests.

Thanks Francis..

On Friday, October 4, 2019 at 12:38:22 PM UTC+3, fra...@adeven.com wrote:
>
> Serhat,
>
> That implementation looks very tidy. But it still uses uintptr. So it 
> doesn't solve the GC problems discussed above.
>
> On Thursday, September 26, 2019 at 10:59:08 PM UTC+2, Serhat Şevki Dinçer 
> wrote:
>>
>> Hi,
>>
>> I wrote a string utility library <https://github.com/jfcg/sixb> with 
>> many tests to ensure the conversion assumptions are correct. It could be 
>> helpful.
>>
>> Cheers..
>>
>> On Wednesday, September 18, 2019 at 12:42:31 PM UTC+3, Francis wrote:
>>>
>>> I am looking at the correct way to convert from a byte slice to a string 
>>> and back with no allocations. All very unsafe.
>>>
>>> I think these two cases are fairly symmetrical. So to simplify the 
>>> discussion below I will only talk about converting from a string to []byte.
>>>
>>> func StringToBytes(s string) (b []byte)
>>>
>>> From what I have read it is currently not clear how to perform this 
>>> correctly.
>>>
>>> When I say correctly I mean that the function returns a `[]byte` which 
>>> contains all of and only the bytes in the string and never confuses the 
>>> garbage collector. We fully expect that the `[]byte` returned will contain 
>>> the same underlying memory as the string and modifying its contents will 
>>> modify the string, making the string dangerously mutable. We are 
>>> comfortable with the dangerously mutable string.
>>>
>>> Following the directions in unsafe you _might_ think that this would be 
>>> a good solution.
>>>
>>> func StringToBytes(s string) []byte {
>>> return *(*[]byte)(unsafe.Pointer({
>>> Data: 
>>> (*(*reflect.StringHeader)(unsafe.Pointer())).Data,
>>> Len:  len(s),
>>> Cap:  len(s),
>>> }))
>>> }
>>>
>>> The line
>>>
>>> Data: (*(*reflect.StringHeader)(unsafe.Pointer())).Data,
>>>
>>> here is a really carefully commented example of this approach from github
>>>
>>> seems to satisfy unsafe <https://golang.org/pkg/unsafe/> rule 5 about 
>>> converting uintptr to and from unsafe.Pointer in a singe expression.
>>>
>>> However, it clearly violates rule 6 which states `...SliceHeader and 
>>> StringHeader are only valid when interpreting the content of an actual 
>>> slice or string value.`. The `[]byte` we are returning here is built from a 
>>> `` and not based on an existing `[]byte`.
>>>
>>> So we can switch to
>>>
>>> func StringToBytes(s string) (b []byte) {
>>> stringHeader := (*reflect.StringHeader)(unsafe.Pointer())
>>> sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer())
>>> sliceHeader.Data = stringHeader.Data
>>> sliceHeader.Len = stringHeader.Len
>>> sliceHeader.Cap = stringHeader.Len
>>> return b
>>> }
>>>
>>> Now we are using an existing []byte to build `sliceHeader` which is 
>>> good. But we end up with a new problem. sliceHeader.Data and 
>>> stringHeader.Data are both uintptr. So by creating them in one expression 
>>> and then writing them in another expression we violate the rule that 
>>> `uintptr cannot be stored in variable`.
>>>
>>> There is a possible sense that we are protected because both of our 
>>> `uinptr`s are actually real pointers inside a real string and []byte. This 
>>> seems to be indicated by the line `In this usage hdr.Data is really an 
>>> alternate way to refer to the underlying pointer in the string header, not 
>>> a uintptr variable itself.`
>>>
>>> This feels very unclear to me.
>>>
>>> In particular the code example in the unsafe package
>>>
>>> var s string
>>> hdr := (*reflect.StringHeader)(unsafe.Pointer()) // case 1
>>> hdr.Data = uintptr(unsafe.Pointer(p))  // case 6 (this case)
>>> hdr.Len = n
>>>
>>>
>>> is not the same as the case we are dealing with here. Specifically in 
>>> the unsafe package documentation we are writing from a uintpr stored in a 
>>> separate variable to another uinptr. They are probably very similar in 
>>> practice, but it isn't obvious and in my experience subtly 

[go-nuts] Re: [ANN] sorty

[Serhat Şevki Dinçer]

Hi,

sorty v0.4.0 is released with:

// Sort3 concurrently sorts underlying collection of length n via
// lesswap() which must be equivalent to:
//  if less(i, k) {
//  if r != s {
//  swap(r, s)
//  }
//  return true
//  }
//  return false
func Sort3(n int, lesswap func(i, k, r, s int) bool)

This function based method is faster than sorty.Collection2, which in turn 
is faster than sort.Interface (just as a way to access generic collections).

Let me know what you think, cheers..

On Wednesday, September 25, 2019 at 10:17:40 PM UTC+3, Serhat Şevki Dinçer 
wrote:
>
> Hi everyone,
>
> I've been improving type-specific concurrent sorting library sorty 
> <https://github.com/jfcg/sorty> for a while and the results seem really 
> competitive on different data types for such a small library (effective 
> code is less than 200 lines). It also implements sort.Interface, though no 
> stable sorting or worst case guarantees yet.
>
> Let me know what you think. I am especially interested on sorting timings 
> on non-x86 platforms.
>
> Thanks..
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8e938595-3d45-437b-a386-e44abe03c095%40googlegroups.com.

[go-nuts] Re: [ANN] sorty

[Serhat Şevki Dinçer]

Hello,

sorty <https://github.com/jfcg/sorty> v1.0 was released with:

- Concurrent dual partitioning (for initial long ranges)
- Median-of-5 / 9 pivotting with scheduling
- Sub-range assistive pivotting
- Simplified api (no interfaces)
- Other small improvements

Let me know what you think,
Cheers..

On Friday, January 3, 2020 at 12:11:08 PM UTC+3, Serhat Şevki Dinçer wrote:
>
> Hi,
>
> sorty v0.4.0 is released with:
>
> // Sort3 concurrently sorts underlying collection of length n via
> // lesswap() which must be equivalent to:
> //  if less(i, k) {
> //  if r != s {
> //  swap(r, s)
> //  }
> //  return true
> //  }
> //  return false
> func Sort3(n int, lesswap func(i, k, r, s int) bool)
>
> This function based method is faster than sorty.Collection2, which in turn 
> is faster than sort.Interface (just as a way to access generic collections).
>
> Let me know what you think, cheers..
>
> On Wednesday, September 25, 2019 at 10:17:40 PM UTC+3, Serhat Şevki Dinçer 
> wrote:
>>
>> Hi everyone,
>>
>> I've been improving type-specific concurrent sorting library sorty 
>> <https://github.com/jfcg/sorty> for a while and the results seem really 
>> competitive on different data types for such a small library (effective 
>> code is less than 200 lines). It also implements sort.Interface, though no 
>> stable sorting or worst case guarantees yet.
>>
>> Let me know what you think. I am especially interested on sorting timings 
>> on non-x86 platforms.
>>
>> Thanks..
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9c7bc01d-02d5-49d8-80b1-680cedac5170%40googlegroups.com.