Re: Picky, but much more efficient arc4random_uniform!

2022-05-21 Thread Theo de Raadt
We will not be using your code. End of story. Luke Small wrote: > Marc, all you all have to do is say is that you all refuse to provide it.

Re: Picky, but much more efficient arc4random_uniform!

2022-05-21 Thread Luke Small
Marc, all you all have to do is say is that you all refuse to provide it. I was asked to at least provide evidence for correctness. I did so; and I’d say I did a stellar job aside from getting some kind of statistical program. The following has an attached source code for my test (along with

Re: Picky, but much more efficient arc4random_uniform!

2022-05-21 Thread Marc Espie
On Sat, May 21, 2022 at 07:47:40AM -0500, Luke Small wrote: > Perhaps it was rude sending off list stuff to the list. Your email sounded > "less than friendly" and more of a professional challenge that you were > definitely in the works to produce; much like Damien Miller’s challenge to > prove

Re: Picky, but much more efficient arc4random_uniform!

2022-05-21 Thread Luke Small
Perhaps it was rude sending off list stuff to the list. Your email sounded "less than friendly" and more of a professional challenge that you were definitely in the works to produce; much like Damien Miller’s challenge to prove correctness. So, whatever. Aside from that unpleasantness: I worked

Re: Picky, but much more efficient arc4random_uniform!

2022-05-21 Thread Crystal Kolipe
On Fri, May 20, 2022 at 09:48:28PM -0500, Luke Small wrote: > Crystal: You can prove that for random, repetitive, correct, database > record name generation using small upperbounds, the demonstrated 1/3-1/2 > runtime isn???t worth it for an upperbound like 26 - 92 in a business context > that

Re: Picky, but much more efficient arc4random_uniform!

2022-05-20 Thread Luke Small
Crystal: You can prove that for random, repetitive, correct, database record name generation using small upperbounds, the demonstrated 1/3-1/2 runtime isn’t worth it for an upperbound like 26 - 92 in a business context that fights for every last millisecond? Bring it. Prove the correctness of

Re: Picky, but much more efficient arc4random_uniform!

2022-05-20 Thread Luke Small
I appreciate your response, Damien. I did do the bare minimum of correctness testing and it was the post right after Guenther was congratulated on proving incorrectness: https://marc.info/?l=openbsd-tech=165259528425835=2 The post includes software to reproduce the results. I wrote a highly

Fwd: Picky, but much more efficient arc4random_uniform!

2022-05-20 Thread Luke Small
I appreciate your response, Damien. I did do the bare minimum of correctness testing and it was the post right after Guenther was congratulated on proving incorrectness: https://marc.info/?l=openbsd-tech=165259528425835=2 The post includes software to reproduce the results. I wrote a highly

Re: Picky, but much more efficient arc4random_uniform!

2022-05-18 Thread Steffen Nurpmeso
Joerg Sonnenberger wrote in : |Am Wed, May 18, 2022 at 12:49:21PM +0200 schrieb Steffen Nurpmeso: |> What surprised me was that the Apple code requires more calls, and |> that today divisions and multiplications still matter. I think it |> was the Cyrix 166+ (or was it Athlon 1600+) where

Re: Picky, but much more efficient arc4random_uniform!

2022-05-18 Thread Joerg Sonnenberger
Am Wed, May 18, 2022 at 12:49:21PM +0200 schrieb Steffen Nurpmeso: > What surprised me was that the Apple code requires more calls, and > that today divisions and multiplications still matter. I think it > was the Cyrix 166+ (or was it Athlon 1600+) where +,-,<<,>> was > one cycle, * was ten

Re: Picky, but much more efficient arc4random_uniform!

2022-05-18 Thread Steffen Nurpmeso
Steffen Nurpmeso wrote in <20220518104921.sr0ht%stef...@sdaoden.eu>: ... |The web site that has been linked from the man from the country |that has an even much worse Earth Country Overshoot Day than |Germany and is almost en par with Australia or even USA (April |3rd, pooh; never again a

Re: Picky, but much more efficient arc4random_uniform!

2022-05-18 Thread Steffen Nurpmeso
Philip Guenther wrote in : |On Tue, May 17, 2022 at 1:10 PM Steffen Nurpmeso \ |wrote: |> Joerg Sonnenberger wrote in |> : |>|Am Fri, May 13, 2022 at 09:43:26AM -0500 schrieb Luke Small: |>|> I made a couple new versions of a new kind of arc4random_uniform-like |> ... |>|If your main

Re: Picky, but much more efficient arc4random_uniform!

2022-05-18 Thread Otto Moerbeek
On Wed, May 18, 2022 at 05:08:02PM +1000, Damien Miller wrote: > On Wed, 18 May 2022, Otto Moerbeek wrote: > > > instrumenting the code to count the number of arc4random calls I see thsi: > > > > openbsd; elapsed = 2.835819; calls = 12340949 > > bitmask; elapsed = 4.335576; calls = 17836216 > >

Re: Picky, but much more efficient arc4random_uniform!

2022-05-18 Thread Damien Miller
On Wed, 18 May 2022, Otto Moerbeek wrote: > instrumenting the code to count the number of arc4random calls I see thsi: > > openbsd; elapsed = 2.835819; calls = 12340949 > bitmask; elapsed = 4.335576; calls = 17836216 > bitmask+reuse; elapsed = 3.710277; calls = 15245337 > > (this is a different

Re: Picky, but much more efficient arc4random_uniform!

2022-05-17 Thread Otto Moerbeek
On Wed, May 18, 2022 at 02:50:28PM +1000, Damien Miller wrote: > On Tue, 17 May 2022, Raimo Niskanen wrote: > > > Why reinvent the wheel? > > > > Here is a pretty good walkthrough of established methods: > > > > https://www.pcg-random.org/posts/bounded-rands.html > > > > It sounds to me

Re: Picky, but much more efficient arc4random_uniform!

2022-05-17 Thread Theo de Raadt
> However, I don't see a speedup for either of the alternate approaches. Or maybe, just maybe, the underlying function (arc4random, which in the dominant case it is just a small chacha step) is so inexpensive relative to the proposed higher-level logic changes? So this is all tilting at

Re: Picky, but much more efficient arc4random_uniform!

2022-05-17 Thread Damien Miller
On Tue, 17 May 2022, Raimo Niskanen wrote: > Why reinvent the wheel? > > Here is a pretty good walkthrough of established methods: > > https://www.pcg-random.org/posts/bounded-rands.html > > It sounds to me as if your suggested methor essentially is > "Bitmask with Rejection -- Apple's

Re: Picky, but much more efficient arc4random_uniform!

2022-05-17 Thread Philip Guenther
On Tue, May 17, 2022 at 1:10 PM Steffen Nurpmeso wrote: > Joerg Sonnenberger wrote in > : > |Am Fri, May 13, 2022 at 09:43:26AM -0500 schrieb Luke Small: > |> I made a couple new versions of a new kind of arc4random_uniform-like > ... > |If your main use case is limiting the amount of

Re: Picky, but much more efficient arc4random_uniform!

2022-05-17 Thread Steffen Nurpmeso
Steffen Nurpmeso wrote in <20220517220924.xohqc%stef...@sdaoden.eu>: |Joerg Sonnenberger wrote in | : ||Am Fri, May 13, 2022 at 09:43:26AM -0500 schrieb Luke Small: ||> I made a couple new versions of a new kind of arc4random_uniform-like ... |0 bytes "do not occur", so a 32-bit RNG value

Re: Picky, but much more efficient arc4random_uniform!

2022-05-17 Thread Steffen Nurpmeso
Joerg Sonnenberger wrote in : |Am Fri, May 13, 2022 at 09:43:26AM -0500 schrieb Luke Small: |> I made a couple new versions of a new kind of arc4random_uniform-like ... |If your main use case is limiting the amount of cryptography when using |small bounds, there is a much simpler approach to

Re: Picky, but much more efficient arc4random_uniform!

2022-05-17 Thread Joerg Sonnenberger
Am Fri, May 13, 2022 at 09:43:26AM -0500 schrieb Luke Small: > I made a couple new versions of a new kind of arc4random_uniform-like > function and some example functions which use them. Instead of having a > sufficiently large random number greater than the modulus, I pick a random > number using

Re: Picky, but much more efficient arc4random_uniform!

2022-05-16 Thread Damien Miller
On Mon, 16 May 2022, Luke Small wrote: > Yeah, I see your point. > > I suppose it depends on how conservative you want to be and whether > you want to supply options to people like getchar_unlocked when it > isn’t essential. > > It could be made manually fork-safe if I could make a simple feature

Re: Picky, but much more efficient arc4random_uniform!

2022-05-16 Thread Luke Small
Yeah, I see your point. I suppose it depends on how conservative you want to be and whether you want to supply options to people like getchar_unlocked when it isn’t essential. It could be made manually fork-safe if I could make a simple feature where “arc4random_uniform_unlocked(0);” with a 0

Re: Picky, but much more efficient arc4random_uniform!

2022-05-16 Thread Stuart Henderson
On 2022/05/16 15:13, Luke Small wrote: > If you’re not running a threaded program, my function wouldn’t be “less > safe.” > > I’d imagine that 99% of programs aren’t multithreaded. code is reused in different places. non threaded programs are sometimes turned into threaded programs and when that

Re: Picky, but much more efficient arc4random_uniform!

2022-05-16 Thread Luke Small
No...am I incorrect, especially on OpenBSD? Of course since you made such a remark, you seem like the kind of fellow that would put the nail in the coffin for spite. ...now I sound like an asshole. On Mon, May 16, 2022 at 4:00 PM Theo de Raadt wrote: > hey luke you know you sound like an

Re: Picky, but much more efficient arc4random_uniform!

2022-05-16 Thread Theo de Raadt
hey luke you know you sound like an asshole right? Luke Small wrote: > If you’re not running a threaded program, my function wouldn’t be “less > safe.” > > I’d imagine that 99% of programs aren’t multithreaded. > > On Mon, May 16, 2022 at 1:01 PM wrote: > > > > There is the specifically

Re: Picky, but much more efficient arc4random_uniform!

2022-05-16 Thread Luke Small
If you’re not running a threaded program, my function wouldn’t be “less safe.” I’d imagine that 99% of programs aren’t multithreaded. On Mon, May 16, 2022 at 1:01 PM wrote: > > There is the specifically non-threadsafe call getchar_unlocked() on > OpenBSD > > which is presumably available for

Re: Picky, but much more efficient arc4random_uniform!

2022-05-16 Thread Steffen Nurpmeso
Philip Guenther wrote in : |On Sun, 15 May 2022, Steffen Nurpmeso wrote: |> Stuart Henderson wrote in |... |>|what's the perceived problem you're wanting to solve? and does that |>|problem actually exist in the first place? |> |> The problem is that if have a low upper bound then modulo

Re: Picky, but much more efficient arc4random_uniform!

2022-05-16 Thread Sven M . Hallberg
Luke Small on Sun, May 15 2022: > The current implementation is nothing more than a naive arc4random() % > upper_bound which trashes initial arc4random() calls it doesn’t like, Yes, that's exactly what it is. > The whole transformation by modulus of perfectly decent random data > seems so

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Luke Small
Yeah. It most likely won't go in. From past experience and advice, not necessarily just from a perceived lack of merit. However, many, if not all of the arguments are based upon non-facts and misconceptions from earlier submissions or just not understanding what the software is doing. The only

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Luke Small
I’m not trying to be rude, but you don’t realize what’s going on here: uuu is a bitmask: ‘uuu’ (or (1 << bits)-1 ) in “ret = rand_holder & uuu;“ , only puts the lower ‘bit’ quantity of bits of rand_holder into ret, then it right shifts rand_holder afterward to trash them every time in the loop

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Mike Larkin
On Sun, May 15, 2022 at 08:40:19PM -0500, Luke Small wrote: > https://marc.info/?l=openbsd-tech=165259528425835=2 > > This one (which is strongly based upon my first of two versions) which I > submitted after Guenther correctly trashed version 2, doesn’t reuse any > part of the sample. It picks up

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Luke Small
https://marc.info/?l=openbsd-tech=165259528425835=2 This one (which is strongly based upon my first of two versions) which I submitted after Guenther correctly trashed version 2, doesn’t reuse any part of the sample. It picks up a clean new bitfield upon failure. I think Guenther didn’t, perhaps

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Damien Miller
On Sat, 14 May 2022, Luke Small wrote: > Look at my code. I don’t even use a modulus operator. I perform hit and > miss with a random bitstream. > > How can I have a bias of something I don’t do? I return a bitstream which > meets the parameters of being a value less than the upper bound. Much

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Damien Miller
On Sun, 15 May 2022, Luke Small wrote: > Do I really have to use specific terminology to make a point? > > I'm not educated enough on chacha20 enough to know whether, like I > pointed out, whether choosing 5 bits from the middle of (or even from > the tail end of one and the beginning of another)

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Luke Small
Do I really have to use specific terminology to make a point? I'm not educated enough on chacha20 enough to know whether, like I pointed out, whether choosing 5 bits from the middle of (or even from the tail end of one and the beginning of another) 32 bit pseudorandom cipher is "correct."

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Philip Guenther
On Sun, 15 May 2022, Luke Small wrote: > The current implementation is nothing more than a naive arc4random() % > upper_bound which trashes initial arc4random() calls it doesn’t like, then > transforms over a desired modulus. The whole transformation by modulus of > perfectly decent random data

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Luke Small
The current implementation is nothing more than a naive arc4random() % upper_bound which trashes initial arc4random() calls it doesn’t like, then transforms over a desired modulus. The whole transformation by modulus of perfectly decent random data seems so awkward. It’s not like it is used as

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Otto Moerbeek
On Sun, May 15, 2022 at 08:57:09AM -0300, Crystal Kolipe wrote: > On Sun, May 15, 2022 at 11:44:29AM +0200, Otto Moerbeek wrote: > > On Sun, May 15, 2022 at 04:27:30AM -0500, Luke Small wrote: > > > How did someone prove the current implementation was cryptographically > > > sound? Did they run

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Crystal Kolipe
On Sun, May 15, 2022 at 11:44:29AM +0200, Otto Moerbeek wrote: > On Sun, May 15, 2022 at 04:27:30AM -0500, Luke Small wrote: > > How did someone prove the current implementation was cryptographically > > sound? Did they run massive simulations which ran the gamut of the uint32_t > > range which

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Marc Espie
Random generators have been analyzed for years. Pick up "Concrete mathematics" by Don E. Knuth, read it, then come back with actual science.

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Otto Moerbeek
On Sun, May 15, 2022 at 04:27:30AM -0500, Luke Small wrote: > I have a feeling that making it threadsafe under any quantity of threads > and still perform is likely not feasible, but if I could; or if making a > nonthreadsafe variant was possible: > > Creating a mathematical proof that somehow

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Luke Small
I have a feeling that making it threadsafe under any quantity of threads and still perform is likely not feasible, but if I could; or if making a nonthreadsafe variant was possible: Creating a mathematical proof that somehow is “simple and probably correct” enough, would be an impossible task

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Otto Moerbeek
On Sun, May 15, 2022 at 01:12:28AM -0500, Luke Small wrote: > This is version 1, which I was pretty sure was secure. > > I revamped it with a few features and implanted the binary search for 'bit' > > in most cases, which aren't intentionally worst-case, it's pretty darned > fast! > > This is

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Luke Small
This is version 1, which I was pretty sure was secure. I revamped it with a few features and implanted the binary search for 'bit' in most cases, which aren't intentionally worst-case, it's pretty darned fast! This is a sample run of my program with your piece of code included: 1 99319

Re: Picky, but much more efficient arc4random_uniform!

2022-05-15 Thread Otto Moerbeek
On Sat, May 14, 2022 at 05:03:17PM -0700, Philip Guenther wrote: > On Sun, 15 May 2022, Steffen Nurpmeso wrote: > > Stuart Henderson wrote in > ... > > |what's the perceived problem you're wanting to solve? and does that > > |problem actually exist in the first place? > > > > The problem is

Re: Picky, but much more efficient arc4random_uniform!

2022-05-14 Thread Philip Guenther
On Sun, 15 May 2022, Steffen Nurpmeso wrote: > Stuart Henderson wrote in ... > |what's the perceived problem you're wanting to solve? and does that > |problem actually exist in the first place? > > The problem is that if have a low upper bound then modulo will "remove a > lot of

Re: Picky, but much more efficient arc4random_uniform!

2022-05-14 Thread Steffen Nurpmeso
Stuart Henderson wrote in : |On 2022/05/14 06:56, Luke Small wrote: |> If I use arc4random_uniform() repeatedly to create a random distribution \ |> of |> say numbers less than 0x1000 or even something weird like 0x1300 will the |> random distribution be better with arc4random_uniform() or

Re: Picky, but much more efficient arc4random_uniform!

2022-05-14 Thread Matthew Martin
int main() { int results[3] = { 0, 0, 0 }; for (int i = 0; i < 10; i++) { results[arc4random_uniform_fast_simple(3)]++; } for (int i = 0; i < 3; i++) printf("%d: %d\n", i, results[i]); return 0; } % ./a.out 0: 24809 1:

Re: Picky, but much more efficient arc4random_uniform!

2022-05-14 Thread Stuart Henderson
On 2022/05/14 06:56, Luke Small wrote: > If I use arc4random_uniform() repeatedly to create a random distribution of > say numbers less than 0x1000 or even something weird like 0x1300 will the > random distribution be better with arc4random_uniform() or with mine? there's no point to have a

Re: Picky, but much more efficient arc4random_uniform!

2022-05-14 Thread Luke Small
Look at my code. I don’t even use a modulus operator. I perform hit and miss with a random bitstream. How can I have a bias of something I don’t do? I return a bitstream which meets the parameters of being a value less than the upper bound. Much like arc4random_buf(). If I use

Re: Picky, but much more efficient arc4random_uniform!

2022-05-14 Thread Otto Moerbeek
On Sat, May 14, 2022 at 05:48:10AM -0500, Luke Small wrote: > arc4random_uniform_fast2 that I made, streams in data from arc4random() and > uses the datastream directly and uses it as a bit by bit right "sliding > window" in the last loop. arc4random_uniform() uses a modulus which I is > simple

Re: Picky, but much more efficient arc4random_uniform!

2022-05-14 Thread Luke Small
arc4random_uniform_fast2 that I made, streams in data from arc4random() and uses the datastream directly and uses it as a bit by bit right "sliding window" in the last loop. arc4random_uniform() uses a modulus which I is simple to implement, but I wonder how cryptographically sound or even how

Re: Picky, but much more efficient arc4random_uniform!

2022-05-14 Thread Otto Moerbeek
In general, wasting some arc4random bits is not a problem at all. I see a lot of code with no demonstration, explanation or proof why it would be correct (i.e. produces uniform results). You only talk about speed. The current code might not be optimal, but at least it is very clear it's correct.

Picky, but much more efficient arc4random_uniform!

2022-05-13 Thread Luke Small
I made a couple new versions of a new kind of arc4random_uniform-like function and some example functions which use them. Instead of having a sufficiently large random number greater than the modulus, I pick a random number using arc4random() from a bitfield where the length of the bitfield is