Re: [racket-users] Speeding up the conversion of flvectors to string
Bogdan Popa writes: > Ah! You're right. When I make the change you suggest, the program > takes 5s to run. Filed under "Accidentally Not Quadratic" . The program actually takes 2.5s to run. I missed the fact that I had made two separate calls to `string-append!` in that version of the program because the `string-append!` where the source and the destination were the same ended up being so cheap. The fixed version: https://gist.github.com/Bogdanp/f36201cd852ac2e363fdaebba1d02709#file-quadratic-no-alloc-rkt I also wrote a version of that same code with a custom `string-append` function: https://gist.github.com/Bogdanp/f36201cd852ac2e363fdaebba1d02709#file-quadratic-with-alloc-rkt That version takes about 6.7s to run. When I did that, I finally realized that what I had misinterpreted as allocation overhead was actually mostly the overhead of zero-filling the newly-allocated string, which I'd forgotten about. Here's that same code, but using bytes instead of strings: https://gist.github.com/Bogdanp/f36201cd852ac2e363fdaebba1d02709#file-quadratic-with-alloc-bytes-rkt It runs in 2.2s. Finally, a version of that code that uses a custom byte string implementation that doesn't perform an initialization step: https://gist.github.com/Bogdanp/f36201cd852ac2e363fdaebba1d02709#file-quadratic-with-alloc-no-fill-rkt That runs in 870ms, which is a little under half the time of the previous version, which makes sense to me since the number of iterations is halved and the GC is being side-stepped. All that to say I was completely wrong yesterday about allocation being a big factor here! Assuming I haven't made any other mistakes and the same 2x improvement could apply to a version of `string-append` that doesn't try to fill the newly-allocated string before copying, I wonder if that would be a worthwhile improvement to make to it. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2k0mecvqc.fsf%40defn.io.
Re: [racket-users] Speeding up the conversion of flvectors to string
Matthew Flatt writes: > At Sun, 27 Jun 2021 21:36:09 +0300, Bogdan Popa wrote: > Your program does run fast on my machine, but I think it's because this > line doesn't have the intended bad effect: > >> (string-copy! dst 0 dst 0 len) ;; intentionally performing pointless >> ;; work to be n^2 > > A `string-copy!` like this will eventually use the C library's > `memmove`, which apparently (and sensibly) doesn't do anything if the > source and destination are the same. Try copying to a `dst2` to get > quadratic behavior. Ah! You're right. When I make the change you suggest, the program takes 5s to run. Filed under "Accidentally Not Quadratic" . -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2mtrbcbpg.fsf%40defn.io.
Re: [racket-users] Speeding up the conversion of flvectors to string
While I think the complexity piece is important, I feel like it's worth pointing out just how much more expensive the allocation -- and its ramifications, like the resulting GC pressure and CPU cache misses -- is than one might think. Here's a quadratic version of the code that avoids allocations: #lang racket (require racket/flonum) (define (xy->string x y) (string-append (~r x #:precision 1) "," (~r y #:precision 1))) (define len 0) (define dst (make-string 500)) (define (string-append! b) (string-copy! dst 0 dst 0 len) ;; intentionally performing pointless work to be n^2 (string-copy! dst len b) (set! len (+ len (string-length b (define (xy-vectors->string x-vec y-vec) (for ((x (in-flvector x-vec)) (y (in-flvector y-vec))) (string-append! (xy->string x y)) (string-append! " "))) (time (let ([x (make-flvector 10)] [y (make-flvector 10)]) (xy-vectors->string x y))) On my machine (running Racket CS 8.1), this is faster than the `call-with-output-string` version I posted earlier by about 50ms. Jens Axel Søgaard writes: > Den søn. 27. jun. 2021 kl. 18.58 skrev Alessandro Motta >: > >> I also like Jens' code for its pure functional elegance. I'm surprised >> that building up a long list of short strings before joining them is so >> much faster. After all, isn't the final `string-join` essentially doing >> what the original code snipped did? (Clearly not! I will have a look at >> the implementation.) >> > > The following is the same answer as Robby's, but fills in some details. > I wrote this mainly because I couldn't find a web-page with the explanation. > > Let's say we have two strings a and b of lengths m and n respectively. > > How much work must (string-append a b) do? > > Conceptually the following steps need to be done: > 1. Allocate a new string c of length m+n. > 2. Copy the string a into c. > 3. Copy the string b into c. > > This means that the time (work) needed to append the strings a and b are > proportional to m+n. > > How much work is done here? > > (string-append "x" (string-append "y" (string-append "z" ""))) > > First (string-append "z" "w") takes time 1+0=1 with the result "z". > Then (string-append "y" (string-append "z" "")) or (string-append "y" "z") > takes time 1+1=2 with the result "yz". > Then (string-append "x" (string-append "y" (string-append "z" ""))) or > (string-append "x" "yz") takes time 1+2 = 3 with result "xyz". > In total we have used time 1+2+3 =6. > > We see that if we 3 times use string-append to add strings of length 1, > then we use time 1+2+3=6. > In general, if we n times use string-append to add strings of length 1, > then we use time 1+2+3+...+n. > The last sum is more or less n^2. See [1]. > > The same thing happens in your loop, where you repeatedly use string-append > to append a small string. > The length of the small string is no longer 1, but the same happens - and > the time used by the > loop is proportional to n^2. > > > Now suppose we have a list of the strings to append > (list "x" "y" "z") > and we need to append them. How much work is there now? > > Well, first we can calculate the length of the result 1+1+1 = 3. > Then we allocate a new string of length 3. Then each individual > string is copied and that takes time 1+1+1=3. > > Here, each string is only copied once. For comparison in the loop version > the string z is copied multiple times. > > > But wait - why does a loop that uses cons multiple times to build up > a list not have the same problem? > > Because in (cons x xs) the existing list xs isn't copied. > The steps are >1. a new pair with two cells (the car and the cdr) are allocated >2. the contents of the car is set to x >3. the contents of the cdr is set to xs. > > This always takes the same amount of time, no matter how long the list xs > is. > This means that the time used by cons is constant (that is, proportional to > 1). > > Note: This phenomenon that using string-append in a loop is not a Racket > only problem. > It is a common pitfall in many languages. > > > > [1] Remember the famous story of Gauss as a child that calculated > 1+2+...1000 ? > https://www.youtube.com/watch?v=Dd81F6-Ar_0 > > > /Jens Axel - https://racket-stories.com -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2pmw7cfza.fsf%40defn.io.
Re: [racket-users] Speeding up the conversion of flvectors to string
I forgot to mention this earlier, but it might be enlightening to run the two versions of the program with the `PLTSTDERR` environment variable set to "error debug@GC" to see GC debug messages. For example: env PLTSTDERR='error debug@GC' racket flvector-to-coordinates-string.rkt >/dev/null Bogdan Popa writes: > Hi Alessandro, > > Here is a version of your program that is about 30 times faster on my > machine (9s -> 300ms): > > #lang racket/base > > (require racket/flonum > racket/format > racket/port) > > (define (xy-vectors->string x-vec y-vec) > (call-with-output-string >(lambda (out) > (for ([i (in-naturals)] >[x (in-flvector x-vec)] >[y (in-flvector y-vec)]) >(unless (zero? i) > (write-char #\space out)) >(write-string (~r x #:precision 1) out) >(write-char #\, out) >(write-string (~r y #:precision 1) out) > > (time > (let ([x (make-flvector 10)] >[y (make-flvector 10)]) >(xy-vectors->string x y))) > > All the calls to `string-append` in your original program end up > allocating larger and larger strings and then immediately discarding > them on subsequent iterations, which is costly over many iterations. > > Hope that helps, > Bogdan > > Alessandro Motta writes: > >> Hi racket-users! >> >> I've recently become interested in Lisp/Scheme and have started to hack >> in Racket. The excellent documentation, the fast integrated search, and >> DrRacket have made that a real pleasure. >> >> Thank you for that! >> >> I've been working on a tool to convert notes from the reMarkable 2 >> tablet to SVG files. At the core is the conversion of (x, y) coordinate >> pairs from two `flvector`s to a string of the form "x1,y1 x2,y2 x3,y3". >> >> ``` >> (define (xy->string x y) >> (string-append >>(~r x #:precision 1) "," >>(~r y #:precision 1))) >> >> (define (xy-vectors->string x-vec y-vec) >> (for/fold ((coordinates "") >> (separator "") >> #:result coordinates) >> ((x (in-flvector x-vec)) >> (y (in-flvector y-vec))) >> (values (string-append >> coordinates >> separator >> (xy->string x y)) >> " "))) >> ``` >> >> This is currently the bottleneck for large conversion jobs. >> >> Profiling these functions with `profile-flame-graph` resulted in >> https://gist.githubusercontent.com/amotta/cfe4b19e24455af219521c9e94455c67/raw/dbbc87bd2f6dd4e27c33831749baa90fffdaed55/flvector-to-coordinates-string-flamegraph.svg >> >> The full profiling script is available at >> https://gist.github.com/amotta/e76197082bb1bf63538ede01872917f3 >> >> Roughly 90% of time is spent in `contract/private/arrow-val-first.rkt`. >> Based on my very limited understanding of Racket, it seems that ~38% of >> time is spent handling keyword arguments (presumably `#:precision 1`?). >> The `catnp` function (the conversion from flonum to string, I think) >> takes up only ~11% of time. >> >> Is this interpretation of the flame graph correct? If so, are there any >> obvious blunders on my part? Any ideas for how to speed up this code? >> >> >> Best wishes, >> Alessandro -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2sg13crgy.fsf%40defn.io.
Re: [racket-users] Speeding up the conversion of flvectors to string
Hi Alessandro, Here is a version of your program that is about 30 times faster on my machine (9s -> 300ms): #lang racket/base (require racket/flonum racket/format racket/port) (define (xy-vectors->string x-vec y-vec) (call-with-output-string (lambda (out) (for ([i (in-naturals)] [x (in-flvector x-vec)] [y (in-flvector y-vec)]) (unless (zero? i) (write-char #\space out)) (write-string (~r x #:precision 1) out) (write-char #\, out) (write-string (~r y #:precision 1) out) (time (let ([x (make-flvector 10)] [y (make-flvector 10)]) (xy-vectors->string x y))) All the calls to `string-append` in your original program end up allocating larger and larger strings and then immediately discarding them on subsequent iterations, which is costly over many iterations. Hope that helps, Bogdan Alessandro Motta writes: > Hi racket-users! > > I've recently become interested in Lisp/Scheme and have started to hack > in Racket. The excellent documentation, the fast integrated search, and > DrRacket have made that a real pleasure. > > Thank you for that! > > I've been working on a tool to convert notes from the reMarkable 2 > tablet to SVG files. At the core is the conversion of (x, y) coordinate > pairs from two `flvector`s to a string of the form "x1,y1 x2,y2 x3,y3". > > ``` > (define (xy->string x y) > (string-append >(~r x #:precision 1) "," >(~r y #:precision 1))) > > (define (xy-vectors->string x-vec y-vec) > (for/fold ((coordinates "") > (separator "") > #:result coordinates) > ((x (in-flvector x-vec)) > (y (in-flvector y-vec))) > (values (string-append > coordinates > separator > (xy->string x y)) > " "))) > ``` > > This is currently the bottleneck for large conversion jobs. > > Profiling these functions with `profile-flame-graph` resulted in > https://gist.githubusercontent.com/amotta/cfe4b19e24455af219521c9e94455c67/raw/dbbc87bd2f6dd4e27c33831749baa90fffdaed55/flvector-to-coordinates-string-flamegraph.svg > > The full profiling script is available at > https://gist.github.com/amotta/e76197082bb1bf63538ede01872917f3 > > Roughly 90% of time is spent in `contract/private/arrow-val-first.rkt`. > Based on my very limited understanding of Racket, it seems that ~38% of > time is spent handling keyword arguments (presumably `#:precision 1`?). > The `catnp` function (the conversion from flonum to string, I think) > takes up only ~11% of time. > > Is this interpretation of the flame graph correct? If so, are there any > obvious blunders on my part? Any ideas for how to speed up this code? > > > Best wishes, > Alessandro -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2v95zcs9v.fsf%40defn.io.
[racket-users] recent sqlite3 versions (was Re: preview of a cross-compilation tool)
kamist...@gmail.com writes: > (So I guess the only question that comes up is: is it possible to update > the sqlite version?) > [vaguely remember seeing a mailing list thread about this but, currently I > only find other ones] My libsqlite3[1] package provides recent versions (currently 3.35.5) of SQLite for Linux, macOS and Windows. If you add it to your app's `deps`, everything should just work. [1]: https://pkgd.racket-lang.org/pkgn/package/libsqlite3 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m28s4casnh.fsf%40192.168.0.142.
Re: [racket-users] Word Count program/benchmark performance
On top of these changes, replacing the hash function with Chez's
`equal-hash` saves another 50ms. That gets the runtime down to around
900ms on my machine, including Racket startup time. Ignoring startup
time, this version beats the `simple.c` implementation in the original
repo by about 50ms (800ms vs 850ms).
https://gist.github.com/Bogdanp/b9256e1a91de9083830cb616b3659ff8
Gustavo Massaccesi writes:
> With two additional tricks I saved like 100ms.
>
> * Saving the output port instead of reading the parameter implicitly each
> time.
>
> * Replacing (write (cdr p)) with (write-fx cdr p)) where
>
> (define (write-fx n [o (current-output-port)])
> ; TODO: Add negatives :)
> (if (fx> n 0)
> (let loop ([n n])
> (when (fx> n 10)
> (loop (fxquotient n 10)))
> (write-byte (fx+ 48 (fxremainder n 10)) o))
> (write-byte 48 o)))
>
> and at the end of a program something like
>
> (define o (current-output-port))
> (time (for ([p (in-vector items)]
> #:break (not (pair? p)))
> (write-bytes (car p) o)
> (write-byte 32 o)
> (write-fx (cdr p) o)
> (write-byte 10 o))) ; and a closing )
>
> Gustavo
>
> On Fri, Mar 19, 2021 at 1:22 PM Sam Tobin-Hochstadt
> wrote:
>
>> I went from numbers around 1000 ms to 950 ms to 900 ms. There was
>> variance around those numbers, but it was pretty consistent.
>>
>> For more precise answers, there are a few things you can try. One is
>> to measure instructions instead of time (ie, with perf). Another is to
>> run it a bunch of times and take an average. The `hyperfine` tool is
>> good for that. But probably the best advice is to make the program
>> take longer so differences are more apparent -- variation usually
>> increases sub-linearly.
>>
>> Sam
>>
>> On Fri, Mar 19, 2021 at 12:17 PM Laurent wrote:
>> >
>> > Sam: How do you accurately measure such small speed-ups? On my machines,
>> if I run the same program twice, I can sometimes see more than 10% time
>> difference.
>> >
>> > On Fri, Mar 19, 2021 at 4:10 PM Sam Tobin-Hochstadt <
>> sa...@cs.indiana.edu> wrote:
>> >>
>> >> Use `#:authentic`, and `unsafe-vector*-{ref,set!}` saved about 50 more
>> >> ms on my machine.
>> >>
>> >> Then getting rid of `set!` and just re-binding the relevant variables
>> >> produced another 50 ms speedup.
>> >>
>> >> https://gist.github.com/7fc52e7bdc327fb59c8858a42258c26a
>> >>
>> >> Sam
>> >>
>> >> On Fri, Mar 19, 2021 at 7:21 AM Sam Tobin-Hochstadt
>> >> wrote:
>> >> >
>> >> > One minor additional suggestion: if you use #:authentic for the
>> struct, it will generate slightly better code for the accessors.
>> >> >
>> >> > Sam
>> >> >
>> >> > On Fri, Mar 19, 2021, 6:18 AM Bogdan Popa wrote:
>> >> >>
>> >> >> I updated the gist with some cleanups and additional improvements
>> that
>> >> >> get the runtime down to a little over 1s (vs ~350ms for the
>> optimized C
>> >> >> and Rust code) on my maxed-out 2019 MBP and ~600ms on my M1 Mac Mini.
>> >> >>
>> >> >> Pawel Mosakowski writes:
>> >> >>
>> >> >> > Hi Bogdan,
>> >> >> >
>> >> >> > This is a brilliant solution and also completely over my head. It
>> finishes
>> >> >> > in ~3.75s on my PC and is faster than the Python version which
>> basically
>> >> >> > delegates all the work to C. I will need to spend some time on
>> >> >> > understanding it but I am looking forward to learning something
>> new.
>> >> >> >
>> >> >> > Many thanks,
>> >> >> > Pawel
>> >> >> >
>> >> >> > On Thursday, March 18, 2021 at 7:22:10 PM UTC bogdan wrote:
>> >> >> >
>> >> >> >> I managed to get it about as fast as Python by making it really
>> >> >> >> imperative and rolling my own hash:
>> >> >> >>
>> >> >> >> https://gist.github.com/Bogdanp/fb39d202037cdaadd55dae3d45737571
>> >> >> >>
>> >> >> >> Sam Tobin-Hochstadt writes:
>> >> >> >>
>> >> >> >> > Here are several variants of the code:
>> >> >> >&g
Re: [racket-users] Word Count program/benchmark performance
Nice! It's worth pointing out, though, that by limiting yourself to alpha chars, you're processing about 8% less data and the results don't pass the tests. :P $ wc kjvbible_x10.txt 998170 8211330 43325060 $ sed 's/[a-zA-Z ]//g' < kjvbible_x10.txt | wc 998170 739310 3600800 I think your version would still be faster, but I'm curious what the numbers would look like if only whitespace chars were considered word separators. Dominik Pantůček writes: > Another attack of [1]. But yeah, why not do some [2]. > > Trees to the rescue [3]. > > $ racket --version > Welcome to Racket v8.0 [cs]. > > $ racket countwords-bogdan2.rkt cpu time: 135 real time: 135 gc time: 8 > > $ racket countwords-dzoe2.rkt cpu time: 69 real time: 69 gc time: 3 > > I just changed (countwords) to (time (countwords)) in Bogdan's code to > measure the running time. > > The difference is that I am positively defining which letters form words > (a-z, A-Z) and that all others are treated as word separators. The > buffer size is the same - and honestly, the speedup between 1024 and > 1024^2 bytes buffer is barely measurable. > > The only option for further speedup I can immediately think of is to > allocate a huge vector of wtnodes and change chld field to be a starting > index into this big vector (should reduce allocations). > > Btw, making it unsafe does not speed it up at all (probably CS > recognizes the vectors and all those refs are inlined anyway). > > > Cheers, > Dominik > > [1] https://xkcd.com/386/ > [2] http://phdcomics.com/comics/archive.php?comicid=1735 > [3] https://gist.github.com/dzoep/0e081d0544afac539a4829179c601e0e > > On 19. 03. 21 11:18, Bogdan Popa wrote: >> I updated the gist with some cleanups and additional improvements that >> get the runtime down to a little over 1s (vs ~350ms for the optimized C >> and Rust code) on my maxed-out 2019 MBP and ~600ms on my M1 Mac Mini. >> >> Pawel Mosakowski writes: >> >>> Hi Bogdan, >>> >>> This is a brilliant solution and also completely over my head. It finishes >>> in ~3.75s on my PC and is faster than the Python version which basically >>> delegates all the work to C. I will need to spend some time on >>> understanding it but I am looking forward to learning something new. >>> >>> Many thanks, >>> Pawel >>> >>> On Thursday, March 18, 2021 at 7:22:10 PM UTC bogdan wrote: >>> >>>> I managed to get it about as fast as Python by making it really >>>> imperative and rolling my own hash: >>>> >>>> https://gist.github.com/Bogdanp/fb39d202037cdaadd55dae3d45737571 >>>> >>>> Sam Tobin-Hochstadt writes: >>>> >>>>> Here are several variants of the code: >>>>> https://gist.github.com/d6fbe3757c462d5b4d1d9393b72f9ab9 >>>>> >>>>> The enabled version is about the fastest I can get without using >>>>> `unsafe` (which the rules said not to do). It's possible to optimize a >>>>> tiny bit more by avoiding sorting, but only a few milliseconds -- it >>>>> would be more significant if there were more different words. >>>>> >>>>> Switching to bytes works correctly for the given task, but wouldn't >>>>> always work in the case of general UTF8 input. But those versions >>>>> appeared not be faster for me. Also, writing my own string-downcase >>>>> didn't help. And using a big buffer and doing my own newline splitting >>>>> didn't help either. >>>>> >>>>> The version using just a regexp matching on a port (suggested by >>>>> Robby) turned out not to be faster either, so my suspicion is that the >>>>> original slowness is just using regexps for splitting words. >>>>> >>>>> Sam >>>>> >>>>> On Thu, Mar 18, 2021 at 11:28 AM Sam Tobin-Hochstadt >>>>> wrote: >>>>>> >>>>>> Here's a somewhat-optimized version of the code: >>>>>> >>>>>> #lang racket/base >>>>>> (require racket/string racket/vector racket/port) >>>>>> >>>>>> (define h (make-hash)) >>>>>> >>>>>> (time >>>>>> (for* ([l (in-lines)] >>>>>> [w (in-list (string-split l))] >>>>>> [w* (in-value (string-downcase w))]) >>>>>> (hash-update! h w* add1 0))) >>>>>> >>>>>> (define v >>>>>> (time >
Re: [racket-users] Word Count program/benchmark performance
I updated the gist with some cleanups and additional improvements that get the runtime down to a little over 1s (vs ~350ms for the optimized C and Rust code) on my maxed-out 2019 MBP and ~600ms on my M1 Mac Mini. Pawel Mosakowski writes: > Hi Bogdan, > > This is a brilliant solution and also completely over my head. It finishes > in ~3.75s on my PC and is faster than the Python version which basically > delegates all the work to C. I will need to spend some time on > understanding it but I am looking forward to learning something new. > > Many thanks, > Pawel > > On Thursday, March 18, 2021 at 7:22:10 PM UTC bogdan wrote: > >> I managed to get it about as fast as Python by making it really >> imperative and rolling my own hash: >> >> https://gist.github.com/Bogdanp/fb39d202037cdaadd55dae3d45737571 >> >> Sam Tobin-Hochstadt writes: >> >> > Here are several variants of the code: >> > https://gist.github.com/d6fbe3757c462d5b4d1d9393b72f9ab9 >> > >> > The enabled version is about the fastest I can get without using >> > `unsafe` (which the rules said not to do). It's possible to optimize a >> > tiny bit more by avoiding sorting, but only a few milliseconds -- it >> > would be more significant if there were more different words. >> > >> > Switching to bytes works correctly for the given task, but wouldn't >> > always work in the case of general UTF8 input. But those versions >> > appeared not be faster for me. Also, writing my own string-downcase >> > didn't help. And using a big buffer and doing my own newline splitting >> > didn't help either. >> > >> > The version using just a regexp matching on a port (suggested by >> > Robby) turned out not to be faster either, so my suspicion is that the >> > original slowness is just using regexps for splitting words. >> > >> > Sam >> > >> > On Thu, Mar 18, 2021 at 11:28 AM Sam Tobin-Hochstadt >> > wrote: >> >> >> >> Here's a somewhat-optimized version of the code: >> >> >> >> #lang racket/base >> >> (require racket/string racket/vector racket/port) >> >> >> >> (define h (make-hash)) >> >> >> >> (time >> >> (for* ([l (in-lines)] >> >> [w (in-list (string-split l))] >> >> [w* (in-value (string-downcase w))]) >> >> (hash-update! h w* add1 0))) >> >> >> >> (define v >> >> (time >> >> (for/vector #:length (hash-count h) >> >> ([(k v) (in-hash h)]) >> >> (cons k v >> >> (time (vector-sort! v > #:key cdr)) >> >> (define p (current-output-port) #;(open-output-nowhere)) >> >> (time >> >> (for ([pair (in-vector v)]) >> >> (write-string (car pair) p) >> >> (write-string (number->string (cdr pair)) p) >> >> (newline p))) >> >> >> >> It's much more imperative, but also pretty nice and compact. The >> >> `printf` optimization is significant for that portion of the program, >> >> but that isn't much of the running time. The overall running time for >> >> 10 copies of the KJV is about 9 seconds on my laptop. >> >> >> >> I think the remaining difference between Racket and other languages is >> >> likely the `string-split` and `string-downcase` functions, plus the >> >> relatively-inefficient string representation that Racket uses. >> >> >> >> Sam >> >> >> >> >> >> On Thu, Mar 18, 2021 at 10:28 AM Pawel Mosakowski >> wrote: >> >> > >> >> > Hi David, >> >> > >> >> > Yes, the 21 seconds includes the interpreter startup time. I have >> done a simple test to see how long it takes: >> >> > >> >> > $ time racket -e '(displayln "Hello, world")' >> >> > Hello, world >> >> > >> >> > real 0m0.479s >> >> > user 0m0.449s >> >> > sys 0m0.030s >> >> > >> >> > I have also put my code inside a main function and profiled it: >> >> > >> >> > Profiling results >> >> > - >> >> > Total cpu time observed: 20910ms (out of 20970ms) >> >> > Number of samples taken: 382 (once every 55ms) >> >> > (Hiding functions with self<1.0% and local<2.0%: 1 of 12 hidden) >> >> > >> >> > == >> >> > Caller >> >> > Idx Total Self Name+src Local% >> >> > ms(pct) ms(pct) Callee >> >> > == >> >> > [1] 20910(100.0%) 0(0.0%) [running body] >> ...word-occurences-profile.rkt":##f >> >> > profile-thunk [2] 100.0% >> >> > -- >> >> > [running body] [1] 100.0% >> >> > [2] 20910(100.0%) 0(0.0%) profile-thunk >> ...ket/pkgs/profile-lib/main.rkt:9:0 >> >> > run [3] 100.0% >> >> > -- >> >> > profile-thunk [2] 100.0% >> >> > [3] 20910(100.0%) 0(0.0%) run >> ...share/racket/pkgs/profile-lib/main.rkt:39:2 >> >> > main [4] 100.0% >> >> > -- >> >> > run [3] 100.0% >> >> > [4] 20910(100.0%) 50(0.2%) main >> ...cket/count-word-occurences-profile.rkt:5:0 >> >> > read-from-stdin-it [5] 98.5% >> >> > ??? [6] 0.2% >> >> > -- >> >> > main [4] 100.0% >> >> > [5] 20606(98.5%)
Re: [racket-users] Word Count program/benchmark performance
I managed to get it about as fast as Python by making it really imperative and rolling my own hash: https://gist.github.com/Bogdanp/fb39d202037cdaadd55dae3d45737571 Sam Tobin-Hochstadt writes: > Here are several variants of the code: > https://gist.github.com/d6fbe3757c462d5b4d1d9393b72f9ab9 > > The enabled version is about the fastest I can get without using > `unsafe` (which the rules said not to do). It's possible to optimize a > tiny bit more by avoiding sorting, but only a few milliseconds -- it > would be more significant if there were more different words. > > Switching to bytes works correctly for the given task, but wouldn't > always work in the case of general UTF8 input. But those versions > appeared not be faster for me. Also, writing my own string-downcase > didn't help. And using a big buffer and doing my own newline splitting > didn't help either. > > The version using just a regexp matching on a port (suggested by > Robby) turned out not to be faster either, so my suspicion is that the > original slowness is just using regexps for splitting words. > > Sam > > On Thu, Mar 18, 2021 at 11:28 AM Sam Tobin-Hochstadt > wrote: >> >> Here's a somewhat-optimized version of the code: >> >> #lang racket/base >> (require racket/string racket/vector racket/port) >> >> (define h (make-hash)) >> >> (time >> (for* ([l (in-lines)] >> [w (in-list (string-split l))] >> [w* (in-value (string-downcase w))]) >>(hash-update! h w* add1 0))) >> >> (define v >> (time >>(for/vector #:length (hash-count h) >>([(k v) (in-hash h)]) >> (cons k v >> (time (vector-sort! v > #:key cdr)) >> (define p (current-output-port) #;(open-output-nowhere)) >> (time >> (for ([pair (in-vector v)]) >>(write-string (car pair) p) >>(write-string (number->string (cdr pair)) p) >>(newline p))) >> >> It's much more imperative, but also pretty nice and compact. The >> `printf` optimization is significant for that portion of the program, >> but that isn't much of the running time. The overall running time for >> 10 copies of the KJV is about 9 seconds on my laptop. >> >> I think the remaining difference between Racket and other languages is >> likely the `string-split` and `string-downcase` functions, plus the >> relatively-inefficient string representation that Racket uses. >> >> Sam >> >> >> On Thu, Mar 18, 2021 at 10:28 AM Pawel Mosakowski >> wrote: >> > >> > Hi David, >> > >> > Yes, the 21 seconds includes the interpreter startup time. I have done a >> > simple test to see how long it takes: >> > >> > $ time racket -e '(displayln "Hello, world")' >> > Hello, world >> > >> > real0m0.479s >> > user0m0.449s >> > sys0m0.030s >> > >> > I have also put my code inside a main function and profiled it: >> > >> > Profiling results >> > - >> > Total cpu time observed: 20910ms (out of 20970ms) >> > Number of samples taken: 382 (once every 55ms) >> > (Hiding functions with self<1.0% and local<2.0%: 1 of 12 hidden) >> > >> > == >> > Caller >> > IdxTotal Self Name+srcLocal% >> > ms(pct) ms(pct) Callee >> > == >> > [1] 20910(100.0%) 0(0.0%) [running body] >> > ...word-occurences-profile.rkt":##f >> > profile-thunk [2] 100.0% >> > -- >> > [running body] [1]100.0% >> > [2] 20910(100.0%) 0(0.0%) profile-thunk >> > ...ket/pkgs/profile-lib/main.rkt:9:0 >> > run [3] 100.0% >> > -- >> > profile-thunk [2] 100.0% >> > [3] 20910(100.0%) 0(0.0%) run >> > ...share/racket/pkgs/profile-lib/main.rkt:39:2 >> > main [4] 100.0% >> > -- >> > run [3] 100.0% >> > [4] 20910(100.0%)50(0.2%) main >> > ...cket/count-word-occurences-profile.rkt:5:0 >> > read-from-stdin-it [5] 98.5% >> > ??? [6] 0.2% >> > -- >> > main [4] 100.0% >> > [5] 20606(98.5%) 11796(56.4%) read-from-stdin-it >> > ...-occurences-profile.rkt:19:6 >> > internal-split [7] 42.8% >> > -- >> > main [4] 100.0% >> > [6]51(0.2%) 0(0.0%) ??? >> > ...cket/collects/racket/private/sort.rkt:369:3 >> >
Re: [racket-users] Re: [racket] Web Framework Benchmarks
Round 20 was recently published and Racket's score improved, as expected:
https://www.techempower.com/benchmarks/#section=data-r20=ph=composite
We're now beating many of the popular frameworks in the composite scores
and I think there are still plenty of improvements that could be made.
hashim muqtadir writes:
> A new version of these benchmarks was just published ("Round 19"):
> https://www.techempower.com/benchmarks/#section=data-r19
>
> Racket scores rather poorly (0 errors is nice, though). Anybody has any
> guesses as to why this is? Racket performs well enough as far as I'm
> concerned, and while I don't use the web server much, sounds unlikely that
> it should be so far below django, for example. Maybe they run it in a
> handicapped configuration or something.
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/racket-users/m2zh0cqm6d.fsf%40192.168.0.142.
Re: [racket-users] Utah snapshots will switch to CS by default
Makes sense, thanks! For anybody using the `setup-racket' GitHub action to install snapshots, I cut a new release[1] that uses the new URL structure. All you have to do to update is change the `uses' line from `Bogdanp/setup-racket@v0.0X' to `Bogdanp/setup-racket@v0.10'. You only need to do this if you use `current' for the Racket version. [1]: https://github.com/Bogdanp/setup-racket/releases/tag/v0.10 Matthew Flatt writes: > CS will be built on Precise, so the "current" alias with "precise" will > continue to work, and BC will continue to be built on Precise. > > But CS will also be built on Xenial --- mostly because that build is > set up, but maybe it's a step toward migrating the build. > > At Wed, 12 Aug 2020 10:17:15 +0300, Bogdan Popa wrote: >> Thanks for the heads-up! Will CS continue to be built on Xenial and BC >> on Precise? >> >> Matthew Flatt writes: >> >> > As you may know, the Racket Git repo recently switched to Racket CS as >> > the default build. That is, if you check out the repo and `make`, then >> > you get an in-place Racket CS build instead of Racket BC. >> > >> > The next step in moving toward Racket CS is to try switching the >> > snapshot builds. Out current plan is to switch the Utah build on August >> > 13. The Northwestern snapshot build will be unchanged, for now. This >> > change will have no effect on the regular, release downloads. >> > >> > The layout of the page at >> > >> > https://www.cs.utah.edu/plt/snapshots/ >> > >> > will change so that CS and BC builds are grouped together, more like >> > the Northwestern snapshot layout, but an installer that doesn't have >> > "BC" in its link will download a Racket CS installer. >> > >> > If you have an automated process that pulls from the snapshot site, >> > then it probably downloads something like >> > >> > >> https://www.cs.utah.edu/plt/snapshots/current/installers/racket-current-x86_64- >> linux-precise.sh >> > >> > As of August 13, that will download a Racket CS installer instead of a >> > BC installer, because there's no "-bc" suffix in the name. >> > >> > Racket BC builds will still be available; you'll just have to >> > specifically select a download option with "BC" in the link name or >> > "-bc" in the URL. For example, >> > >> > >> https://www.cs.utah.edu/plt/snapshots/current/installers/racket-current-x86_64- >> linux-precise-bc.sh >> > >> > will download a Racket BC installer. For now, 32-bit Windows and Mac OS >> > installers will be available only for BC, because we view those as >> > legacy platforms. >> > >> > For consistently, you'll be able to access the CS installers using a >> > "-cs" suffix. The installer name without a "-cs" or "-bc" suffix is an >> > aliases for one with "-cs". Of course, the "current" names are all >> > aliases for installers that have a specific version number. >> > >> > While we're adjusting installer names, the canonical name for Minimal >> > Racket installers will change to "racket-minimal-..." instead of >> > "min-racket-...". The new name matches the names used for releases. For >> > the foreseeable future, "min-racket-..." aliases will still work, but >> > you should migrate to "racket-minimal-..." if you have an automated >> > process that uses "min-racket-...". >> > >> > >> > Matthew >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Racket Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to racket-users+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/racket-users/m2wo242v0k.fsf%40192.168.0.142. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2tux53cxn.fsf%40192.168.0.142.
Re: [racket-users] Utah snapshots will switch to CS by default
Thanks for the heads-up! Will CS continue to be built on Xenial and BC on Precise? Matthew Flatt writes: > As you may know, the Racket Git repo recently switched to Racket CS as > the default build. That is, if you check out the repo and `make`, then > you get an in-place Racket CS build instead of Racket BC. > > The next step in moving toward Racket CS is to try switching the > snapshot builds. Out current plan is to switch the Utah build on August > 13. The Northwestern snapshot build will be unchanged, for now. This > change will have no effect on the regular, release downloads. > > The layout of the page at > > https://www.cs.utah.edu/plt/snapshots/ > > will change so that CS and BC builds are grouped together, more like > the Northwestern snapshot layout, but an installer that doesn't have > "BC" in its link will download a Racket CS installer. > > If you have an automated process that pulls from the snapshot site, > then it probably downloads something like > > https://www.cs.utah.edu/plt/snapshots/current/installers/racket-current-x86_64-linux-precise.sh > > As of August 13, that will download a Racket CS installer instead of a > BC installer, because there's no "-bc" suffix in the name. > > Racket BC builds will still be available; you'll just have to > specifically select a download option with "BC" in the link name or > "-bc" in the URL. For example, > > https://www.cs.utah.edu/plt/snapshots/current/installers/racket-current-x86_64-linux-precise-bc.sh > > will download a Racket BC installer. For now, 32-bit Windows and Mac OS > installers will be available only for BC, because we view those as > legacy platforms. > > For consistently, you'll be able to access the CS installers using a > "-cs" suffix. The installer name without a "-cs" or "-bc" suffix is an > aliases for one with "-cs". Of course, the "current" names are all > aliases for installers that have a specific version number. > > While we're adjusting installer names, the canonical name for Minimal > Racket installers will change to "racket-minimal-..." instead of > "min-racket-...". The new name matches the names used for releases. For > the foreseeable future, "min-racket-..." aliases will still work, but > you should migrate to "racket-minimal-..." if you have an automated > process that uses "min-racket-...". > > > Matthew -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2wo242v0k.fsf%40192.168.0.142.
Re: [racket-users] Re: [racket] Web Framework Benchmarks
George Neuner writes: > Multiple Racket applications *should* all be able to listen on the > same port without having been spawned from the same ancestor process. > If that isn't working now, something has gotten hosed. I don't know whether this used to work in the past or not, but currently only `SO_REUSEADDR' is set on TCP sockets: https://github.com/racket/racket/blob/60bf8f970e97caae391bfe919b78c370b2d01bdd/racket/src/rktio/rktio_network.c#L1427 I think we'd need to also set `SO_REUSEPORT', which is not available on all platforms, to support multiple processes listening on the same port without reusing file descriptors. Running a second instance of this program: #lang racket/base (require racket/tcp) (tcp-listen 512 #t) (sync/enable-break never-evt) Currently fails with: tcp-listen: listen failed port number: system error: Address already in use; errno=48 context...: ... Greg Hendershott pointed out to me on Slack a while back that the `ffi/usafe/port' module could be used to bind sockets with custom options using the FFI and then convert them to ports, but I have yet to try that. > I'm not sure what you mean by "sharing" a listener, but using a > single listener with a pool of processing places actually is possible > (though tricky). I mean passing a `tcp-listener?' around between places so that each place can call `tcp-accept' on it. Something like this: #lang racket/base (require racket/place racket/tcp) (define ch (place ch (define listener (place-channel-get ch)) (tcp-accept listener) (place-channel-put 'ok))) (module+ main (define listener (tcp-listen 512 #t)) (place-channel-put ch listener) (place-channel-get ch)) Currently, this fails with: place-channel-put: contract violation expected: place-message-allowed? given: # context...: ... My understanding is that there's nothing preventing this from working apart from the fact that no one's yet added support for this in the rktio layer. As you mentioned, though, even if this was supported we might run into other limitations when running many places at once. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m24krkih6h.fsf%40192.168.0.142.
Re: [racket-users] Re: [racket] Web Framework Benchmarks
Alex Harsanyi writes: > Question 1: Based on this benchmark, is there any reason to chose anything > else but "drogon"? Even if one chooses the second best on that list, which > is "actix", they already loose about 6% performance and things degrade > quickly afterwards. The framework at position 10 is already half the speed > of the top one. My take on these benchmarks is all that matters is that the framework doesn't get in your way once you add business logic. The vast majority of "real" web applications out there don't (and most likely can't) do 50k rps. You can see that in the "multiple queries" and "data updates" tests where the results are packed closer together because the logic is closer to what a lot of database-backed web applications do and the database ends up being a bottleneck. A description of the requirements of each test can be found here: https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview If you know and are willing to deal with writing and maintaining C++ then drogon looks like it might be a great choice. > Question 2: Based on Bogdans message in this thread, it seems that most of > the performance improvement for the Racket benchmark comes from the nginx > configuration (which has nothing to do with Racket) and the next > improvement has to do with how the user program is written (by supplying a > "Content-Length" header). So, is this benchmark really testing the Racket > web server performance, or is it testing a very specific deployment? The largest improvement comes from making the Racket application take advantage of all the hardware threads on the machine. Because Racket doesn't currently have a way to share TCP listeners across places and because fork isn't natively supported (I mentioned that it works via the FFI earlier in the thread, but I believe it needs some support from the runtime (handling of `EAGAIN') to work efficiently and not cause a lot of churn) I did the next best thing: I made the benchmark run one Racket process for each thread[1] and added nginx as a load balancer in front. The nginx process listens on port 8080, forks one subprocess per core (which lets the subprocesses reuse the same port) and then proxies any incoming requests on that port to one of the Racket processes so every single request is ultimately served by the Racket app. What this means in terms of this benchmark is that, compared to others, we're actually paying a toll for using nginx here because its own workers are consuming resources on the machine, but, to my knowledge, we don't have a better alternative at the moment. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/988f052c8170da661c49dd51d1f33d500a871031/frameworks/Racket/racket/scripts/run#L15-L19 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m27dwgiuwo.fsf%40192.168.0.142.
Re: [racket-users] Re: [racket] Web Framework Benchmarks
Yury Bulka writes: > Wow, from 695 requests per second to 49,516 is a huge improvement! > > Since we were comparing to django previously, it's now much closer with > django (which does 78,132 rps.) I expect the Racket benchmark will do even better on TechEmpower's hw than it did on mine because their machines are more powerful and they run the benchmarking code, the database and the servers on different machines so the gap should end up being even smaller. Also worth keeping in mind that the server Django uses for these benchmarks, meinheld, is completely written in C: * https://github.com/mopemope/meinheld/blob/311acbc4e7bd38fa3f3d0e158b35cde9ef73f8e5/meinheld/gmeinheld.py#L11 * https://github.com/mopemope/meinheld/tree/311acbc4e7bd38fa3f3d0e158b35cde9ef73f8e5/meinheld/server So I think it's really impressive that we're able to get this close with just pure Racket, with the overhead of running one nginx process per core in front of the app and with the overhead of connecting those nginx processes to the app via TCP. Speaking of which, another thing we could do to improve performance in this benchmark is define a custom `tcp^' unit based on `unix-socket-lib' and have nginx connect to the backends through unix sockets rather than TCP. > So do I undestand correctly that the factors that contribute to the > improvement are: > > 1. running multiple worker processes behind nginx > 2. adding content-length to all responses > 3. using CS variant of Racket > 4. using Racket 7.7 > 5. tuning nginx config (enabling http 1.1 especially) I'm afraid I haven't kept notes on how much improvement each of these improvements yielded. From what I remember: I don't think #4 was much of a factor, #1 and #5 were the biggest factors followed by #2 and #3, which I changed at the same time so I couldn't say which made more of a difference. > #2 is something that seems to require manual work in the client code, > but maybe that can be made easier on web-server-lib side somehow. The work one has to do is pretty minimal[1] and I personally like that the default is to stream data so I think the only thing to improve here is awareness: the docs for `response' should be updated to mention that unless a `Content-Length' is provided, the responses will use chunked transfer encoding. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5727/files#diff-b21f7e3ecfa09726dac9ce079f612719R47-R70 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2a71dismf.fsf%40192.168.0.142.
Re: [racket-users] Re: [racket] Web Framework Benchmarks
Small update on this: I've updated the benchmarks to run multiple Racket processes with an Nginx load balancer in front. After some tuning[1], here is what the results look like on my 12 core AMD Ryzen 9 3900 server: https://www.techempower.com/benchmarks/#section=test=669bfab7-9242-4c26-8921-a4fe9ccd8530=ph=composite=2 50k/s is a respectable number for the plaintext benchmark IMO and we could get it to go higher if we could ditch Nginx or spend more time improving the server's internals, as Sam suggested. The `racket-perf' benchmark is for a branch[2] that I have where I've made some small improvements to the server's internals. [0]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5727 [1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5737 [2]: https://github.com/racket/web-server/pull/94 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2o8pupyge.fsf%40192.168.0.142.
Re: [racket-users] Re: [racket] Web Framework Benchmarks
The reason for the poor performance relative to the other langs/frameworks is
that there is currently no easy way to take advantage of multiple cores using
the web framework so what’s being benchmarked is single core perf. This is
mainly a problem for benchmarks such as this, but not really an issue in the
real world where you’d just run multiple processes w/ a load balancer in front.
> On Jun 1, 2020, at 1:43 PM, hashim muqtadir wrote:
>
>
> A new version of these benchmarks was just published ("Round 19"):
> https://www.techempower.com/benchmarks/#section=data-r19
>
> Racket scores rather poorly (0 errors is nice, though). Anybody has any
> guesses as to why this is? Racket performs well enough as far as I'm
> concerned, and while I don't use the web server much, sounds unlikely that it
> should be so far below django, for example. Maybe they run it in a
> handicapped configuration or something.
> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/2a1062a4-e4f1-484c-a8a2-f63c161959e8%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/racket-users/FD7C8C62-8AE2-411E-A32D-3AC180DFB3D6%40defn.io.
Re: [racket-users] Re: [racket] Web Framework Benchmarks
Bogdan Popa writes: > Only dispatchers get direct access to the output port for the socket. > If you use `dispatch/servlet', then it takes care of taking your > `response' value and calling `output-response' on it. Unless the server > knows the connection should be closed and unless the request was a HEAD > request, then it outputs the response by chunking it: > > https://github.com/racket/web-server/blob/547b3fd736684651e94ebd78902633374be6bcae/web-server-lib/web-server/http/response.rkt#L133-L181 I was wrong about this. The web-server does the right thing if the `Content-Length' header is present in the response value: https://github.com/racket/web-server/blob/547b3fd736684651e94ebd78902633374be6bcae/web-server-lib/web-server/http/response.rkt#L120 When I change the benchmark to take advantage of that, throughput doubles: https://www.techempower.com/benchmarks/#section=test=ab930604-6b19-4ab4-a2a5-93e674a81804=ph=plaintext=2 I'll make a PR against the main repo to make sure all of the responses have content lengths. Out of curiosity, I made the app fork through FFI calls[1]: https://www.techempower.com/benchmarks/#section=test=0197c25f-d544-4357-b3be-a40d4a8760ef=ph=plaintext=2 This worked although multiple processes wake up when a connection can be accepted and the IO layer doesn't seem to handle `EAGAIN': racket: Connection error: tcp-accept: accept from listener failed racket: system error: Resource temporarily unavailable; errno=11 racket: context...: racket:.../more-scheme.rkt:261:28 racket:/root/.racket/7.7/pkgs/compatibility-lib/mzlib/thread.rkt:75:14: loop [1]: https://gist.github.com/Bogdanp/b7b72ff7845f7f2c51e64bde553128d0#file-fork-app-rkt-L232-L256 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m28sh5rc87.fsf%40192.168.0.142.
Re: [racket-users] Re: [racket] Web Framework Benchmarks
George Neuner writes: > What I did say is that Python's threads are core limited - and *that* > is true. As a technical matter, Python *may* in fact start threads > on different cores, but the continual need to take the GIL quickly > forces every running thread in the process onto the same core. I was pointing out that the GIL is irrelevant in this case. All the Python implementations in this benchmark use either green threads by monkeypatching the standard threading and IO modules (gevent, meinheld) or coroutines (asyncio) for concurrency and they fork subprocesses for parallelism. > That one actually is expected: Racket's JSON (de)serializer is > relatively slow. It's the same as the plaintext test, except JSON is written to the client instead of plain text: https://github.com/TechEmpower/FrameworkBenchmarks/blob/bfd7c1442f33e620524b6aa0751c11576b412e72/frameworks/Racket/racket/servlet.rkt#L140-L143 > What wasn't expected was Sam's results from the "plain text" test > which also showed Racket much slower than Python. That does hint at a > lot of overhead in the Racket framework. Here's single-core Python vs Racket in the plaintext benchmark on my machine: https://www.techempower.com/benchmarks/#section=test=464938de-3ec0-4931-bc68-41566c22=ph=plaintext=2 It is surprising that Racket does worse on this benchmark that it does on the JSON one, despite the fact that `response/json' uses `response/output' under the hood. I see these errors from Racket when I run the plaintext benchmark, but they don't occur in any of the others: racket: tcp-addresses: could not get peer address racket: system error: Transport endpoint is not connected; errno=107 racket: context...: racket: /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33 racket:.../more-scheme.rkt:261:28 racket: /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33 racket: /root/.racket/7.6/pkgs/web-server-lib/web-server/private/dispatch-server-with-connect-unit.rkt:144:4: connection-loop I'll try to figure out what's causing these. > To my knowledge, continuations will not be a factor unless either 1) > the application is written in the #web-server language (which converts > everything to CPS), or 2) the code invokes one of the send/suspend/* > functions. Whether you use the web interaction functions or not, servlets have to do some bookkeeping (create new "instances", insert continuation prompts) to support continuations: * https://github.com/racket/web-server/blob/547b3fd736684651e94ebd78902633374be6bcae/web-server-lib/web-server/servlet-dispatch.rkt#L78-L92 * https://github.com/racket/web-server/blob/547b3fd736684651e94ebd78902633374be6bcae/web-server-lib/web-server/servlet/setup.rkt#L52-L72 * https://github.com/racket/web-server/blob/547b3fd736684651e94ebd78902633374be6bcae/web-server-lib/web-server/dispatchers/dispatch-servlets.rkt#L63-L100 Bypassing all of this is what I considered cheating, because most people probably won't. At the same time, though, I haven't measured what the overhead of all this stuff is. It could be minimal. > My understanding is that the port passed to response/output is the > actual socket ... so you can front-end it and write directly. But > that might be "cheating" under your definition. That's what I did in the benchmark: https://github.com/TechEmpower/FrameworkBenchmarks/blob/bfd7c1442f33e620524b6aa0751c11576b412e72/frameworks/Racket/racket/servlet.rkt#L135-L138 Only dispatchers get direct access to the output port for the socket. If you use `dispatch/servlet', then it takes care of taking your `response' value and calling `output-response' on it. Unless the server knows the connection should be closed and unless the request was a HEAD request, then it outputs the response by chunking it: https://github.com/racket/web-server/blob/547b3fd736684651e94ebd78902633374be6bcae/web-server-lib/web-server/http/response.rkt#L133-L181 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2h7vtrjv9.fsf%40192.168.0.142.
Re: [racket-users] Re: [racket] Web Framework Benchmarks
George Neuner writes: > But Python's DB pool is threaded, and Python's threads are core > limited by the GIL in all the major implementations (excepting > Jython). Python's Postgres pooling does not[1] use POSIX threads under the hood to manage the connections if that's what you mean, nor is the concurrency of the Python applications based on system threads. All of the Python examples use either asyncio + fork(2) or green threads + fork(2). This includes the django example[3]. > There are a few things Python can do faster than Racket, but the VAST > difference in performance shown in the techempower tests isn't > explained by them. Here's a benchmark that doesn't touch the DB at all, showing an even bigger difference in throughput between the two: https://www.techempower.com/benchmarks/#section=data-r19=ph=json Here's the same benchmark running on my local machine where I intentionally limited the Django app to a single CPU and I made it use the `gevent' library for its workers so it is more comparable to the Racket implementation: https://www.techempower.com/benchmarks/#section=test=14ecbf16-cdb3-4501-8b7d-a2b8a549f73c=ph=json=2 And here's what happens when I let it use as much parallelism as it can: https://www.techempower.com/benchmarks/#section=test=d3ad4d79-c7a7-4ca0-b297-ffda549947c8=ph=json=2 I do agree that improving the parallelism part wouldn't be enough to catch up (clearly, there's a 2x difference even on a single core), but it is a large factor here. I wrote the latest implementation of the Racket code for that benchmark and I considered doing things like bypassing the "standard" `dispatch/servlet' implementation to avoid the overhead of all the continuation machinery in the web server, but that felt like cheating. Another area where the web server does more work than it should is in generating responses: the web server uses chunked transfer encoding for all responses; whereas all the Python web servers simply write the response directly to the socket when the length of the content is known ahead of time. Another thing of note about the django implementation is that it uses ujson, written in C with the express intent of being as fast as possible, to generate the JSON data. [1]: They call the default implementation a `ThreadedConnectionPool'[2], but that's just because it uses the mutexes that the `threading' module provides. [2]: https://github.com/psycopg/psycopg2/blob/779a1370ceeac130de07edc0510f2c55846be1bd/lib/pool.py#L155 [3]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/c49524762379a2cdf82627b0032c654f3a9eafb6/frameworks/Python/django/gunicorn_conf.py#L8-L21 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2mu5mr0zy.fsf%40192.168.0.142.
Re: [racket-users] Re: [racket] Web Framework Benchmarks
I replied earlier today off of my Phone, but, for whatever reason
(caught in the moderation queue?), it's not showing up in this thread.
Here's what it said:
The reason for poor performance relative to the other
langs/frameworks is that there is currently no easy way to take
advantage of multiple cores using the web framework so that's being
benchmarked is single-core performance.
This is mainly a problem for benchmarks such as this, but not really
an issue in the real world where you'd just run multiple processes
with a load balancer in front.
hashim muqtadir writes:
> A new version of these benchmarks was just published ("Round 19"):
> https://www.techempower.com/benchmarks/#section=data-r19
>
> Racket scores rather poorly (0 errors is nice, though). Anybody has any
> guesses as to why this is? Racket performs well enough as far as I'm
> concerned, and while I don't use the web server much, sounds unlikely that
> it should be so far below django, for example. Maybe they run it in a
> handicapped configuration or something.
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/racket-users/m2tuzusn0u.fsf%40192.168.0.142.
Re: [racket-users] Exception throwing in web-server
Norman Gray writes: > But what happens in this case (the my-app/error case in my example) is > that the (server) program keeps going but the client stalls. The > unexpected error in the response-output procedure is caught, and (as > far as I can see) handled by killing the producer thread _without_ > closing the connection. To be clear, I think that the handler should > do both. The handler you pointed to in your other email is intended to catch network errors and abrupt hangups from the client, but will not catch exceptions raised by the response's output function. For that, there needs to be an exception handler in the chunker thread. I've opened a PR to add such a handler here: https://github.com/racket/web-server/pull/93 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2mu5vu1v5.fsf%40192.168.0.142.
Re: [racket-users] Re: Should I stop sending packages to the catalog?
Sage Gerard writes: > As long as I don't have to keep reorganizing my code to accommodate > the tooling, then it's a night and day improvement. Would you mind > terribly if I worked with you? I was mulling over this myself in the > dev list, but I am happy to aid any existing effort. I'd appreciate it! Let's sync up on Slack. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2ees2mkvs.fsf%40192.168.0.142.
Re: [racket-users] Re: Should I stop sending packages to the catalog?
Sam Tobin-Hochstadt writes: > This is very cool! One question -- for "main-distribution" packages, > are you snapshotting the most-recent release catalog? Or > pkgs.racket-lang.org? The latter is simpler, but the former is > probably needed to make the examples in the README work. Otherwise, > if, say, "typed-racket-lib" on May 1 depended on a newer-than-7.6 > version of "base", using the snapshot first (or only the snapshot) > would get you a version of "typed-racket-lib" that didn't work on 7.6. At the moment, I'm snapshotting pkgs.racket-lang.org but not updating packages from the main distribution. That is, I start with the main distribution of regular Racket 7.6 and add packages on top, but I don't change any existing ones. I think I can change the first example in the README to just raco pkg config --set catalogs \ https://racksnaps.defn.io/snapshots/2020/05/01/catalog/ \ https://pkgs.racket-lang.org \ https://planet-compats.racket-lang.org without changing the behavior. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2h7wzmth6.fsf%40192.168.0.142.
Re: [racket-users] Re: Should I stop sending packages to the catalog?
Alex Harsanyi writes: > As an application writer, I am on the other side of this problem, by > depending on other packages. Having limited time to work on my project I > want to upgrade package dependencies at my own pace. I'm in a similar position since I operate a few Racket applications. I'd been mulling over the idea of a snapshotting service for the main package catalog to help automate some of this over the past couple of months, but this thread finally pushed me to get started on building it. More details here: https://github.com/bogdanp/racksnaps It's not quite ready for general use yet (the first snapshot is being built as we speak), but I should be able to iron out the kinks over the next couple of weeks. I'm curious what folks think about this approach. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2k11vmw0d.fsf%40192.168.0.142.
Re: [racket-users] Question about generating urls with dispatch-rules from imported modules
I solved this in koyo by defining an alternate version[1] of `dispatch-rules' whose generated `reverse-uri' functions take route names rather than functions as arguments: https://koyoweb.org/dispatch/index.html#%28form._%28%28lib._koyo%2Fdispatch..rkt%29._dispatch-rules%2Broles%29%29 The generated function is installed into a parameter on app init and another function[2] knows how to look it up and apply it when called: https://koyoweb.org/url/index.html#%28def._%28%28lib._koyo%2Furl..rkt%29._current-reverse-uri-fn%29%29 I hope that helps! [1]: https://github.com/Bogdanp/koyo/blob/331701e9e8d7f553955ea5a950df424ee13ce9d4/koyo-lib/koyo/dispatch.rkt [2]: https://github.com/Bogdanp/koyo/blob/331701e9e8d7f553955ea5a950df424ee13ce9d4/koyo-lib/koyo/url.rkt#L60 Yury Bulka writes: > Oops, small correction: > >> The x-expressions need to include hyperlinks to urls >> defined via dispatch-rules in main.rkt, i.e., call the url-generating >> function. > > Should be `routing.rkt` instead of `main.rkt`. > > -- > Yury Bulka > https://mamot.fr/@setthemfree > #NotOnFacebook > > > > Yury Bulka writes: > >> Dear Racket community, >> >> First of all I want to say hello since this is my first post here. >> >> I have a question about using the url dispatch library: >> https://docs.racket-lang.org/web-server/dispatch.html >> >> More specifically about the url-generation function in the context of >> inter-module dependencies. >> >> Let's say in `routing.rkt` I define some request handlers and then map >> them to urls using dispatch-rules. >> >> The request handlers call functions from another module that are >> responsible for generating the x-expressions (let's call this module >> `templates.rkt`). The x-expressions need to include hyperlinks to urls >> defined via dispatch-rules in main.rkt, i.e., call the url-generating >> function. >> >> The url-generating function, in turn, needs to be provided the request >> handler as the first argument. >> >> That means that the xexpr-generating function needs to have access to >> the url-generating function and all the request handlers it might >> need to include hyperlinks to, or it may need to be provided with all >> the generated urls in advance. >> >> Something like this wouldn't work if render-post needed to generate a url: >> >> ;; A request handler associated with a url >> (define (single-post-view request post-id) >> (response/xexpr (render-post (post-id))) ;; render-post is in another >> file/module >> >> I wouldn't want to make templates.rkt require routing.rkt (I'm not sure >> this is technically possible to have modules depend on each other like >> that), nor would I want to provide all the request handlers as arguments >> to `render-post`. I also thought about putting the url-generating >> function into a parameter, but `render-post` also needs access to the >> request handlers. >> >> What approach should I take? >> >> -- >> Yury Bulka >> https://mamot.fr/@setthemfree >> #NotOnFacebook -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m21robnvcf.fsf%40192.168.0.142.
Re: [racket-users] rename-in issue
I'd recommend using `prefix-in' with the dispatcher modules, as that avoids these types of issues altogether: (require (prefix-in files: web-server/dispatchers/dispatch-files) (prefix-in sequencer: web-server/dispatchers/dispatch-sequencer)) (sequencer:make (files:make ...) ...) Sorawee Porncharoenwase writes: > There are two colliding names. make and interface-version. As shown in the > error message, you fixed one but not the other. > > (require net/url > web-server/dispatchers/filesystem-map > web-server/dispatchers/dispatch-files > (rename-in web-server/dispatchers/dispatch-sequencer > [make make*] > [interface-version interface-version*])) > > works for me. > > On Thu, Apr 9, 2020 at 2:04 AM Lucas Liendo wrote: > >> Hi everyone! I've a very dumb question (as still not very experienced >> Racket programmer), I'm doing the following: >> >> (require net/url >> web-server/dispatchers/filesystem-map >> (rename-in web-server/dispatchers/dispatch-files >> (make dispatch-files)) >> (rename-in web-server/dispatchers/dispatch-sequencer >> (make dispatch-sequencer))) >> >> As you can see `make` is defined both in >> `web-server/dispatchers/dispatch-files` and >> `web-server/dispatchers/dispatch-sequencer` and I think that's the >> reason for this error: >> >> /tmp/rename-example.rkt:7:20: module: identifier already required >> at: interface-version >> in: web-server/dispatchers/dispatch-sequencer >> location...: >>/tmp/rename-example.rkt:7:20 >> >> Traceback continues but I think that's enough. So is there a >> workaround to this? I'd really like to do a rename instead of using >> `prefix-in`. Shouldn't this error not happen as it defeats `rename-in` >> purpose at all? >> >> -- >> Cheers, >> Lucas. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Racket Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to racket-users+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/racket-users/CABCAxXuZmGguei3gsfeVOcWUHZP52JR0ciULo%3DeFKGraNqADAA%40mail.gmail.com >> . >> -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m27dyorg7s.fsf%40192.168.0.142.
Re: [racket-users] Implications of stateless servlets & how/whether to avoid them
Jon Zeppieri writes: > When you stream the response, it doesn't use a chunked transfer encoding? -Jon The web server chunks all responses on HTTP/1.1 connections[1]. I can confirm that the web server works great[2] for streaming uses cases like long polling! [1]: https://github.com/racket/web-server/blob/564120c0f5e4bb959d4592a37037146d8411948e/web-server-lib/web-server/http/response.rkt#L56 [2]: https://github.com/Bogdanp/nemea/blob/5c8a58121ed9f264277d7eb56187fbeaad506138/nemea/http/reporting.rkt#L18-L35 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2mu9ddt4m.fsf%40192.168.0.142.
Re: [racket-users] How to stream file uploads with the Racket web server?
The version of the web-server that will be included with Racket 7.6 changes the way file uploads are handled so that they get offloaded to disk after a certain threshold (similar to how that nginx module you linked to works). You can check out the pre-release docs for details: * https://pre-release.racket-lang.org/doc/web-server/http.html?q=binding%3Afile#%28def._%28%28lib._web-server%2Fhttp%2Frequest-structs..rkt%29._make-binding~3afile%2Fport%29%29 * https://pre-release.racket-lang.org/doc/web-server-internal/dispatch-server-unit.html#%28part._safety-limits%29 To get these changes ahead of the release, you should be able to install an updated version of `web-server-lib' from the package server or from git. Hope that helps! - Bogdan Brian Adkins writes: > I'm posting a file to my web app using the following form: > >method="post" *enctype="multipart/form-data"* > class="file-upload-form"> > ... > > > I use a simple function to create a hashtable of attributes: > > (define (form-values req) > (for/hash ([ b (in-list (request-bindings/raw req)) ]) > (cond [ (binding:form? b) (values >(bytes->string/utf-8 (binding-id b) #\space) >(bytes->string/utf-8 (binding:form-value b) > #\space)) ] > [ (binding:file? b) (values >(bytes->string/utf-8 (binding-id b) #\space) >(binding:file-content b)) ]))) > > It appears that the entire file contents are in the binding by the time the > request is available to me. This is fine for "small enough" files, but for > larger files, it would be great to be able to stream the file contents. The > solution may be to use something like nginx's upload module: > > https://www.nginx.com/resources/wiki/modules/upload/ > > But before I go down that route, I thought I'd ask if the Racket web server > provides a more direct way to accomplish this. > > Thanks, > Brian -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2blq2wk4t.fsf%40192.168.0.142.
Re: [racket-users] Web server + authentication
Jens Axel Søgaard writes: > But I haven't implemented "reset password" functionality yet. I need > to figure out how best to handle sending mails from a server. My > concern is being flagged for spamming - so I don't dare use my normal > mail. I like using Postmark[0] for this. Their free plan lets you send up to 100 e-mails a month, their paid plans come at a reasonable price and they have helpful docs and validators to help you set up SPF, DMARC and DKIM. I use this client that I wrote with Racket: http://docs.racket-lang.org/postmark@postmark-client/index.html [0]: https://postmarkapp.com -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m236c7nco4.fsf%40192.168.0.142.
Re: [racket-users] Re: raco distribute and runtime paths, conditioned on whether a package was installed at compile time
James Platt writes: > Would this now be the preferred method to include a copy of the SQLite > library. required by the DB module, to your distribution? The > software I am working on requires a newer version of SQLite than is > included in at least some of the operating systems we want to support > and it's too much to expect end users to install it themselves. I released two packages[1][2] that distribute more recent versions of SQLite3 to Linux and macOS a couple of months ago. I build the shared libraries in CI (using GitHub Actions) then add the .so and .dylib files to each package, using #lang info's `copy-foreign-libs'[3] to tell Racket that it should add the libs to a folder where the ffi library will know to look them up. The source code is here[4]. I'd be happy to add more architectures and OSs if you need them. Here's an example[5] where I use the linux package to get the tests for `deta' passing on the package build server. I hope that helps! [1]: https://pkgd.racket-lang.org/pkgn/package/libsqlite3-x86_64-linux [2]: https://pkgd.racket-lang.org/pkgn/package/libsqlite3-x86_64-macosx [3]: https://docs.racket-lang.org/raco/setup-info.html?q=copy-foreign-libs#%28idx._%28gentag._22._%28lib._scribblings%2Fraco%2Fraco..scrbl%29%29%29 [4]: https://github.com/Bogdanp/racket-libsqlite3 [5]: https://github.com/Bogdanp/deta/blob/master/deta-test/info.rkt#L5 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2mucwi4tq.fsf%40192.168.0.139.
Re: [racket-users] Re: Package server broken, at least five hundred intro cs students affected!
> (The issue was that some of the packages had started to take advantage of a > feature of the dependencies list in info.rkt that allowed regular > expressions in the platform restriction clause. When it came time for the > indexer to convert the dependencies list to JSON, it failed because it > didn't know how to handle regular expressions.) I feel like that was me. Sorry! -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2bluu3q80.fsf%40192.168.0.139.
Re: [racket-users] [ANN] RacketScript working for 7.x
This is fantastic news! Thank you! Stephen Chang writes: > The RacketScript compiler [1] has been updated to work with Racket 7.x > programs, ie all tests pass (though we are still working to support > more primitives). > > Try it out here: http://rapture.twistedplane.com:8080/ > > The next milestone is to improve RacketScript enough to compile the > Racket expander [2], with the longer term goal being to self-host, so > programs may be independently compiled and run in the browser. > > If anyone is interested in helping out, let me know. > > > [1] https://github.com/vishesh/racketscript > [2] https://github.com/racket/racket/tree/master/racket/src/expander -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2a7ajugmd.fsf%40192.168.0.139.
Re: [racket-users] Re: i18n / translations
This is great! Thank you. Roman Klochkov writes: > Now we have gettext: https://docs.racket-lang.org/gettext/index.html > > суббота, 9 февраля 2019 г., 18:56:59 UTC+5 пользователь cwebber написал: >> >> How are people currently translating their programs? >> I expected to see a gettext library or something, but haven't. >> >> I see that DrRacket is translated, but I did a few simple greps through >> the DrRacket repo and couldn't figure out how. >> >> Am I missing something? Or is this tooling that needs to be written? >> - cwebb >> -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2eezwvnik.fsf%40192.168.0.139.
Re: [racket-users] What is a (the?) way to track requirements for a web or app with Racket?
Marc Kaufmann writes:
> So the easiest to do is to create such an info.rkt file and call `raco pkg
> install app-name/` -- and that should work even if I don't do the `raco
> exe` and `raco distribute`?
Yes, that's right. `raco exe' and `raco distribute' are not required.
I realize now that I should've mentioned a couple caveats to treating
apps as packages like this. I often need to tell Racket to recompile my
package after I make certain changes because the interpreter doesn't try
to recompile stale dependent modules of changed modules on the fly (if
`a.rkt' depends on `b.rkt' and you change `b.rkt', without recompiling
`a.rkt', the interpreter won't do it for you). This means you can run
into errors along the lines of "instantiate-linket: mismatch". When
that happens, you can either remove all the compiled bytecode from your
project
$ find app-name-here/ -type d -name compiled -exec rm -rf \{\} \;
or tell Racket to recompile all the modules in your package's collection
$ raco setup --avoid-main --tidy --check-pkg-deps --unused-pkg-deps
app-name-here
I usually do the latter, because large projects can take a very long
time to compile/load from scratch. That said, that also comes with its
own downside in the form of worse error reporting. `errortrace' can no
longer peer into the source code since it's been compiled which leads to
some pretty terse error reports.
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/racket-users/m2lfuo8slm.fsf%40192.168.0.139.
Re: [racket-users] What is a (the?) way to track requirements for a web or app with Racket?
Alex Harsanyi writes: >> Although it's not exactly the same thing, I use `info.rkt'[1][2] >> (similar to `setup.py') for this purpose and each one of >> my web apps is its own package. >> > > Does this mean that the application itself is available as a package and > you can require files from inside the application from another program? Yes. In fact, I leverage this property to keep my apps' tests in a separate package. My e-commerce site, for example, is composed of two packages: `matchacha' and `matchacha-tests'. This means I don't have to ship my tests and their dependencies to the server and it also has the added benefit that compilation times are improved. The latter is important in development since my development server watches the FS and it recompiles any files that change before it restarts the server. > More importantly, can you have two copies of the application installed on > your system, for example for development purposes? On my local machine, I only have the application installed once and it is linked to my local git checkout. I think that if you try to install the same package under multiple names, `raco pkg' will complain. I can't imagine wanting to have the same application installed multiple times for development, though, so I don't think I fully grasp your question. On my production server, I run the binaries as two separate systemd services: one for production and another one[1] that I use to test releases before they go out. Since I only ship distributions to the production server, I don't need to install any packages. In fact, I don't even have Racket installed on the production server. [1]: https://sandbox.matchacha.ro/ -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2muf48tna.fsf%40192.168.0.139.
Re: [racket-users] What is a (the?) way to track requirements for a web or app with Racket?
Marc Kaufmann writes: > this is surely answered somewhere, but I have not made much progress. If > you know python, what I want to do is essentially > > $ pip install requirements.txt > Although it's not exactly the same thing, I use `info.rkt'[1][2] (similar to `setup.py') for this purpose and each one of my web apps is its own package. Here's an example of what that ends up looking like: https://github.com/Bogdanp/koyo/blob/master/koyo-lib/blueprints/standard/app-name-here/info.rkt and https://github.com/Bogdanp/koyo/tree/master/koyo-lib/blueprints/standard#first-time-setup For deployment, I leverage `raco exe' and `raco distribute' to create self-contained distributions that I can ship to my server. [1]: https://docs.racket-lang.org/pkg/metadata.html [2]: https://docs.racket-lang.org/raco/setup-info.html -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2o8zk8vd7.fsf%40192.168.0.139.
Re: [racket-users] Persistent job queue
> I haven't been able to find a persistent job queue for Racket, so I thought > I'd ask if one exists before writing a simple one. I did see Jay's job > queue package, but it doesn't appear to persist to disk, and that is a > requirement. FWIW, I am working on a new Redis client package[1] that should be complete sometime next week and I have a little task queue example here[2] based on Redis streams. I plan on writing a task queueing library in the future based on this client (I have some experience with the problem domain[3]), but it may be a while before I get to it. > I typically use Postgres for my apps, and I would prefer to also use > Postgres for the persistence of the job queue to avoid introducing > something new (e.g. Redis) to my server setup. Amazon RDS provides great > multi-site availability, so I want to continue to leverage that. Recent versions of Postgres make this pretty easy to do so you might not need a library. Here's a little snippet from one of my older Racket applications that uses a Postgres table as a persistent task queue, in case it helps: https://gist.github.com/Bogdanp/c796b15a0fa8d577a5f46ff25230baa6 [1]: https://github.com/Bogdanp/racket-redis [2]: https://github.com/Bogdanp/racket-redis/tree/master/examples/task-queue [3]: https://dramatiq.io -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2d0glkbpr.fsf%40192.168.0.139.
Re: [racket-users] Trying out a Racket-powered Webpack alternative
Sage Gerard writes: > - unlike-assets: A build tool for interdependent assets. Similar to Webpack, > if you are familiar with the JS ecosystem. Cool stuff! I was very close to building an asset pipeline for koyo[1] a couple months ago, but I settled on just using broccoli.js instead. I'm glad to see this, though, and I'll definitely play around with it and potentially integrate it with koyo's standard blueprint at some point. I would love to eventually be able to get rid of any NPM deps in my applications. This reminds me that I have a little package that binds libsass stashed somewhere. I'll try to polish it up and release it next week! [1]: https://koyo.defn.io/ -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m21rx9acmr.fsf%40192.168.0.139.
Re: [racket-users] define/contract for private functions
> When I want to enable these private contracts (e.g. when running tests or > debugging a problem) I simply change the source code of `def/c` so that it > is wired up to `define/contract` and the contract is checked. Here is an > example: http://pasterack.org/pastes/4224 You can run arbitrary code in macros so you could use environment variables to achieve this. For example: (define-syntax-rule (def/c head contract body ...) (if (getenv "PRIVATE_CONTRACTS") (define/contract head contract body ...) (define head body ...))) And then run your code with env PRIVATE_CONTRACTS=x racket some-module.rkt The drawbacks to this approach is it makes compilation significantly slower and, once code is compiled (via raco setup, raco make or raco pkg install), the flag stops having any effect -- the code is compiled according to whatever the environment looks like at compile time and then maintains that behavior until it is recompiled. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2r26gk9oa.fsf%40192.168.0.139.
Re: [racket-users] Re: i18n / srfi-29
> Is it OK to wrap declare-bundle! + store-bundle inside (begin-for-syntax > ...) to make side effect during raco setup? If not, what is preferred way > to do it? Is there any reason not to do it at runtime? This[1] is what I do in forms-lib. Other modules depend on that module so the bundles are declared the first time it is loaded. It's not optimal for versioning, but it gets the job done. [1]: https://github.com/Bogdanp/racket-forms/blob/master/forms-lib/private/l10n.rkt#L27-L33 -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2v9vtjkvw.fsf%40192.168.0.139.
Re: [racket-users] dogfood a simple structured discussion forum in racket
In case someone sets out to do this, here are some things I have open sourced (from my closed source projects) in the web space that may help: * koyo[1]: a general toolking built on top of web-server-lib that adds many of the things you need in a real web application (CSRF, CORS, database utils, flash messages, localization, profiling, sessions, and others -- the documentation is still a little rough, but if there is interest then I can spend some more time on improving it) * deta[2]: a library for mapping between database tables and Racket structs * forms[3]: declarative web form (and JSON) validation -- different from formlets in that validation and presentation are separate * north[4]: database migrations * marionette[5]: control Firefox programmatically; great for end-to-end tests * postmark-client[6]: a client for https://postmarkapp.com * sentry[7]: runtime error reporting via https://sentry.io (they also offer an open source variant of the software which the client will work with just fine) * geoip[8]: geolocation using the MaxMind databases And a couple that are a little more niche and may not apply here: * mobilpay[9]: a client for a popular credit card gateway in Romania * twilio[10]: a client for the https://twilio.com API And some libraries I myself have found extremely helpful: * sql[11]: s-expression combinators for generating SQL * struct-plus-plus[12]: structs on steroids * gregor[13]: my personal favorite; a library for working with dates and times [1]: https://koyo.defn.io/ [2]: https://deta.defn.io/ [3]: https://docs.racket-lang.org/forms/index.html [4]: https://pkg-build.racket-lang.org/doc/north@north/index.html [5]: https://docs.racket-lang.org/marionette/index.html [6]: https://pkg-build.racket-lang.org/doc/postmark@postmark-client/index.html [7]: https://pkg-build.racket-lang.org/doc/sentry@sentry-doc/index.html [8]: https://docs.racket-lang.org/geoip/index.html [9]: https://github.com/Bogdanp/mobilpay [10]: https://github.com/Bogdanp/racket-twilio [11]: https://pkg-build.racket-lang.org/doc/sql@sql/index.html [12]: https://docs.racket-lang.org/struct-plus-plus/index.html [13]: https://docs.racket-lang.org/gregor/index.html -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m21ryjl602.fsf%40192.168.0.139.
[racket-users] [ann] marionette: control Firefox from Racket
Hey everybody! I just wrote `marionette`, a library that lets you control the Firefox web browser from Racket and I figured I'd share it! * the source code is here: https://github.com/bogdanp/marionette, * you can install the package by running `raco pkg install marionette` and * the docs will be available on the package server within the next day or so. It's still early days but a lot of the basic functionality you would expect is already supported. If you give it a try, let me know what you think! - Bogdan -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2lfyc2i6q.fsf%40192.168.0.139. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] i18n / translations
> How are people currently translating their programs? I was searching for a way to do l10n yesterday[1] and the best I could find was SRFI 29[2] from srfi-lib. [1]: https://github.com/Bogdanp/racket-webapp-template/commit/7647b2f2f460d1ede4f468e00f4dea62a541ee6e#diff-9978fbdfa51cde9daace605e88cc9c6c [2]: https://srfi.schemers.org/srfi-29/srfi-29.html -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[racket-users] ann: forms
Hey all, I needed a web form validation library last week and web-server-lib's formlets don't, afaict, provide any facilities for collecting and displaying validation errors to the end user so I made "forms" to scratch that particular itch. I hope it proves useful to some of you! Documentation: http://docs.racket-lang.org/forms/index.html Source code: https://github.com/Bogdanp/racket-forms - Bogdan -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] Re: Signal handling (or alternative)
Brian Adkins writes: > I just did a quick test, and "kill " will stop the Racket web server, > but not gracefully. In other words, it doesn't allow the current request to > finish. Maybe another signal will gracefully stop it? I personally run the server with serve[1] which runs it in a background thread and I keep a reference to the returned stopper function and finally do something along the lines of (with-handlers ([exn:break? (lambda (e) (stopper))]) (sync/enable-break never-evt)) in my main thread. As I understand it[2], exn:break? will be truthy for SIGINT, SIGTERM and SIGHUP (though there are also the exn:break:hang-up? and exn:break:terminate? predicates) on UNIX, so this should "catch" all of those and gracefully terminate my server. [1]: https://docs.racket-lang.org/web-server-internal/web-server.html?q=serve#%28def._%28%28lib._web-server%2Fweb-server..rkt%29._serve%29%29 [2]: https://docs.racket-lang.org/reference/breakhandler.html?q=breaks -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] Re: Multiple places listening to a TCP port [was: Racket application servers]
One way you might also solve this on UNIX systems is you could fork after
listening on the socket. The file descriptor for the socket from the parent
would then be shared with the children and so their accept calls would be "load
balanced" by way of them all accepting connections from the same queue. This
solves the problem without requiring SO_REUSEPORT (the disadvantage being that
there's only a single kernel-level accept queue with this approach, whereas
with SO_REUSEPORT each process would get its own queue).
Is there a way to fork(2) in Racket?
On Sun, Nov 25th, 2018 at 10:51 AM, George Neuner wrote:
>
>
>
> On 11/24/2018 7:39 PM, Philip McGrath wrote:
>
>
>> On Fri, Nov 23, 2018 at 5:44 PM George Neuner < gneun...@comcast.net >
>> wrote:
>>
>>
>>> Multiple (identical) server instances running on the same machine can
>>> listen for connections on the same network port - an incoming client call
>>> will be connected to only one of them. You only need to use different
>>> ports when services are not interchangeable [i.e. it matters which one you
>>> are talking to].
>>>
>>> Process parallelism, in effect, can give you automatic connection load
>>> balancing on a single machine. Using a separate load balancer in front
>>> technically is overkill for such situations, but it is a good design if
>>> you want to preserve the option to relocate your services to different
>>> machines.
>>>
>>>
>>
>>
>>
>> I didn't know this, and it sounds useful! Unfortunately, this
>> functionality doesn't currently seem to be exposed at the Racket level.
>> This example program:
>>
>>
>> #lang racket
>>
>> (provide launch-places)
>>
>> (module+ main
>> (launch-places))
>>
>> (define (launch-places [port 8011])
>> (define place-log-pch
>> (let-values ([{in-pch place-log-pch} (place-channel)])
>> (thread (λ ()
>> (let loop ()
>> (write-string (place-channel-get in-pch))
>> (loop
>> place-log-pch))
>> (define workers
>> (for/list ([n (in-range 0 2)])
>> (place/context p
>> (define (place-log msg)
>> (place-channel-put place-log-pch
>> (format "Place ~a: ~a\n" n msg)))
>> (parameterize
>> ([error-display-handler (λ (s v) (place-log s))])
>> (place-log "started")
>> (define listener
>> (tcp-listen port 4 'reuse))
>> (place-log "created listener")
>> ;; don't let listener be gc-ed
>> (tcp-accept listener)
>> (place-log "ended")
>> (apply sync (map place-dead-evt workers))
>> (for-each place-kill workers))
>>
>>
>> prints the following output on both Mac OS and Ubuntu:
>> Place 0: started
>> Place 0: created listener
>> Place 1: started
>> Place 1: tcp-listen: listen failed
>> port number: 8011
>> system error: Address already in use; errno=48
>>
>>
>> It looks like the `reuse?` argument to `tcp-listen` corresponds to
>> `SO_REUSEADDR` rather than `SO_REUSEPORT`. That's consistent with what
>> `udp-bind!` explicitly says it does, and its the only thing I can see
>> happening in the implementation (
>> https://github.com/racket/racket/blob/master/racket/src/rktio/rktio_network.c
>> )
>>
>>
>>
>> It seems like this might be a useful feature in Racket, but I'm not sure
>> of what would be needed to expose it in a good way. In particular, some
>> quick Googling revealed that there are some portability considerations,
>> which I haven't fully digested yet (
>> https://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t
>> ).
>>
>>
>> -Philip
>>
>>
>>
>>
>
>
>
> It's not working because the places in this code are OS threads in the
> SAME process. You need to start separate processes: using *dynamic-place*
> targeting localhost, or using *system* , *process* , *subprocess* , etc.
>
> AIUI 'reuse?' sets both SO_REUSEADDR and SO_REUSEPORT. But a single
> process can't open the same port twice.
>
> George
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
