Re: [Caml-list] Non-blocking IO interface design
On 8 Apr 2012, at 21:37, Daniel Bünzli wrote: I think that the first five points are mostly met and the cps transformation of blocking into non-blocking is relatively straightforward and remains readable in my opinion. Regarding the 6th point, using the included `setrip.native` program on 32 Mo of randomly generated s-expressions seem to indicate that: The non-blocking decoder can be at least 1.35 slower than blocking The non-blocking encoder can be at least 1.1 slower than blocking Now I don't think these bad numbers should be taken to dismiss the approach since in the context of a larger reactive program a blocking codec may actually incur performance and scability issues that cannot be shown by this example program. To track this down (as I'm learning about ocaml profiling at the moment), I firstly ran the blocking and non-blocking decoder through ocamlcp to check call counts, and: (non-blocking) let decode_nb_unix usize fd = (* 2 *) let rec loop d fd buf = (* 18039140 *) match Se.Nb.decode d with | `Lexeme l - (* 18035048 *) loop d fd buf | `End - (* 2 *) `Ok | `Error - (* 0 *) `Error | `Await - (* 4090 *) let rc = unix_read fd buf 0 (String.length buf) in Se.Nb.decode_src d buf 0 rc; loop d fd buf in loop (Se.Nb.decoder `Manual) fd (String.create usize) (blocking) let decode_b src = (* 1 *) let rec loop d = (* 9017525 *) match Se.B.decode d with | `Lexeme l - (* 9017524 *) loop d | `End - (* 1 *) `Ok | `Error - (* 0 *) `Error in loop (Se.B.decoder src) The I/O loop is being called twice for the non-blocking version, as it receives the `Await signal, does the Unix syscall, and then jumps into decode_src. Presumably a full non-blocking version would have to register with a select handler if it gets an EAGAIN at this point, In terms of the number of system calls, the non-blocking one is more efficient, as it uses a 16KB buffer versus the 4K reads done by the blocking version. $ strace ./setrip.native -dec -unix foo.sexp 21 | wc -l 2107 $ strace ./setrip.native -dec -unix -b foo.sexp 21 | wc -l 8238 Running 'perf stat' on the decoder on Linux shows that the non-blocking version is spending more CPU time doing something: $ perf stat --log-fd 1 ./setrip.native -dec -unix -b foo.sexp Performance counter stats for './setrip.native -dec -unix -b': 1095.337180 task-clock#1.000 CPUs utilized 1 context-switches #0.000 M/sec 0 CPU-migrations#0.000 M/sec 800 page-faults #0.001 M/sec $ perf stat --log-fd 1 ./setrip.native -dec -unix foo.sexp Performance counter stats for './setrip.native -dec -unix': 1339.360940 task-clock#1.000 CPUs utilized 2 context-switches #0.000 M/sec 0 CPU-migrations#0.000 M/sec 823 page-faults #0.001 M/sec Setting the GC to verbose shows that the non-blocking one is doing way more heap allocations than the blocking version (14 heap checks, versus 3). So this quick'n'dirty function is stuck in the source to bracket hot-looking calls and see how many minor heap allocations are taking place. let gcbracket fn a = let open Gc in compact (); let s1 = stat () in let r = fn a in let s2 = stat () in let r = fn a in Printf.eprintf gc: %.0f [%.0f %.0f %.0f / %.0f %0.f %0.f]\n%! (s2.minor_words -. s1.minor_words) s1.minor_words s1.promoted_words s1.major_words s2.minor_words s2.promoted_words s2.major_words; r Looking at the two decoders in src/se.ml, it looks like the non-blocking one allocates closures on every loop, which the blocking one doesn't. This is so it can store the continuation in d.k for the next loop. So putting gcbracket around the two decode calls confirms this. Switching to bytecode so we have accurate minor heap states, we have: For the blocking one: gc: 31 [1450548 0 44224 / 1450579 0 44224] gc: 31 [1451229 0 44255 / 1451260 0 44255] gc: 31 [1452506 0 44278 / 1452537 0 44278] gc: 31 [1453187 0 44309 / 1453218 0 44309] The important number is the first one (difference in minor heap size before and after the function, which is solidly 31. With the non-blocking one, more words are allocated in the minor heap: gc: 67 [5368 0 4684 / 5435 0 4684] gc: 42 [6096 0 4723 / 6138
Re: [Caml-list] Re: oasis packaging questions
On 8 Mar 2012, at 22:26, Sylvain Le Gall wrote: 2012/3/8 Adrien camarade...@gmail.com: Hi, On 08/03/2012, Sylvain Le Gall sylv...@le-gall.net wrote: Hi, 2012/3/8 Daniel Bünzli daniel.buen...@erratique.ch: Le jeudi, 8 mars 2012 ŕ 09:31, Sylvain Le Gall a écrit : setup.ml will be enough for me ;-) But I am biased. For distribution, I'm fine with that aswell. For developement setup.ml takes too much time to invoke (adds an overhead of 0.5s on my system). That the parsing time because the file is big. I know this issue and will try to improve that in the future. I think a 50% down size is possible, that will bring this time to 250ms. Although, I don't think that even for development env a 500ms delay is that big... I doubt to be able to reduce to ~0s. For my own stuff, I've been using a Makefile which includes a rule to (re)compile setup.ml into an executable with ocamlc.opt (compiling with ocamlopt was too slow). My usual development machine isn't terribly fast and this has helped a lot. I think there was a mention of an issue though but it can help a lot. Make sense. So super quick hack! Add at the beginning of setup.ml: A Makefile works just fine (and making the autogenerated code less complex rather than more would be better). This is the template Makefile I use: https://github.com/avsm/ocaml-github/blob/master/Makefile -anil -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
[Caml-list] working %.pp.ml target with ocamfind/ocamlbuild
There's a very useful %.pp.ml target in OCamlbuild that runs the source through camlp4 and outputs the result. This doesn't work when ocamlfind is used, as it includes the -pp flags within the ocamlfind invocation. I took a look at adding support for this into ocamlbuild (when -use-ocamlfind is specified), but I couldn't spot any way to get ocamlfind to output the syntax options. I just wanted to check that this is all in fact true, since it requires patching ocamlbuild, findlib, and oasis to make it work. Does everyone else manually run the camlp4o command when using syntax extensions like type-conv or pa_lwt? -anil -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Re: how could the community help with Oasis-DB; towards a CPAN for OCaml?
On 18 Dec 2011, at 20:01, Benedikt Meurer wrote: Am 18.12.2011 um 20:46 schrieb Ashish Agarwal agarwal1...@gmail.com: On Sun, Dec 18, 2011 at 4:35 AM, Benedikt Meurer benedikt.meu...@googlemail.com wrote: the OASIS port is named caml-oasis. Please try to transition to the consistent name OCaml, a decision made at the 2011 OCaml Users Meeting. Using it in lower case is okay in certain contexts such as a package name, so here I would recommend ocaml-oasis. This is the prefix used within MacPorts to identify OCaml packages. I don't know why it isn't ocaml. It's just historical. I imported a bunch of OCaml packages into Macports a few years ago and preserved the prefix since it wasn't really worth the trouble to rename so many ports. I've actually given up on MacPorts and switched to Homebrew recently. They don't like importing libraries in Homebrew and defer that to the language-specific package manager. I'm hoping that odb will be sufficient to act as the lightweight library installer to complement Homebrew. The only annoying thing about installing OCaml from source these days is the 'compiler-libs' convention that most packages seem to do, but isn't really written down anywhere. Is there a Mantis bug about this one (I couldn't find one)? It's really helpful to get the Lwt toplevel working and isn't very obvious where those files come from without diving into the Debian packaging. I've got a small pull request to add it to the Homebrew package at any rate. Comment on it to give it some life and it might even get merged... https://github.com/mxcl/homebrew/pull/9002 -anil -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Storing ocaml values outside ocaml's heap
I think Ancient is exactly what you're looking for. Just disable swap (which you should do anyway on a big modern server), and it will let you keep in-memory, out-of-heap OCaml values that don't get scanned by the GC. Anil On 8 Dec 2011, at 08:19, William Le Ferrand wrote: hi Stephane Yes, but it's not exactly what we're looking for : we plan to access the data at a (very) high rate, so swapping/unswapping would probably kill the performance .. On Wed, Dec 7, 2011 at 10:04 PM, Stéphane Glondu st...@glondu.net wrote: Le 08/12/2011 05:35, William Le Ferrand a écrit : We are building a cache in ocaml and we're wondering if it would make sense to store ocaml values outside the reach of the gc. (gc on a 20GB cache hangs the process for a second or so). Have you heard of the ancient [1] library? [1] http://git.annexia.org/?p=ocaml-ancient.git;a=summary Cheers, -- Stéphane -- William Le Ferrand Mobile : (+1) (415) 683-1484 Web : http://williamleferrand.github.com/ -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] [ANN] Async, a monadic concurrency library
On Wed, Oct 26, 2011 at 12:06:43PM +0100, Mark Shinwell wrote: On Wed, Oct 26, 2011 at 07:03:43AM -0400, Yaron Minsky wrote: I'll leave Stephen to answer that one. I'm just the front-man here. Stephen's the one who did all the actual work on Async... Actually, I think I'm probably responsible for this one. I believe the problem was a kernel bug/feature which caused an internal error code to be exposed to userspace instead of EINTR. Have you ever seen this outside of using ptrace/strace? If it actually leaks to userspace when not used on inside those, that would break a lot of other applications. Anyhow, directly checking the number is bad for portability reasons as the ERESTARTNOHAND is a Linux-ism. I'm taking a shot at porting Core/Async to OpenBSD at the moment, so will patch out that check in our local changes). -- Anil Madhavapeddy http://anil.recoil.org -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Lwt and exceptions
On 15 Sep 2011, at 11:04, Mehdi Dogguy wrote: On 13/09/2011 20:37, ri...@happyleptic.org wrote: The Lwt doc states that you should not use raise when using Lwt but use Lwt.fail instead. So, is it still OK to call functions (for instance from the stdlib) that may raise an exception, provided we catch it soon enough ? And by soon enough I mean: before an Lwt call that could schedule another thread. I guess, not (and it has been answered already). In fact, I was wondering if Lwt's authors would be against adding a function like: let wrap f x = try Lwt.return (f x) with e - Lwt.fail e It is stupid, trivial, etc… but looks what we need most of the time, no? Instead of doing it in our own code, it could land in Lwt directly. But, if it gets integrated into Lwt proper, users should be warned about its behaviour. (especially with impure functions). The try_lwt construct (in pa_lwt) or try_bind already converts normal exceptions into Lwt ones: try_lwt raise (Failure ) with Failure _ - print_endline Fail; Lwt.return () ...will print Fail. You just need to careful about raising exceptions across yield points when a try_lwt isn't present. The slightly painful thing about converting existing code to Lwt is that any function that raises an exception will gain the Lwt type if you convert it to use try_lwt/catch, even if the rest of the code doesn't otherwise block. Anil -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
[Caml-list] output-obj rule in ocamlbuild
Is there a built-in rule in ocamlbuild to generate the -output-obj output instead of a direct link? I note there are these built-in tags in the ocamlbuild source: ocaml_specific.ml:flag [ocaml; link; native; output_obj] (A-output-obj);; ocaml_specific.ml:flag [ocaml; link; byte; output_obj] (A-output-obj);; ...but I can't find a built-in target that uses these tags. Is a custom rule in myocamlbuild.ml needed here, or should I do something in my _tags file? cheers, Anil -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
