[Caml-list] Re: Looking for stubs for sendmsg/recvmsg
On 18-11-2010, Goswin von Brederlow goswin-...@web.de wrote: Hi, I'm looking for stubs for ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags); ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags); Specifically I need those to send (among normal messages) an Unix.file_descr over a Unix Domain Socket. Does anyone know of a module that has them? If you don't find one and plan to write it yourself, this would be a good addition to extunix: http://extunix.forge.ocamlcore.org Regards, Sylvain Le Gall ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
[Caml-list] Re: ocamlbuild and packs
On 18-11-2010, Thomas Gazagnaire thomas.gazagna...@inria.fr wrote: Hi all, I've got a source tree with the following patterns : A/a.ml (defines 'let x = 1') B/a.ml (use A.x and defines 'let y = 2') B/b.ml B/b.mlpack (contains 'A B') C/a.ml (use A.x) C/b.ml (use B.A.y) C/c.mlpack (contains 'A B') Is there any way with ocamlbuild to build that tree (if possible using only _tags files, but if not I would be happy to have at least a solution :-) ? You can try to start building A/a.mlpack and use the generated library in other libraries. Regards, Sylvain Le Gall ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] SMP multithreading
On Thu, 18 Nov 2010 00:08:19 +0100, Christophe Raffalli wrote: And OCaml on GPU ? We just tested a recent GPU card with 480 processors at 900Mhz ... this is qui impressive ... and supported by matlab via cuda-lapack (http://www.culatools.com/) ... I imagine we could at least use cuda-lapack from OCaml ? This is certainly possible since they say that the standard LAPACK functions are available. If you try, let us know! Best, C. ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Re: Native toplevel?
Benedikt Meurer wrote, On 11/19/2010 06:29 AM: This is indeed very interesting. I haven't thought of the native top-level. I haven't done any measurements yet, but from my experience with ocamlopt, I know that the optimizing native compiler is somewhat slower than the byte-code compiler. I doubt that this is related solely to the external as/ld invocations, so there's certainly more work to do than just replacing the calls to as/ld with an in-process assembler/linker. Indeed, there are some more passes in ocamlopt than in ocamlc, but I think that most of the time is spent in allocating registers and generating the assembler file and calling the assembler. As Alain said, we already have the inline assembler, so the next step would be to improve register allocation, either by spilling more variables to avoid the quadratic behavior of graph coloring, either by using some linear scan algorithm (for example, the one of HotSpot). --Fabrice ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
RE: [Caml-list] Looking for stubs for sendmsg/recvmsg
Hi, I'm looking for stubs for ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags); ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags); Specifically I need those to send (among normal messages) an Unix.file_descr over a Unix Domain Socket. Does anyone know of a module that has them? We have some bindings for those: https://github.com/xen-org/xen-api-libs/blob/master/stdext/unixext.mli external send_fd : Unix.file_descr - string - int - int - Unix.msg_flag list - Unix.file_descr - int = stub_unix_send_fd_bytecode stub_unix_send_fd external recv_fd : Unix.file_descr - string - int - int - Unix.msg_flag list - int * Unix.sockaddr * Unix.file_descr = stub_unix_recv_fd You might prefer to extract the relevant functions from the code -- there's a lot of other misc stuff in that repo which you're probably not interested in. We use those functions quite a lot so hopefully they'll work for you. Cheers, Dave ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] help with ocaml makefiles
On Thu, Nov 18, 2010 at 08:43:28AM +0100, ri...@happyleptic.org wrote: Each time I need more debug information for the ocaml runtime itself (which hopefully is not often) I recompile Ocaml adding various -g and -O0 here and there in some makefiles. See the description of how to build ocamlrund in the INSTALL file. -- Eric Cooper e c c @ c m u . e d u ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] SMP multithreading
There seem to be solutions in theory. I think a colleague had pointed out one of the papers below, so there is indeed something like a lock-free garbage collector. Then, why do we worry about much synchronization overhead? I don't quite understand. Maurice Herlihyhttp://www.informatik.uni-trier.de/~ley/db/indices/a-tree/h/Herlihy:Maurice.html, J. Eliot B. Moss: Lock-Free Garbage Collection for Multiprocessors. IEEE Trans. Parallel Distrib. Syst. 3http://www.informatik.uni-trier.de/~ley/db/journals/tpds/tpds3.html#HerlihyM92(3): 304-311 (1992) http://doi.ieeecomputersociety.org/10.1109/71.139204 Hui Gao, Jan Friso Grootehttp://www.informatik.uni-trier.de/~ley/db/indices/a-tree/g/Groote:Jan_Friso.html , Wim H. Hesselinkhttp://www.informatik.uni-trier.de/~ley/db/indices/a-tree/h/Hesselink:Wim_H=.html: Lock-free parallel and concurrent garbage collection by marksweep. Sci. Comput. Program. 64http://www.informatik.uni-trier.de/~ley/db/journals/scp/scp64.html#GaoGH07(3): 341-374 (2007) http://portal.acm.org/citation.cfm?id=1223239 Java's new garbage collector is lock-free? At any rate, we really needn't fall behind a mega-lame language like Java :) The first paper is from 1992, enough time for the knowledge to diffuse. The second 2007 paper is probably what Jon was referring to earlier. In my mind, you can use one of these, and use special pool allocation algorithms for small objects, and also use static lifetime analysis to bypass the garbage collection in many cases. Since there are many runtime designers here, I wonder, is there a language runtime that does all three of these? Cheers, Eray http://doi.ieeecomputersociety.org/10.1109/71.139204 On Wed, Nov 17, 2010 at 6:34 PM, David Allsopp dra-n...@metastack.comwrote: Edgar Friendly wrote: It looks like high-performance computing of the near future will be built out of many machines (message passing), each with many cores (SMP). One could use message passing for all communication in such a system, but a hybrid approach might be best for this architecture, with use of shared memory within each box and message passing between. Of course the best choice depends strongly on the particular task. Absolutely - and the problem in OCaml seems to be that shared memory parallelism is just branded as evil and ignored... In the long run, it'll likely be a combination of a few large, powerful cores (Intel-CPU style w/ the capability to run a single thread as fast as possible) with many many smaller compute engines (GPGPUs or the like, optimized for power and area, closely coupled with memory) that provides the highest performance density. I think the central thing that we can be utterly sure about is that desktops will always have * 1* general purpose CPU. Maybe not be an ever-increasing number of cores but definitely more than one. The question of how to program such an architecture seems as if it's being answered without the functional community's input. What can we contribute? It has often seemed to me when SMP has been discussed in the past on this list that it almost gets dismissed out of hand because it doesn't look future-proof or because we're worried about what's round the corner in technology terms. To me the principal question is not about whether a parallel/thread-safe GC will scale to 12, 16 or even the 2048 cores on something like http://www.hpc.cam.ac.uk/services/darwin.html but whether it will hurt a single-threaded application - i.e. whether you will still be able to implement message passing libraries and other scalable techniques without the parallel GC getting in the way of what you're doing. A parallel/thread-safe GC should be aiming to provide the same sort of contract as the present one - it just works for most things and in a few borderline cases (like HPC - yes, it's a borderline case) you'll need to tune your code or tweak GC parameters because it's causing some problems or because in your particular application squeezing every cycle out of the CPU is important. As long as the GC isn't (hugely) slower than the present one in OCaml then we can continue to use libraries, frameworks and technologies-still-to-come on top of a parallel/thread-safe GC which simply ignores shared memory thread-level parallelism just by not instantiating threads. The argument always seems to focus on utterly maxing out all possible available resources (CPU time, memory bandwidth, etc.) rather than on whether it's simply faster than what we're doing able to do at the moment on the same system. Of course, it may be that the only way to do that is to have different garbage collectors - one invoked when threads.cmxa is linked and then the normal one otherwise (that's so easy to type out as a sentence, summarising a vast amount of potential work!!) Multithreading in OCaml seems to be focused on jumping the entire width of the river of concurrency in one go, rather than coming up with stepping
[Caml-list] OCamlJit 2.0
Hello everybody, OCamlJit 2.0 is a new Just-In-Time engine for Objective Caml 3.12.0 on desktop processors (x86/x86-64). It translates the OCaml byte-code used by the interpreter (ocamlrun and ocaml) to x86/x86-64 native code on-demand and runs the generated native code instead of interpreting the byte-code. It is designed to run with minimal compilation overhead (translating only what is being executed, avoiding costly code generation and optimization techniques), while being 100% compatible with the byte-code runtime (including serialization and hashing of closures, etc.). OCamlJit 2.0 was specifically designed for desktop processors and is not really portable to anything else in its current shape, because the target audience are people using the interactive top-level and the byte-code interpreter for rapid prototyping/development (which is unlikely to happen on anything else but x86/x86-64). The implementation currently requires a system that adheres to the SysV ABI, which includes Linux, BSD, OS X, but excludes Win32/Win64 (patches/ideas are welcome). It was tested on Linux/x86 (Debian), Linux/amd64 (CentOS) and Mac OS X 10.6 (64bit). The x86 implementation requires SSE2 capable processors (otherwise it falls back to the byte-code interpreter), so it won't speedup your OCaml programs running on 486 CPUs. :-) OCamlJit 2.0 runs most benchmarks at 2-6 times faster than the byte-code interpreter. The interactive top-level benefits twice when used with the JIT engine: (a) the compiler stages are JIT compiled and (b) the generated byte-code is JIT compiled. A tech report describing a slightly earlier prototype and including performance measures of OCamlJit 2.0 on Mac OS X (64bit) is available at: http://arxiv.org/abs/1011.1783 The source code is available from the Git repository (master branch) at: http://gitorious.org/ocamljit2/ocamljit2 Installation is similar to installation of Objective Caml, just run ./configure -prefix /path/to/ocamljit2 [options] followed by make world opt make install This will install a fully working Objective Caml 3.12.0 to /path/to/ocamlji2, where /path/to/ocamljit2/bin/ocamlrun and /path/to/ocamljit2/lib/libcamlrun_shared.so include the JIT engine in addition to the byte-code interpreter (fallback to the byte-code interpreter is necessary for debugging with ocamldebug). The configure script prints a line indicating whether the JIT engine is enabled or not (if not, it'll be just a regular OCaml 3.12 installation). Comments are welcome. Enjoy! Benedikt Meurer ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Optimizing garbage collection
On 11/18/2010 09:51 AM, Eray Ozkural wrote: A program I wrote constructs a lot of small lists, and strings and discards them. It's a search algorithm. I profiled this code and saw that garbage collection takes significant time. In C++, we can write custom allocators to optimize the data structures that cause such slowdowns. Any recommended strategies in ocaml? The OCaml garbage collector exposes a variety of tuning parameters through the Gc module[1] and the OCAMLRUNPARAM environment variable[2]. I would tweak those. In particular, I would recommend increasing the minor heap size so that more of your data can be quickly discarded. You can also increase the space overhead, thereby causing the GC to be less aggressive at the expense of higher memory usage/more waste. Lastly, I often increase the heap increment to allow memory to allow the heap to expand more quickly, but I do not know if that will help in your case or not. I have documented my practices more thoroughly at my blog[3]. As I see it, the biggest gains will be by tuning your code and your minor heap size so that ephemeral structures never hit the major heap. My rule of thumb is that one work unit, if you have such a concept, should fit in the minor heap. Collecting dead structures from the minor heap is fast; moving a structure to the major heap only to have it be unreachable by the next GC cycle can cause substantial GC thrashing. You're on to a good start, though, by measuring. I use gprof heavily as I tweak my code's performance. - Michael 1. http://caml.inria.fr/pub/docs/manual-ocaml/libref/Gc.html 2. http://caml.inria.fr/pub/docs/manual-ocaml/manual024.html#toc96 3. http://elehack.net/michael/blog/2010/06/ocaml-memory-tuning ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] help with ocaml makefiles
On 18 Nov 2010, at 07:43, ri...@happyleptic.org wrote: Each time I need more debug information for the ocaml runtime itself (which hopefully is not often) I recompile Ocaml adding various -g and -O0 here and there in some makefiles. But I've noticed there are some special targets and rules, specifically in byterun/Makefile, that seams to be there for that very purpose (the suffix rules for .d.c extention). So I suppose there is a standard way to compile ocaml in debugging mode. Do someone have some knowledge to share about this ? The C files in byterun/ are compiled with and without debug/profiling, and the following files are created: - libasmrun.a (normal) - libasmrunp.a (profiling) - libasmrund.a (debug) Only the normal and profiling ones are installed by default. You can grab the debug version from your ocaml build tree and link against it manually, with something like: $ ocamlopt -output-obj -i foo.o foo.ml $ cc -lasmrund -lm -o foo foo.o It'd be convenient to have a compiler option to select libasmrund without having to go via -output-obj (as -p does for profiling); the code for this is in asmcomp/asmlink.ml:runtime_lib(). Anil ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Optimizing garbage collection
Thanks a lot for the suggestions Michael. Much appreciated. Best, -- Eray Ozkural, PhD candidate. Comp. Sci. Dept., Bilkent University, Ankara http://groups.yahoo.com/group/ai-philosophy http://myspace.com/arizanesil http://myspace.com/malfunct ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] SMP multithreading
Christophe TROESTLER christophe.troestler+oc...@umh.ac.be writes: On Thu, 18 Nov 2010 00:08:19 +0100, Christophe Raffalli wrote: And OCaml on GPU ? We just tested a recent GPU card with 480 processors at 900Mhz ... this is qui impressive ... and supported by matlab via cuda-lapack (http://www.culatools.com/) ... I imagine we could at least use cuda-lapack from OCaml ? This is certainly possible since they say that the standard LAPACK functions are available. If you try, let us know! Best, C. And the functions should enter/leave_blocking_section() in the C stubs so you can have 480 ocaml threads. All of them can run some lapack code while always only one can run ocaml code at any one time. If the lapack functions take long enough almost all threads will be running. This is actually a quick way to use multiple cores with ocaml. Find a often called function that takes considerable time and offload it to C with enter/leave_blocking_section() around it. Isn't always possible and you need to use BigArray for data or copy the arguments. MfG Goswin ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Re: Looking for stubs for sendmsg/recvmsg
Sylvain Le Gall sylv...@le-gall.net writes: On 18-11-2010, Goswin von Brederlow goswin-...@web.de wrote: Hi, I'm looking for stubs for ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags); ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags); Specifically I need those to send (among normal messages) an Unix.file_descr over a Unix Domain Socket. Does anyone know of a module that has them? If you don't find one and plan to write it yourself, this would be a good addition to extunix: http://extunix.forge.ocamlcore.org Regards, Sylvain Le Gall I'm thinking of changing Unix.file_descr from int to a custom block (containing the FD) with finalizer. Unix.close would set the FD to -1 and the finalizer gives an error if FD != -1 and closes it. Actually I want that tunable with 3 possible behaviours: type fd_leak_mode = Silent | Complain | Fail val set_leak_mode : fd_leak_mode - unit = fun Silent just closes the FD if it is still open, Complain (default) outputs to stderr and closes it and Fail aborts. That would change most of the Unix module and mean a complete fork of it. Would that be something for extunix too? MfG Goswin ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Looking for stubs for sendmsg/recvmsg
Dave Scott dave.sc...@eu.citrix.com writes: Hi, I'm looking for stubs for ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags); ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags); Specifically I need those to send (among normal messages) an Unix.file_descr over a Unix Domain Socket. Does anyone know of a module that has them? We have some bindings for those: https://github.com/xen-org/xen-api-libs/blob/master/stdext/unixext.mli external send_fd : Unix.file_descr - string - int - int - Unix.msg_flag list - Unix.file_descr - int = stub_unix_send_fd_bytecode stub_unix_send_fd external recv_fd : Unix.file_descr - string - int - int - Unix.msg_flag list - int * Unix.sockaddr * Unix.file_descr = stub_unix_recv_fd You might prefer to extract the relevant functions from the code -- there's a lot of other misc stuff in that repo which you're probably not interested in. We use those functions quite a lot so hopefully they'll work for you. Cheers, Dave Neigther one is directly suitable but cutpaste into my own stub is easy enough with that. Thanks Dave and Jérémie. MfG Goswin ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
[Caml-list] Re: Looking for stubs for sendmsg/recvmsg
Hello, On 19-11-2010, Goswin von Brederlow goswin-...@web.de wrote: Sylvain Le Gall sylv...@le-gall.net writes: On 18-11-2010, Goswin von Brederlow goswin-...@web.de wrote: Hi, I'm looking for stubs for ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags); ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags); Specifically I need those to send (among normal messages) an Unix.file_descr over a Unix Domain Socket. Does anyone know of a module that has them? If you don't find one and plan to write it yourself, this would be a good addition to extunix: http://extunix.forge.ocamlcore.org Regards, Sylvain Le Gall I'm thinking of changing Unix.file_descr from int to a custom block (containing the FD) with finalizer. Unix.close would set the FD to -1 and the finalizer gives an error if FD != -1 and closes it. Actually I want that tunable with 3 possible behaviours: type fd_leak_mode = Silent | Complain | Fail val set_leak_mode : fd_leak_mode - unit = fun Silent just closes the FD if it is still open, Complain (default) outputs to stderr and closes it and Fail aborts. That would change most of the Unix module and mean a complete fork of it. Not that much, if you proceed in another way. I think what you are looking for is a fd leak detector? Here is a small modules that I have written for this purpose: File UnixExt.ml: (** Count open/close call *) IFNDEF NDBUG THEN let fd_opened = Hashtbl.create 13 ;; let fd_once_opened = Hashtbl.create 13 ;; let fd_open fd fn out = dbug_print (fun () - Printf.sprintf %s '%s' (if out then open-out else open-in) fn); Hashtbl.add fd_opened fd (fn, out) ;; let fd_close fd = try let (fn, out) as data = Hashtbl.find fd_opened fd in dbug_print (fun () - Printf.sprintf %s '%s' (if out then close-out else close-in) fn); Hashtbl.add fd_once_opened fd data; Hashtbl.remove fd_opened fd; with Not_found - begin dbug_print (fun () - let fn = try fst (Hashtbl.find fd_once_opened fd) with Not_found - unknown in Printf.sprintf Trying to close %s again fn) end ;; let () = at_exit (fun () - let exit_error = ref false in Hashtbl.iter (fun fd (fn, out) - if fd Unix.stdin fd Unix.stdout fd Unix.stderr then begin Printf.eprintf Not closed '%s' (out: %b)\n fn out; exit_error := true end) fd_opened; Hashtbl.clear fd_opened; if !exit_error then exit 3 ) ;; let opened_files () = let lst = ref [] in Hashtbl.iter (fun _ e - lst := e :: !lst) fd_opened; List.sort compare !lst; ELSE let fd_open _ _ _ = () ;; let fd_close _ = () ;; let opened_files () = [] ;; ENDIF (** See UnixExt.mli *) let to_file_descr_in fd = fd_open fd converted false; fd ;; (** See UnixExt.mli *) let to_file_descr_out fd = fd_open fd converted true; fd ;; (** See UnixExt.mli *) let close_in fd = Unix.close fd; fd_close fd ;; (** See UnixExt.mli *) let stdout = fd_open Unix.stdout stdout true; Unix.stdout ;; [...override other functions that open/close fd...] Then in the modules using this features, you just have to open UnixExt after Unix... You can even probably design a library that will transparently hide Unix with a custom Unix module providing this feature. Would that be something for extunix too? I don't think so. At least, this is not currently the purpose of extunix... Regards, Sylvain Le Gall ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Native toplevel? (was: OCamlJit 2.0)
Hello, 2010/11/18 Ashish Agarwal agarwal1...@gmail.com: Rapid prototyping for me often involves a couple of lines of code that read in a very large file and do something with it. I have to keep compiling these small programs to native code because the performance of the toplevel is too slow. Then, I have to recompile and re-read the whole file for every little additional thing I want to compute. A high-performance toplevel would help in this kind of work. Or use ocamlscript: http://martin.jambon.free.fr/ocamlscript.html Sincerely yours, david ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Native toplevel? (was: OCamlJit 2.0)
On Fri, Nov 19, 2010 at 1:09 PM, David MENTRE dmen...@linux-france.org wrote: Hello, 2010/11/18 Ashish Agarwal agarwal1...@gmail.com: Rapid prototyping for me often involves a couple of lines of code that read in a very large file and do something with it. I have to keep compiling these small programs to native code because the performance of the toplevel is too slow. Then, I have to recompile and re-read the whole file for every little additional thing I want to compute. A high-performance toplevel would help in this kind of work. Or use ocamlscript: http://martin.jambon.free.fr/ocamlscript.html ocamlscript is certainly a wonderful tool, for prototyping and otherwise. It unfortunately doesn't help specifically with the load a large file and do something with it case. A native-code toplevel allows you to keep the native code speed benefits and load the file only once. Interactive experimentation on the file's data then doesn't require waiting for the data to be read in each time. Hez ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Native toplevel? (was: OCamlJit 2.0)
On Fri, Nov 19, 2010 at 1:24 PM, Hezekiah M. Carty hca...@atmos.umd.eduwrote: ocamlscript is certainly a wonderful tool, for prototyping and otherwise. It unfortunately doesn't help specifically with the load a large file and do something with it case. Right. Also, I should mention that a high-performance toplevel, combined with new libraries like OCaml-R, would essentially make OCaml a competitor to Matlab and R. This would really expand OCaml's scope to an important area. ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Re: Looking for stubs for sendmsg/recvmsg
Sylvain Le Gall sylv...@le-gall.net writes: Hello, On 19-11-2010, Goswin von Brederlow goswin-...@web.de wrote: Sylvain Le Gall sylv...@le-gall.net writes: On 18-11-2010, Goswin von Brederlow goswin-...@web.de wrote: Hi, I'm looking for stubs for ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags); ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags); Specifically I need those to send (among normal messages) an Unix.file_descr over a Unix Domain Socket. Does anyone know of a module that has them? If you don't find one and plan to write it yourself, this would be a good addition to extunix: http://extunix.forge.ocamlcore.org Regards, Sylvain Le Gall I'm thinking of changing Unix.file_descr from int to a custom block (containing the FD) with finalizer. Unix.close would set the FD to -1 and the finalizer gives an error if FD != -1 and closes it. Actually I want that tunable with 3 possible behaviours: type fd_leak_mode = Silent | Complain | Fail val set_leak_mode : fd_leak_mode - unit = fun Silent just closes the FD if it is still open, Complain (default) outputs to stderr and closes it and Fail aborts. That would change most of the Unix module and mean a complete fork of it. Not that much, if you proceed in another way. I think what you are looking for is a fd leak detector? Here is a small modules that I have written for this purpose: File UnixExt.ml: (** Count open/close call *) IFNDEF NDBUG THEN let fd_opened = Hashtbl.create 13 ;; let fd_once_opened = Hashtbl.create 13 ;; let fd_open fd fn out = dbug_print (fun () - Printf.sprintf %s '%s' (if out then open-out else open-in) fn); Hashtbl.add fd_opened fd (fn, out) ;; let fd_close fd = try let (fn, out) as data = Hashtbl.find fd_opened fd in dbug_print (fun () - Printf.sprintf %s '%s' (if out then close-out else close-in) fn); Hashtbl.add fd_once_opened fd data; Hashtbl.remove fd_opened fd; with Not_found - begin dbug_print (fun () - let fn = try fst (Hashtbl.find fd_once_opened fd) with Not_found - unknown in Printf.sprintf Trying to close %s again fn) end ;; let () = at_exit (fun () - let exit_error = ref false in Hashtbl.iter (fun fd (fn, out) - if fd Unix.stdin fd Unix.stdout fd Unix.stderr then begin Printf.eprintf Not closed '%s' (out: %b)\n fn out; exit_error := true end) fd_opened; Hashtbl.clear fd_opened; if !exit_error then exit 3 ) ;; let opened_files () = let lst = ref [] in Hashtbl.iter (fun _ e - lst := e :: !lst) fd_opened; List.sort compare !lst; ELSE let fd_open _ _ _ = () ;; let fd_close _ = () ;; let opened_files () = [] ;; ENDIF (** See UnixExt.mli *) let to_file_descr_in fd = fd_open fd converted false; fd ;; (** See UnixExt.mli *) let to_file_descr_out fd = fd_open fd converted true; fd ;; (** See UnixExt.mli *) let close_in fd = Unix.close fd; fd_close fd ;; (** See UnixExt.mli *) let stdout = fd_open Unix.stdout stdout true; Unix.stdout ;; [...override other functions that open/close fd...] Then in the modules using this features, you just have to open UnixExt after Unix... You can even probably design a library that will transparently hide Unix with a custom Unix module providing this feature. Much less usefull. Using a custom block with finalizer means that the FD will be closed relative close to where/when it was leaked. Makes it easier to find where it was leaked and adding GC.compact calls at strategic locations can narrow it down even more. Leaking FDs also becomes much less serious. The GC will clean them up and close them. So you can use an app that leaks FDs just fine. MfG Goswin ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Native toplevel? (was: OCamlJit 2.0)
On Nov 19, 2010, at 19:30 , Ashish Agarwal wrote: On Fri, Nov 19, 2010 at 1:24 PM, Hezekiah M. Carty hca...@atmos.umd.edu wrote: ocamlscript is certainly a wonderful tool, for prototyping and otherwise. It unfortunately doesn't help specifically with the load a large file and do something with it case. Right. Also, I should mention that a high-performance toplevel, combined with new libraries like OCaml-R, would essentially make OCaml a competitor to Matlab and R. This would really expand OCaml's scope to an important area. This was actually the main motivation for OCamlJit, tho we did not specifically need a competitor to Matlab (we use tuareg mode most of the time, to prototype term rewriting/typing stuff, but also for some complex LaTeX generation tasks). After thinking about it for a few days now, I think a native toplevel would indeed solve the problem, in a better way than a byte-code JITter. I'll try to look into the remaining issues with the native toplevel, but I have some other stuff to finish first. Benedikt ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] OCamlJit 2.0
On Nov 16, 2010, at 6:52 AM, Benedikt Meurer wrote: Hello everybody, Hi, OCamlJit 2.0 is a new Just-In-Time engine for Objective Caml 3.12.0 on desktop processors (x86/x86-64). It translates the OCaml byte-code used by the interpreter (ocamlrun and ocaml) to x86/x86-64 native code on-demand and runs the generated native code instead of interpreting the byte-code. It is designed to run with minimal compilation overhead (translating only what is being executed, avoiding costly code generation and optimization techniques), while being 100% compatible with the byte-code runtime (including serialization and hashing of closures, etc.). OCamlJit 2.0 was specifically designed for desktop processors and is not really portable to anything else in its current shape, because the target audience are people using the interactive top-level and the byte-code interpreter for rapid prototyping/development The target audience seems quite small to me. I think this project is very interesting from an educational point of view (to understand more the internals of OCaml) but I doubt we really need a JIT for OCaml. ocamlc + ocamlopt are very hard to beat. What would be really nice is to make a JIT for a language that really need one, like PHP! There are lots of companies out there (Yahoo, Facebook, wikimedia) that spend hundreds of millions of dollars on machines that run PHP bytecode interpreters implemented by people who are not Xavier Leroy. It's not trivial to statically determine types in PHP which prevent to write really efficient compilers for PHP. JIT are perfect in such situation. I started to develop some basic compiler infrastructure around PHP at https://github.com/facebook/pfff if you are interested. Improving the current PHP implementation by 5% can save millions of dollars to Wikimedia. Think about it next time you see Jimmy Wales asking for money to fund Wikipedia; there are many ways computer scientists can help him. (which is unlikely to happen on anything else but x86/x86-64). The implementation currently requires a system that adheres to the SysV ABI, which includes Linux, BSD, OS X, but excludes Win32/Win64 (patches/ideas are welcome). It was tested on Linux/x86 (Debian), Linux/amd64 (CentOS) and Mac OS X 10.6 (64bit). The x86 implementation requires SSE2 capable processors (otherwise it falls back to the byte-code interpreter), so it won't speedup your OCaml programs running on 486 CPUs. :-) OCamlJit 2.0 runs most benchmarks at 2-6 times faster than the byte-code interpreter. The interactive top-level benefits twice when used with the JIT engine: (a) the compiler stages are JIT compiled and (b) the generated byte-code is JIT compiled. A tech report describing a slightly earlier prototype and including performance measures of OCamlJit 2.0 on Mac OS X (64bit) is available at: http://arxiv.org/abs/1011.1783 The source code is available from the Git repository (master branch) at: http://gitorious.org/ocamljit2/ocamljit2 Installation is similar to installation of Objective Caml, just run ./configure -prefix /path/to/ocamljit2 [options] followed by make world opt make install This will install a fully working Objective Caml 3.12.0 to /path/to/ocamlji2, where /path/to/ocamljit2/bin/ocamlrun and /path/to/ocamljit2/lib/libcamlrun_shared.so include the JIT engine in addition to the byte-code interpreter (fallback to the byte-code interpreter is necessary for debugging with ocamldebug). The configure script prints a line indicating whether the JIT engine is enabled or not (if not, it'll be just a regular OCaml 3.12 installation). Comments are welcome. Enjoy! Benedikt Meurer ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] OCamlJit 2.0
On Nov 19, 2010, at 19:43 , Yoann Padioleau wrote: OCamlJit 2.0 was specifically designed for desktop processors and is not really portable to anything else in its current shape, because the target audience are people using the interactive top-level and the byte-code interpreter for rapid prototyping/development The target audience seems quite small to me. I think this project is very interesting from an educational point of view (to understand more the internals of OCaml) but I doubt we really need a JIT for OCaml. ocamlc + ocamlopt are very hard to beat. What would be really nice is to make a JIT for a language that really need one, like PHP! There are lots of companies out there (Yahoo, Facebook, wikimedia) that spend hundreds of millions of dollars on machines that run PHP bytecode interpreters implemented by people who are not Xavier Leroy. It's not trivial to statically determine types in PHP which prevent to write really efficient compilers for PHP. JIT are perfect in such situation. I started to develop some basic compiler infrastructure around PHP at https://github.com/facebook/pfff if you are interested. Improving the current PHP implementation by 5% can save millions of dollars to Wikimedia. Think about it next time you see Jimmy Wales asking for money to fund Wikipedia; there are many ways computer scientists can help him. Well, it is a research project, and it was driven by actual demand. A JIT engine for PHP is something less interesting from a university point of view, unless there are companies willing to sponsor/help the development. But from my personal experience, there is not really a lot to gain w.r.t. PHP. Delivering website content does not involve complex computations or processing, it is mostly I/O bound, depending on a fast database engine, a fast webserver, decent text processing throughput, etc. I may be wrong here, but I doubt that you'd see relevant speedups on large websites by simply JITting the PHP code. Also PHP code is less likely to change at runtime, so there's no real need to acutally JIT compile it. You could use a lot simpler techniques here to improve performance. For example, just write a simple PHP to C compiler, compiling your PHP code to native code via C, and let the webserver run the native code instead. With some clever compilation scheme, this should outperform any JIT engine, with a lot less effort. Anyway, this seems to be off-topic here... Benedikt ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Re: Native toplevel?
On Nov 19, 2010, at 11:02 , Fabrice Le Fessant wrote: Benedikt Meurer wrote, On 11/19/2010 06:29 AM: This is indeed very interesting. I haven't thought of the native top-level. I haven't done any measurements yet, but from my experience with ocamlopt, I know that the optimizing native compiler is somewhat slower than the byte-code compiler. I doubt that this is related solely to the external as/ld invocations, so there's certainly more work to do than just replacing the calls to as/ld with an in-process assembler/linker. Indeed, there are some more passes in ocamlopt than in ocamlc, but I think that most of the time is spent in allocating registers and generating the assembler file and calling the assembler. As Alain said, we already have the inline assembler, so the next step would be to improve register allocation, either by spilling more variables to avoid the quadratic behavior of graph coloring, either by using some linear scan algorithm (for example, the one of HotSpot). Using the linear scan algorithm with ocamlopt might be a good idea, due to the structure of the generated code, linear scan should give results close to that of the graph coloring algorithm for the generated native code. I'll see if I can hire some students to evaluate an implementation of linear scan for ocamlopt. --Fabrice Benedikt ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] OCamlJit 2.0
Hi, What would be really nice is to make a JIT for a language that really need one, like PHP! There are lots of companies out there (Yahoo, Facebook, wikimedia) that spend hundreds of millions of dollars on machines that run PHP bytecode interpreters implemented by people who are not Xavier Leroy. Actually, Facebook has a compiler that transforms PHP source code into C++ [1], and they claim a 50% reduction in CPU usage. It's an interesting approach, though I suspect that even as we speak a new circle of hell is being warmed up especially for those who are enabling PHP's continued existence thanks to gimmicks such as this one... Cheers, Dario Teixeira [1] https://github.com/facebook/hiphop-php/wiki/ ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] OCamlJit 2.0
On Nov 19, 2010, at 11:46 AM, Dario Teixeira wrote: Hi, What would be really nice is to make a JIT for a language that really need one, like PHP! There are lots of companies out there (Yahoo, Facebook, wikimedia) that spend hundreds of millions of dollars on machines that run PHP bytecode interpreters implemented by people who are not Xavier Leroy. Actually, Facebook has a compiler that transforms PHP source code into C++ [1], and they claim a 50% reduction in CPU usage. Yes, which is good. But if you think about it is only a x2 speedup vs a really slow bytecode interpreter (the Zend PHP interpreter). PHP is known for being more than 30 times slower than C. It's even slower than Ruby on http://shootout.alioth.debian.org/u32/which-programming-languages-are-fastest.php There are lots of opportunities to do better IMHO. It's an interesting approach, though I suspect that even as we speak a new circle of hell is being warmed up Oh yeah. especially for those who are enabling PHP's continued existence thanks to gimmicks such as this one... Cheers, Dario Teixeira [1] https://github.com/facebook/hiphop-php/wiki/ ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
[Caml-list] Re: Looking for stubs for sendmsg/recvmsg
On 19-11-2010, Goswin von Brederlow goswin-...@web.de wrote: Sylvain Le Gall sylv...@le-gall.net writes: On 19-11-2010, Goswin von Brederlow goswin-...@web.de wrote: Sylvain Le Gall sylv...@le-gall.net writes: On 18-11-2010, Goswin von Brederlow goswin-...@web.de wrote: Not that much, if you proceed in another way. I think what you are looking for is a fd leak detector? Here is a small modules that I have written for this purpose: File UnixExt.ml: (** Count open/close call *) [...] [...override other functions that open/close fd...] Then in the modules using this features, you just have to open UnixExt after Unix... You can even probably design a library that will transparently hide Unix with a custom Unix module providing this feature. Much less usefull. Using a custom block with finalizer means that the FD will be closed relative close to where/when it was leaked. Makes it easier to find where it was leaked and adding GC.compact calls at strategic locations can narrow it down even more. Leaking FDs also becomes much less serious. The GC will clean them up and close them. So you can use an app that leaks FDs just fine. It all depends on what you want: fix your program that leaks or live with it. The former piece of code helps to fail if there are leaked FD. On Unix FD leaks is not that problematic, but on Windows it turns to be another problem. The best example about this: you cannot delete a file that has an FD still open on it. This makes harder to remove temporary file (and this piece of code was precisely made to track FD on temporary files, that let 1000s of unremoved temp file). Regards, Sylvain Le Gall ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Optimizing garbage collection
The program I am testing it on is basically a DFS algorithm so it doesn't use much heap memory really. Just a lot of transient objects. Looking at the top the RSIZE looks to be not over 11M under OSX. Yes, the default minor heap size was indeed too low, I've been trying to set it to a higher value, now testing with the OCAMLRUNPARAM settings you recommended. It did result in some speedup, but not an awful lot, it's important to profile it as you say. Best, On Fri, Nov 19, 2010 at 4:54 PM, Michael Ekstrand mich...@elehack.netwrote: On 11/18/2010 09:51 AM, Eray Ozkural wrote: A program I wrote constructs a lot of small lists, and strings and discards them. It's a search algorithm. I profiled this code and saw that garbage collection takes significant time. In C++, we can write custom allocators to optimize the data structures that cause such slowdowns. Any recommended strategies in ocaml? The OCaml garbage collector exposes a variety of tuning parameters through the Gc module[1] and the OCAMLRUNPARAM environment variable[2]. I would tweak those. In particular, I would recommend increasing the minor heap size so that more of your data can be quickly discarded. You can also increase the space overhead, thereby causing the GC to be less aggressive at the expense of higher memory usage/more waste. Lastly, I often increase the heap increment to allow memory to allow the heap to expand more quickly, but I do not know if that will help in your case or not. I have documented my practices more thoroughly at my blog[3]. As I see it, the biggest gains will be by tuning your code and your minor heap size so that ephemeral structures never hit the major heap. My rule of thumb is that one work unit, if you have such a concept, should fit in the minor heap. Collecting dead structures from the minor heap is fast; moving a structure to the major heap only to have it be unreachable by the next GC cycle can cause substantial GC thrashing. You're on to a good start, though, by measuring. I use gprof heavily as I tweak my code's performance. - Michael 1. http://caml.inria.fr/pub/docs/manual-ocaml/libref/Gc.html 2. http://caml.inria.fr/pub/docs/manual-ocaml/manual024.html#toc96 3. http://elehack.net/michael/blog/2010/06/ocaml-memory-tuning ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs -- Eray Ozkural, PhD candidate. Comp. Sci. Dept., Bilkent University, Ankara http://groups.yahoo.com/group/ai-philosophy http://myspace.com/arizanesil http://myspace.com/malfunct ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Option functions (or lack thereof) + operator for composition
On 11/16/2010 05:49 AM, David Allsopp wrote: ExtLib is well-worth having and it's very easy to install (just run ocaml install.ml in its source tree - findlib recommended but I've got a feeling that it can install without findlib as well). It's also very stable, feature-wise, so you're not really adding a tricky dependency. It's now hosted on Google Code at http://code.google.com/p/ocaml-extlib/downloads/list You might also take a look at the batteries project (http://batteries.forge.ocamlcore.org) which I believe includes most if not all of ExtLib. Yes, Batteries includes pretty much all of Extlib (I'm not aware of any missing pieces), and specifically includes the Option module. I would recommend writing any new OCaml code with something like Batteries (or Jane St.'s Core, if you prefer) if at all possible; it fills in a lot of gaps in the standard library. - Michael ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
