Hello Here is the latest OCaml Weekly News, for the week of June 08 to 15, 2021.
Table of Contents ───────────────── Dependency visualization tool Multicore OCaml: May 2021 First release of Popper Orsetto: structured data interchange languages (release 1.1) Old CWN Dependency visualization tool ═════════════════════════════ Archive: <https://discuss.ocaml.org/t/dependency-visualization-tool/7964/2> Chet Murthy announced ───────────────────── In the past, I've found doing this sort of dependency-graphing to be very useful for figuring out where modularization in a large system needs to be applied. And it's not very hard to do, based on ocamldep output or other dependency information that already exists. On a lark, I wrote something that did this for installed findlib packages: <https://github.com/chetmurthy/not-ocamlfind#package-graph> I mention this not to plug the tool, but rather, just to note that it's really easy, and yeah, people should avail themselves of this more. Multicore OCaml: May 2021 ═════════════════════════ Archive: <https://discuss.ocaml.org/t/multicore-ocaml-may-2021/7990/1> Anil Madhavapeddy announced ─────────────────────────── Welcome to the May 2021 [Multicore OCaml] monthly report! This month's update along with the [previous updates] have been compiled by @avsm, @ctk21, @kayceesrk and @shakthimaan. Firstly, all of our upstream activity on the OCaml compiler is now reported as part of the shiny new [compiler development newsletter #2] that @gasche has started. This represents a small but important shift – domains-only multicore is firmly locked in on the upstream roadmap for OCaml 5.0 and the whole OCaml compiler team has been helping and contributing to it, with the [GC safe points] feature being one of the last major multicore-prerequisites (and due to be in OCaml 4.13 soon). This multicore newsletter will now focus on getting our ecosystem ready for domains-only multicore in OCaml 5.0, and on how the (not-yet-official) effect system and multicore IO stack is progressing. It's a long one this month, so settle in with your favourite beverage and let's begin :-) [Multicore OCaml] <https://github.com/ocaml-multicore/ocaml-multicore> [previous updates] <https://discuss.ocaml.org/tag/multicore-monthly> [compiler development newsletter #2] <https://discuss.ocaml.org/t/ocaml-compiler-development-newsletter-issue-2-may-2021/7965> [GC safe points] <https://github.com/ocaml/ocaml/pull/10039> OCaml Multicore: 4.12.0+domains ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ The multicore compiler now supports CTF runtime traces of its garbage collector and there are [tools to display chrome tracing visualisations] of the garbage collector events. A number of performance improvements (see speedup graphs later on) that highlight some ways to make best use of multicore were made to the existing benchmarks in Sandmark. There has also been work on scaling up to 128 cores/domains for task-based parallelism in domainslib using [work stealing deques], bringing us closer to Cilk-style task-parallel performance. As important as new features are what we have decided _not_ to do. We've been working on and evaluating Domain Local Allocation Buffers (DLABs) for some time, with the intention of reducing the cost of minor GCs. We've found that the resulting performance didn't match our expectations (_vs_ the complexity of the change), and so we've decided not to proceed with this for OCaml 5.0. You can find the [DLAB summary] page summarises our experiences. We'll come back to this post-OCaml 5.0 when there are fewer moving parts. [tools to display chrome tracing visualisations] <https://github.com/ocaml-multicore/eventlog-tools/tree/multicore_wip> [work stealing deques] <https://github.com/ocaml-multicore/domainslib/pull/29> [DLAB summary] <https://github.com/ocaml-multicore/ocaml-multicore/wiki/Domain-Local-Allocation-Buffers-Addendum> Ecosystem changes to prepare for 5.0.0 domains-only ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ As we are preparing 5.0 branches with the multicore branches over the coming months, we are stepping up preparations to ensure the OCaml ecosystem is ready. Making the multicore compilers available by default in opam-repo ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ Over the next few week, we will be merging the multicore 4.12.0+domains and associated packages from their opam remote over in [ocaml-multicore/multicore-opam] into the mainline opam-repository. This is to make it more convenient to use the variant compilers to start testing your own packages with ~Domain~s. As part of this change, there are two new base packages that will be available in opam-repository: • `base-domains': This package indicates that the current compiler has the `Domain' module. • `base-effects': This package indicates the current compiler has the experimental effect system. By adding a dependency on these packages, the only valid solutions will be `4.12.0+domains' (until OCaml 5.0 which will have this module) or `4.12.0+effects'. The goal of this is to let community packages more easily release versions of their code using Domains-only parallelism ahead of OCaml 5.0, so that we can start migration and thread-safety early. We do not encourage anyone to take a dependency on base-effects currently, as it is very much a moving target. This opam-repository change isn't in yet, but I'll comment on this post when it is merged. [ocaml-multicore/multicore-opam] <https://github.com/ocaml-multicore/multicore-opam> Adapting the Stdlib for thread-safety ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ One of the first things we have to do before porting third-party libraries is to get the Stdlib ready for thread-safety. This isn't quite as simple as it might appear at first glance: if we adopt the naïve approach of simply putting a mutex around every bit of global state, our sequential performance will slow down. Therefore we are performing a more fine-grained analysis and fixes, which can be seen [on the multicore stdlib page]. For anyone wishing to contribute: hunt through the Stdlib for global state, and categorise it appropriately, and then create a test case exercising that module with multiple Domains running, and submit a PR to [ocaml-multicore]. In general, if you see any build failures or runtime failures now, we'd really appreciate an issue being filed there too. You can see some good examples of such issues [here] (for mirage-crypto) and [here] (for Coqt). [on the multicore stdlib page] <https://github.com/ocaml-multicore/ocaml-multicore/wiki/Safety-of-Stdlib-under-Multicore-OCaml> [ocaml-multicore] <https://github.com/ocaml-multicore/ocaml-multicore> [here] <https://github.com/ocaml-multicore/ocaml-multicore/issues/574> [here] <https://github.com/ocaml-multicore/ocaml-multicore/issues/568> Porting third-party libraries to Domains ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ As I mentioned last month, we put a call out for libraries and maintainers who wanted to port their code over. We're starting with the following libraries and applications this month: • *Lwt*: the famous lightweight-threads library now has a PR to add [Lwt_domains]. This is the first simple(ish) step to using multicore cores with Lwt: it lets you run a pure (non-Lwt) function in another Domain via `detach : ('a -> 'b) -> 'a -> 'b Lwt.t'. • *Mirage-Crypto*: the next library we are adapting is the cryptography library, since it is also low-hanging fruit that should be easy to parallelise (since crypto functions do not have much global state). The port is still ongoing, as there are some minor build failures and also Stdlib functions in Format that aren't yet thread-safe that are [causing failures]. • *Tezos-Node*: the bigger application we are applying some of the previous dependencies too is Tezos-Node, which makes use of the dependency chain here via Lwt, mirage-crypto, Irmin, Cohttp and many other libraries. We've got this [compiling under 4.12.0+domains] now and mostly passing the test suite, but will only report significant results once the dependencies and Stdlib are passing. • *Owl*: OCaml's favourite machine learning library works surprisingly well out-of-the-box with 4.12.0+domains. An experiment for a significant machine-learning codebase written using it saw about a 2-4x speedup before some false-sharing bottlenecks kicked in. This is pretty good going given that we made no changes to the codebase itself, but stay tuned for more improvements over the coming months as we analyse the bottleneck. This is hopefully a signal to all of you to start "having a go" with 4.12.0+domains on your own applications, and particularly with respect to seeing how wrapping it in Domains works out and identifying global state. You can read our handy [tutorial on parallel programming with Multicore OCaml]. We are developing some tools to help find global state, but we're going to all need to work together to identify some of these cases and begin migration. Crucially, we need some diversity in our dependency chains – if you have interesting applications using (e.g.) Async or the vanilla `Thread' module and have some cycles to work with us, please get in touch with me or @kayceesrk . [Lwt_domains] <https://github.com/ocsigen/lwt/pull/860> [causing failures] <https://github.com/ocaml-multicore/ocaml-multicore/issues/563> [compiling under 4.12.0+domains] <https://gitlab.com/tezos/tezos/-/merge_requests/2671> [tutorial on parallel programming with Multicore OCaml] <https://github.com/ocaml-multicore/parallel-programming-in-multicore-ocaml> 4.12.0+effects ╌╌╌╌╌╌╌╌╌╌╌╌╌╌ The effects-based [eio library] is coming together nicely, and the interface and design rationales are all up-to-date in the README of the repository. The primary IO backend is [ocaml-uring], which we are preparing for a separate release to opam-repository now as it also works fine on the sequential runtime for Linux (as long as you have a fairly recent kernel. Otherwise the kernel crashes). We also have a [Grand Central Dispatch effect backend] to give us a totally different execution model to exercise our effect handler abstractions. While we won't publish the performance numbers for the effect-based IO this month, you can get a sense of the sorts of tests we are running by looking at the [retro-httpaf-bench] repository, which now has various permutations of effects-based, uring-based and select-based webservers. We've submitted a talk to the upcoming OCaml Workshop later this summer, which, if accepted, will give you a deepdive into our effect-based IO. As always, we begin with the Multicore OCaml ongoing and completed tasks. The ecosystem improvements are then listed followed by the updates to the Sandmark benchmarking project. Finally, the upstream OCaml work is mentioned for your reference. For those of you that have read this far and can think of nothing more fun than hacking on multicore programming runtimes, we are hiring in the UK, France and India – please find the job postings at the end! [eio library] <https://github.com/ocaml-multicore/eio> [ocaml-uring] <https://github.com/ocaml-multicore/ocaml-uring> [Grand Central Dispatch effect backend] <https://github.com/ocaml-multicore/eio/pull/26> [retro-httpaf-bench] <https://github.com/ocaml-multicore/retro-httpaf-bench> Multicore OCaml ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ Ongoing ┄┄┄┄┄┄┄ • [ocaml-multicore/ocaml-multicore#552] Add a force_instrumented_runtime option to configure A new `--enable-force-instrumented-runtime' option is introduced to facilitate use of the instrumented runtime on linker invocations to obtain event logs. • [ocaml-multicore/ocaml-multicore#553] Testsuite failures with flambda enabled A list of tests are failing on `b23a416' with flambda enabled, and they need to be investigated further. • [ocaml-multicore/ocaml-multicore#555] runtime: CAML_TRACE_VERSION is now set to a Multicore specific value Define a `CAML_TRACE_VERSION' to distinguish between Multicore OCaml and trunk for the runtime. • [ocaml-multicore/ocaml-multicore#558] Refactor Domain.{spawn/join} to use no critical sections The PR removes the use of `Domain.wait' and critical sections in `Domain.{spawn/join}'. • [ocaml-multicore/ocaml-multicore#559] Improve the Multicore GC Stats A draft PR to include more Multicore GC statistics when using `OCAMLRUNPARAM=v=0x400'. [ocaml-multicore/ocaml-multicore#552] <https://github.com/ocaml-multicore/ocaml-multicore/pull/552> [ocaml-multicore/ocaml-multicore#553] <https://github.com/ocaml-multicore/ocaml-multicore/issues/553> [ocaml-multicore/ocaml-multicore#555] <https://github.com/ocaml-multicore/ocaml-multicore/pull/555> [ocaml-multicore/ocaml-multicore#558] <https://github.com/ocaml-multicore/ocaml-multicore/pull/558> [ocaml-multicore/ocaml-multicore#559] <https://github.com/ocaml-multicore/ocaml-multicore/pull/559> Completed ┄┄┄┄┄┄┄┄┄ • [ocaml-multicore/ocaml-multicore#508] Domain Local Allocation Buffers The Domain Local Allocation Buffer implementation for OCaml Multicore has been dropped for now. A discussion is on the PR itself and there is a wiki page [here]. • [ocaml-multicore/ocaml-multicore#527] Port eventlog to CTF The porting of the `eventlog' implementation to the Common Trace Format is now complete. For an introduction to producing Chrome trace visualizations of the runtime events see [eventlog-tools]. This postprocessing tool turns the CTF trace into the Chrome tracing format that allows interactive visualizations like this: <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/optimized/2X/7/7984354f12695102e97bd6a37202ea8da9a6decf_2_1380x950.jpeg> • [ocaml-multicore/ocaml-multicore#543] Parallel version of weaklifetime test A parallel version of the `weaklifetime.ml' test is now added to the test suite. • [ocaml-multicore/ocaml-multicore#546] Coverage of domain life-cycle in domain_dls and ephetest_par tests Additional tests to increase test coverage for domain life-cycle for `domain_dls.ml' and `ephetest_par.ml'. • [ocaml-multicore/ocaml-multicore$#550] Lazy effects test Inclusion of a test to address effects with Lazy computations for a number of different use cases. • [ocaml-multicore/ocaml-multicore#557] Remove unused domain functions A clean-up to remove unused functions in `domain.c' and `domain.h'. [ocaml-multicore/ocaml-multicore#508] <https://github.com/ocaml-multicore/ocaml-multicore/pull/508> [here] <https://github.com/ocaml-multicore/ocaml-multicore/wiki/Domain-Local-Allocation-Buffers-Addendum> [ocaml-multicore/ocaml-multicore#527] <https://github.com/ocaml-multicore/ocaml-multicore/pull/527> [eventlog-tools] <https://github.com/ocaml-multicore/eventlog-tools/tree/multicore_wip> [ocaml-multicore/ocaml-multicore#543] <https://github.com/ocaml-multicore/ocaml-multicore/pull/543> [ocaml-multicore/ocaml-multicore#546] <https://github.com/ocaml-multicore/ocaml-multicore/pull/546> [ocaml-multicore/ocaml-multicore$#550] <https://github.com/ocaml-multicore/ocaml-multicore/pull/550> [ocaml-multicore/ocaml-multicore#557] <https://github.com/ocaml-multicore/ocaml-multicore/pull/557> Ecosystem ╌╌╌╌╌╌╌╌╌ Ongoing ┄┄┄┄┄┄┄ • [ocaml-multicore/eventlog-tools#2] Add a pausetimes tool The `eventlog_pausetimes' tool takes a directory of eventlog files and computes the mean, max pause times, as well as the distribution up to the 99.9th percentiles. For example: ┌──── │ ocaml-eventlog-pausetimes /home/engil/dev/ocaml-multicore/trace3/caml-426094-* name │ { │ "name": "name", │ "mean_latency": 718617, │ "max_latency": 33839379, │ "distr_latency": [191,250,707,16886,55829,105386,249272,552640,1325621,13312993,26227671] │ } └──── • [domainslib#29] Task stealing with CL deques This ongoing work to use task-stealing Chase Lev deques for scheduling tasks across domains is looking very promising. Particularly for machines with 128 cores. * [ocaml-multicore/retro-httpaf-bench#10] Add Eio benchmark The addition of an Eio benchmark for retro-httpaf-bench. This is a work-in-progress. • [ocaml-multicore/eio#26] Grand Central Dispatch Backend An early draft PR that implements the Grand Central Dispatch (GCD) backend for Eio. • [ocsigen/lwt#860] Lwt_domain: An interfacet to Multicore parallelism An on-going effort to introduce `Lwt_domain' for performing computations to CPU cores using Multicore OCaml's Domains. [ocaml-multicore/eventlog-tools#2] <https://github.com/ocaml-multicore/eventlog-tools/pull/2> [domainslib#29] <https://github.com/ocaml-multicore/domainslib/pull/29> [ocaml-multicore/retro-httpaf-bench#10] <https://github.com/ocaml-multicore/retro-httpaf-bench/pull/10> [ocaml-multicore/eio#26] <https://github.com/ocaml-multicore/eio/pull/26> [ocsigen/lwt#860] <https://github.com/ocsigen/lwt/pull/860> Completed ┄┄┄┄┄┄┄┄┄ retro-httpaf-bench ┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ The `retro-httpaf-bench' repository contains scripts for running HTTP server benchmarks. • [ocaml-multicore/retro-httpaf-bench#6] Move OCaml to 4.12 The build scripts have been updated to use 4.12.0. • [ocaml-multicore/retro-httpaf-bench#8] Adds a Rust benchmark using hyper The inclusion of the Hyper benchmark limited to a single core to match the other existing benchmarks. • [ocaml-multicore/retro-httpaf-bench#9] Release builds for dune, stretch request volumes, rust fixes and remove mimalloc The Dockerfile, README, build_benchmarks.sh and run_benchmarks.sh files have been updated. • [ocaml-multicore/retro-httpaf-bench#15] Make benchmark more realistic The PR enhances the implementation to correctly simulate a hypothetical database request, and the effects code has been updated accordingly. [ocaml-multicore/retro-httpaf-bench#6] <https://github.com/ocaml-multicore/retro-httpaf-bench/pull/6> [ocaml-multicore/retro-httpaf-bench#8] <https://github.com/ocaml-multicore/retro-httpaf-bench/pull/8> [ocaml-multicore/retro-httpaf-bench#9] <https://github.com/ocaml-multicore/retro-httpaf-bench/pull/9> [ocaml-multicore/retro-httpaf-bench#15] <https://github.com/ocaml-multicore/retro-httpaf-bench/pull/5> eio ┈┈┈ The `eio' library provides an effects-based parallel IO stack for Multicore OCaml. • [ocaml-multicore/eio#18] Add fibreslib library The `promise' library has been renamed to `fibreslib' to avoid naming conflict with the existing package in opam, and the API (waiters and effects) has been split into its own respective modules. • [ocaml-multicore/eio#19] Update to latest ocaml-uring The code and configuration files have been updated to use the latest `ocaml-uring'. • [ocaml-multicore/eio#20] Add Fibreslib.Semaphore Implemented the `Fibreslib.Semaphone' module that is useful for rate-limiting, and based on OCaml's `Semaphore.Counting'. • [ocaml-multicore/eio#21] Add high-level Eio API A new Eio library with interfaces for sources and sinks. The README documentation has been updated with motivation and usage. • [ocaml-multicore/eio#22] Add switches for structured concurrency Implementation of structured concurrency with documentation examples for tracing and testing with mocks. • [ocaml-multicore/eio#23] Rename repository to eio The Effects based parallel IO for OCaml repository has now been renamed from `eioio' to `eio'. • [ocaml-multicore/eio#24] Rename lib_eioio to lib_eunix The names have been updated to match the dune file. • [ocaml-multicore/eio#25] Detect deadlocks An exception is now raised to detect deadlocks if the scheduler finishes while the main thread continues to run. • [ocaml-multicore/eio#27] Convert expect tests to MDX The expected tests have been updated to use the MDX format, and this avoids the need for ppx libraries. • [ocaml-multicore/eio#28] Use splice to copy if possible The effect Splice has been implemented along with the update to ocaml-uring, and necessary documentation. • [ocaml-multicore/eio#29] Improve exception handling in switches Additional exception checks to handle when multiple threads fail, and for `Switch.check' and `Fibre.fork_ignore'. • [ocaml-multicore/eio#30] Add eio_main library to select backend automatically Use `eio_main' to select the appropriate backend (`eunix', for example) based on the platform. • [ocaml-multicore/eio#31] Add Eio.Flow API Implemented a Flow module that allows combinations such as bidirectional flows and closable flows. • [ocaml-multicore/eio#32] Initial support for networks Eio provides a high-level API for networking, and the `Network' module has been added. • [ocaml-multicore/eio#33] Add some design rationale notes to the README The README has been updated with design notes, and reference to further reading on the principles of Object-capability model. • [ocaml-multicore/eio#34] Add shutdown, allow closing listening sockets, add cstruct_source Added cstruct_source, `shutdown' method along with source, sink and file descriptor types. • [ocaml-multicore/eio#35] Add Switch.on_release to auto-close FDs We can now attach resources such as file descriptors to switches, and these are freed when the the switch is finished. [ocaml-multicore/eio#18] <https://github.com/ocaml-multicore/eio/pull/18> [ocaml-multicore/eio#19] <https://github.com/ocaml-multicore/eio/pull/19> [ocaml-multicore/eio#20] <https://github.com/ocaml-multicore/eio/pull/20> [ocaml-multicore/eio#21] <https://github.com/ocaml-multicore/eio/pull/21> [ocaml-multicore/eio#22] <https://github.com/ocaml-multicore/eio/pull/22> [ocaml-multicore/eio#23] <https://github.com/ocaml-multicore/eio/pull/23> [ocaml-multicore/eio#24] <https://github.com/ocaml-multicore/eio/pull/24> [ocaml-multicore/eio#25] <https://github.com/ocaml-multicore/eio/pull/25> [ocaml-multicore/eio#27] <https://github.com/ocaml-multicore/eio/pull/27> [ocaml-multicore/eio#28] <https://github.com/ocaml-multicore/eio/pull/28> [ocaml-multicore/eio#29] <https://github.com/ocaml-multicore/eio/pull/29> [ocaml-multicore/eio#30] <https://github.com/ocaml-multicore/eio/pull/30> [ocaml-multicore/eio#31] <https://github.com/ocaml-multicore/eio/pull/31> [ocaml-multicore/eio#32] <https://github.com/ocaml-multicore/eio/pull/32> [ocaml-multicore/eio#33] <https://github.com/ocaml-multicore/eio/pull/33> [ocaml-multicore/eio#34] <https://github.com/ocaml-multicore/eio/pull/34> [ocaml-multicore/eio#35] <https://github.com/ocaml-multicore/eio/pull/35> Sundries ┈┈┈┈┈┈┈┈ • [ocaml-multicore/domainslib#23] Running tests: moving to `dune runtest' from manual commands in `run_test' target The `dune runtest' command is now used to execute the tests. • [ocaml-multicore/domainslib#24] Move to Mutex & Condition from Domain.Sync.{notify/wait} The channel implementation using `Mutex' and `Condition' is now complete. The performance results are shown in the following graph: <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/optimized/2X/c/c3525f387ece415069bbac1f76525cf2ca96ef1d_2_930x1000.png> • [ocaml-multicore/multicore-opam#53] Add base-domains and base-effects packages The `base-domains' and `base-effects' opam files have now been added to multicore-opam. • [ocaml-multicore/multicore-opam#54] Shift all multicore packages to unique versions and base-domains dependencies The naming convention is to now use `base-effects' and `base-domains' everywhere. [ocaml-multicore/domainslib#23] <https://github.com/ocaml-multicore/domainslib/issues/23> [ocaml-multicore/domainslib#24] <https://github.com/ocaml-multicore/domainslib/pull/24> [ocaml-multicore/multicore-opam#53] <https://github.com/ocaml-multicore/multicore-opam/pull/53> [ocaml-multicore/multicore-opam#54] <https://github.com/ocaml-multicore/multicore-opam/pull/54> Benchmarking ╌╌╌╌╌╌╌╌╌╌╌╌ Ongoing ┄┄┄┄┄┄┄ • [ocaml-bench/sandmark#230] Build for 4.13.0+trunk with dune.2.8.1 A work-in-progress to upgrade Sandmark to use dune.2.8.1 to build 4.13.0+trunk and generate the benchmarks. You can test the same using: ┌──── │ TAG='"macro_bench"' make run_config_filtered.json │ RUN_CONFIG_JSON=run_config_filtered.json make ocaml-versions/4.13.0+trunk.bench └──── [ocaml-bench/sandmark#230] <https://github.com/ocaml-bench/sandmark/pull/230> Completed ┄┄┄┄┄┄┄┄┄ Sandmark ┈┈┈┈┈┈┈┈ ◊ Performance • [ocaml-bench/sandmark#221] Fix up decompress iterations of work The use of `parallel_for', simplification of `data_to_compress' to use `String.init', and fix to correctly count the amount of work configured and done produces the following speed improvements: <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/4/49a4b0b3f2e8e7c89e23c91c5f5307cdaa160d5c.png> <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/4/49bfcda236b0a900565c73b37867a7d35565b266.png> • [ocaml-bench/sandmark#223] A better floyd warshall An improvement to the Floyd Warshall implementation that fixes the random seed so that it is repeatable, and improves the pattern matching. <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/4/4a89577052976a094a4813b194aca1ebfb316a73.png> <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/b/be902a7ddc891b318c42a0e625b3335fbfeba1d0.png> <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/c/cbc5f44ea2b594b7d644a1ced93407e545c0bdb1.png> • [ocaml-bench/sandmark#224] Some improvements for matrix multiplication The `matrix_multiplication' and `matrix_multiplication_multicore' code have been updated for easier maintenance, and results are written only after summing the values. <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/a/ac194a6ecd26bdb6d9e602378ed4b44eb1f67d8e.png> <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/4/4ecdd0c82cd59f50fffd9078cd85b5bbcf740e09.png> • [ocaml-bench/sandmark#225] Better Multicore EA Benchmark The Evolutionary Algorithm now inserts a poll point into `fittest' to improve the benchmark results. <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/6/613a8f51682c02da672d12232bd8b244c65ba6d7.png> <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/b/b135d1269f57af9df575ab51ef7c31583713b42c.png> • [ocaml-bench/sandmark#226] Better scaling for mandelbrot6_multicore The `mandelbrot6_multicore' scales well now with the use of `parallel_for' as observed in the following graphs: <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/3/3ae484f3942252583abc4d44cebc0051adec2b32.png> <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/b/b7dbb656928988548ae9668c0d1edfe5186479b2.png> <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/2/241f9360e8b3fee78b6f5d95ed40f7dcff24f813.png> • [ocaml-bench/sandmark#227] Improve nbody_multicore benchmark with high core counts The `energy' function is now parallelised with `parallel_for_reduce' for larger core counts. <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/d/d5bb284534e0f32e24c7d8937afa4cac022ef79c.png> <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/b/be718f3e5cc676f44c827dfa56bdfa01e376e447.png> • [ocaml-bench/sandmark#229] Improve game_of_life benchmarks The hot functions are now inlined to improve the `game_of_life' benchmarks, and we avoid initialising the temporary matrix with random numbers. <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/5/50c1b7e799a4c71ee4585409d1eba575bf0748ed.png> <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/e/eb7b75ddaa314fa80908f12758f974223eb70490.png> [ocaml-bench/sandmark#221] <https://github.com/ocaml-bench/sandmark/pull/221> [ocaml-bench/sandmark#223] <https://github.com/ocaml-bench/sandmark/pull/223> [ocaml-bench/sandmark#224] <https://github.com/ocaml-bench/sandmark/pull/224> [ocaml-bench/sandmark#225] <https://github.com/ocaml-bench/sandmark/pull/225> [ocaml-bench/sandmark#226] <https://github.com/ocaml-bench/sandmark/pull/226> [ocaml-bench/sandmark#227] <https://github.com/ocaml-bench/sandmark/pull/227> [ocaml-bench/sandmark#229] <https://github.com/ocaml-bench/sandmark/pull/229> ◊ Sundries • [ocaml-bench/sandmark#215] Remove Gc.promote_to from treiber_stack.ml The 4.12+domains and 4.12+domains+effects branches have `Gc.promote_to' removed from the runtime. • [ocaml-bench/sandmark#216] Add configs for 4.12.0+stock, 4.12.0+domains, 4.12.0+domains+effects The ocaml-version configuration files for 4.12.0+stock, 4.12.0+domains, and 4.12.0+domains+effects have now been included to Sandmark. • [ocaml-bench/sandmark#220] Attempt to improve the OCAMLRUNPARAM documentation The README has been updated with more documentation on the use of OCAMLRUNPARAM configuration when running the benchmarks. • [ocaml-bench/sandmark#222] Deprecate 4.06.1 and 4.10.0 and upgrade to 4.12.0 The 4.06.1, 4.10.0 ocaml-versions have been removed and the CI has been updated to use 4.12.0 as the default version. [ocaml-bench/sandmark#215] <https://github.com/ocaml-bench/sandmark/pull/215> [ocaml-bench/sandmark#216] <https://github.com/ocaml-bench/sandmark/pull/216> [ocaml-bench/sandmark#220] <https://github.com/ocaml-bench/sandmark/pull/220> [ocaml-bench/sandmark#222] <https://github.com/ocaml-bench/sandmark/pull/222> current-bench ┈┈┈┈┈┈┈┈┈┈┈┈┈ • [ocurrent/current-bench#103] Ability to set scale on UI to start at 0 The graph origins now start from `[0, y_max+delta]' for the y-axis for better comparison. ![current-bench frontend fix 0 baseline](images/Current-bench-PR-74.png) • [ocurrent/current-bench#121] Use string representation for docker cpu setting. The `OCAML_BENCH_DOCKER_CPU' setting now switches from Integer to String to support a range of CPUs for parallel execution. [ocurrent/current-bench#103] <https://github.com/ocurrent/current-bench/issues/103> [ocurrent/current-bench#121] <https://github.com/ocurrent/current-bench/pull/121> OCaml ╌╌╌╌╌ Ongoing ┄┄┄┄┄┄┄ • [ocaml/ocaml#10039] Safepoints The Sandmark benchmark runs to obtain the performance numbers for the Safepoints PR for 4.13.0+trunk have been published. The PR is ready to be merged. [ocaml/ocaml#10039] <https://github.com/ocaml/ocaml/pull/10039> Job Advertisements ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ • [Multicore OCaml Runtime Systems Engineer] OCaml Labs (UK), Tarides (France) and Segfault Systems (India) • [Benchmark Tooling Engineer] Tarides Our thanks to all the OCaml users, developers and contributors in the community for their continued support to the project. Stay safe! [Multicore OCaml Runtime Systems Engineer] <https://discuss.ocaml.org/t/runtime-systems-engineer-ocaml-labs-uk-tarides-fr-segfault-systems-in-remote/7959> [Benchmark Tooling Engineer] <https://tarides.com/jobs/benchmark-tooling-engineer> Acronyms ╌╌╌╌╌╌╌╌ • AMD: Advanced Micro Devices • API: Application Programming Interface • CI: Continuous Integration • CPU: Central Processing Unit • CTF: Common Trace Format • DLAB: Domain Local Allocation Buffer • EA: Evolutionary Algorithm • GC: Garbage Collector • GCD: Grand Central Dispatch • HTTP: Hypertext Transfer Protocol • OPAM: OCaml Package Manager • MVP: Minimal Viable Product • PR: Pull Request • TPS: Transactions Per Second • UI: User Interface First release of Popper ═══════════════════════ Archive: <https://discuss.ocaml.org/t/ann-first-release-of-popper/7992/1> Joel Bjornson announced ─────────────────────── I am happy to announce a first release of [Popper] — an OCaml testing library for writing simple unit tests as well as property-based tests, using a uniform API. A few things that distinguishes [Popper] from other property-based testing libraries: 1) Generic shrinking — when a property-based test fails, the test function attempts to shrink the sample data. This works for any data type. 2) Any invariants used when constructing samples for property-based tests are also respected when shrinking. 3) There is a [ppx] for automatically deriving sample functions (generators in QuickCheck style libs) for custom data types. The library also supports bundling and nesting tests arbitrarily, colorful output inspired by [Alcotest], logging facilities and timing info. Here are some starting points: • The [source repo]. • A [getting started guide]. • The [API docs]. Below is an example of what [Popper] output looks like. It was generated by [this code]. <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/8/86ab1380a1c443170656bc937a872c725cf6b78a.png> This is a first release and there are few work-in-progress items. For instance: • A CLI for filtering and sorting tests. • An API for better blending with [Lwt] code bases. • A better way for limiting shrinking times for failing test samples. Any suggestions or other feedback is much appreciated! [Popper] <https://github.com/jobjo/popper> [ppx] <https://opam.ocaml.org/packages/ppx_deriving_popper/> [Alcotest] <https://github.com/mirage/alcotest> [source repo] <https://github.com/jobjo/popper> [getting started guide] <https://jobjo.github.io/popper/getting_started/> [API docs] <https://jobjo.github.io/popper/api/> [this code] <https://gist.github.com/jobjo/32f1b7297a330dc16d7a7c9fb15008fe> [Lwt] <https://github.com/ocsigen/lwt> Orsetto: structured data interchange languages (release 1.1) ════════════════════════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/ann-orsetto-structured-data-interchange-languages-release-1-1/7995/1> james woodyatt announced ──────────────────────── Announcing the release to OPAM of [Orsetto 1.1], an update to a personal project of mine not sponsored by my employer. Licensed with BSD 2-Clause. [Orsetto 1.1] <http://opam.ocaml.org/packages/orsetto/> Q. What is Orsetto? ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ Aspires to do eventually for OCaml more or less what [Serde] has done for Rust, i.e. to be a curated and self-contained collection of structured data interchange languages with a cohesive and unified model of serialization and deserialization. Two interchange languages are currently supported: [JSON] and [CBOR]. [Serde] <https://serde.rs> [JSON] <https://json.org> [CBOR] <https://cbor.io> Q. What is new in this release? ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ The change log for the release is here: [CHANGES.md] • Drops support for OCaml < 4.08. • Major improvements in test coverage. • Many corrections for logic errors. • Issues [database] migrated to Atlassian Jira Cloud. • A few important usability improvements (see below). Some usability improvements are meant to replace obsolescent interfaces (now marked obsolescent with `@ocaml.deprecated'). A few improvements to some interfaces with especially suboptimal design were updated with breaking changes, but these are not expected to cause major upgrade problems for anyone. Some things have not changed: • Still requires *ocamlfind* and does not build with *dune*. • Still only supports JSON and CBOR. [CHANGES.md] <https://bitbucket.org/jhw/orsetto/src/r1.1/CHANGES.md> [database] <https://conjury.atlassian.net/browse/ORS> Q. It looks incomplete. What are your plans for future development? ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ Yes, it's a personal project. I'd welcome opportunities to collaborate with others who share my vision for the project. As long as it's just me working on this, development will be somewhat slow. I have a lot of projects. This is the only open source one. • *Orsetto 1.0.3* is the previous release. It offered parsers and emitter combinators for JSON and CBOR for OCaml >= 4.06.1 (including 4.13~alpha1). The quality of its JSON support is adequate, and it scores well on the [nst/JSONTestSuite] tests. The quality of its CBOR support is provisional, and not recommended. • *Orsetto 1.1* is the current release. It adds generalized and extensible structured data interchange models with specializations for producing emitters and parsers for JSON and CBOR. The quality of the CBOR support is much improved. • *Orsetto 1.2* is the next planned release. It will drop interfaces marked `@caml.deprecated' in the 1.1 release. It will also drop support for OCaml < 4.10, and I'd like to stop depending on *ocamlfind*. I hope to add a PPX for deriving parsers and emitters from OCaml data type definitions. I might also consider one or more new interchange languages— suggestions are heartily encouraged. [nst/JSONTestSuite] <https://github.com/nst/JSONTestSuite> Old CWN ═══════ If you happen to miss a CWN, you can [send me a message] and I'll mail it to you, or go take a look at [the archive] or the [RSS feed of the archives]. If you also wish to receive it every week by mail, you may subscribe [online]. [Alan Schmitt] [send me a message] <mailto:alan.schm...@polytechnique.org> [the archive] <https://alan.petitepomme.net/cwn/> [RSS feed of the archives] <https://alan.petitepomme.net/cwn/cwn.rss> [online] <http://lists.idyll.org/listinfo/caml-news-weekly/> [Alan Schmitt] <https://alan.petitepomme.net/>
_______________________________________________ caml-news-weekly mailing list caml-news-weekly@lists.idyll.org http://lists.idyll.org/listinfo/caml-news-weekly