Chapel community, Wanted to announce that a new Mesos Framework for the Chapel programming language implemented in C++ has been released. Mesosphere currently advertises a Chapel framework written in the Go language - the C++ framework has some design differences that might be of interest. You can get the source code here:
https://github.com/ct-clmsn/mesos4chpl Quick description of both systems and how they are different... Quick Mesos Background - Mesos is a "cloud operating system". An alternative to SLURM and some of the more traditional HPC job scheduling systems. It attempts to host multiple types of applications on a cluster (websites, hadoop/spark jobs, MPI, and chapel programs). Mesos frameworks consist of two programs, one that schedules tasks for a job and another program that executes a task on each of the scheduled nodes. Each Mesos node can host multiple tasks. Each compute task is "sandboxed"...it's an interesting system. The Go framework or Chapel is a little complicated and targets an older Chapel release. Deploying programs with the Go framework requires two programs to be compressed and copied to a follower node (using a curl accessible URI). The leader is expected to run another compiled Go program that waits for all the followers to contact it before running the "wrapper" program compiled by Chapel. The compressed file copied to each follower contains the *_real output of the Chapel compiler and then a compiled Go program that "phones back" to the leader. After every follower has called the leader (or phoned home), the leader transmits an execution command to each follower using a json formatted string. The command includes all of the environment variables currently in the user's shell - I've had some issues with how Go formats those json strings and it was a real block toward getting Chapel programs to run on a Mesos cluster. I suspect this as done to make sure each Chapel job was successfully provisioned by the Mesos orchestrator before running the Chapel job. Last thing to note, to run a Chapel program with the Go framework, you have to run the compiled Go framework program - the Go framework does not use GASNet's customized job spawning feature. The C++ framework is expected to work with the latest Chapel release and it was designed to use GASNet's customized job spawning feature. The framework does not require deploying multiple programs to each follower node - that said, if a provisioned task fails prematurely the whole Chapel program will terminate (this ideally is a rare edge case). In the C++ framework, none of the follower nodes "phone home" pior to execution - the followers pull the *_real program using a curl friendly URI that is provided to each follower by the Mesos orchestrator. The C++ framework might be a little easier to use for job deployment on a Mesos cluster. To run different programs you will need to modify the $ GASNET_CSPAWN_CMD environment varible. Both the Go and C++ frameworks use a greedy offer scheduling strategy - this is the typical approach used by all Mesos frameworks in the wild. I'm working on some C++ code to spice that aspect of scheduling up a bit (expect the C++ framework to reflect these options soon). Cheers! -- ct
------------------------------------------------------------------------------
_______________________________________________ Chapel-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-users
