That's awesome Pablo! Will definitely take this for a spin. Thanks for sharing! Niklas
On 14 May 2015 at 18:45, Pablo Cingolani <[email protected]> wrote: > > Hi Everyone, > I've been working on a simple programming language to create large > data pipelines on Mesos. The language is called BDS which stands > for BigDataScript (yes, the name is kind of a joke for all jargon-lovers > out there) and here is the web page: > > http://pcingola.github.io/BigDataScript/ > > Needles to say, it's open source and the code is available is GitHub. > At the moment I'm using BDS mostly for analysis of large genetic datasets > on our 25,000 core cluster, but it should scale to large(er) clusters as > well. > > BDS has a few interesting features: > - Runs on Mesos (obviously) as well as SunGridEngine, Torque, > MOAB, a large server or just your laptop. > > - You can develop on your laptop (without having to install Mesos or > any cluster management system) and then deploy your script to a > Mesos > cluster/datacenter without modification. > > - It performs automatic task dependency and schedules tasks according > to > the implicit (or explicit) DAG. > > - It has lazy processing. Checks whether performing a task is > necessary and > skips tasks whose output does not need to be updated (make-style). > > - It performs automatic checkpointing and has absolute serialization, > so you > can copy the checkpoint file to another computer and continue > running > exactly where you left. > > - It can handle several parallel pipeline branches (threads). > > - Allows to define DAGs in a declarative form (using 'goals'). > > - Cleans up stale files (and queues tasks in non-Mesos cluster). > > Other cool features: > > - Automatically parses command line options in your scripts (it also > creates "help" for you) > - Logs all processes's stdout / stderr and exit status > - It has a built in debugger > - It has a built in unity testing framework > > You can read more about all these features here: > > http://pcingola.github.io/BigDataScript/bigDataScript_manual.html > > I hope you find it useful and please do send me any > feedback you have. > Yours > > Pablo > > > >
