Thanks for your interest. I think my situation is much simpler though: - splitting the task into parallel processes is straightforward (one process for each combination of the dataset and parameters) - data and code transfer between nodes is also straightforward since the file system is shared (NFS) - aggregating the result is also easy: a simple way is to save the result in a file, properly labeled, and aggregate these files at the end of the computation.
The constraint is that the computation has to be done under Matlab (is this possible with Hadoop? my guess is no). So the only 'difficulty' is to open matlab sessions on remote hosts, send them some instructions, detect when the job is complete, reassign a new job and so on. This seems very basic to me, I think I could do it in a week or so. I have some other things to do though, hence the job offer. Any volunteer. Even a relatively inexperienced programmer could do it. The only pre-requisite is to know a programming language that supports multi-threading and system calls, and to know a bit about multi-threading of course. There are some existing solutions that could be investigated first too. Regards Brice On Thu, Apr 15, 2010 at 10:07 AM, Kelvin Quee <[email protected]> wrote: >> There are a few out there, but most are proprietary AFAIK, for good reason >> cause google as patented it. > > Sylvain, what about - > > http://hadoop.apache.org/ > > Please do some research before spreading FUD. > > Kelvin Quee > +65 9177 3635 > > > > On Thu, Apr 15, 2010 at 10:02 AM, Sylvain Bougerel > <[email protected]> wrote: >> It seems to me a map/reduce implementation can solve your problem. >> >> There are a few out there, but most are proprietary AFAIK, for good reason >> cause google as patented it. >> >> So that can be a good point of departure to solve your problem. Althought >> depending on the flexibility required, it may take some time to write. >> >> Sylvain. >> >> On 14 Apr 2010 17:24, "Luther Goh Lu Feng" <[email protected]> wrote: >> >> Pardon the email formatting... apparently gmail and yahoo mail don't work >> very well together :( >> >> >> >> ----- Forwarded Message ---- >>> From: Brice Rebsamen <[email protected]> >>> To: linuxnus <[email protected]> >>> Sent: Wed, April 14, 2010 4:52:06 PM >>> Subject: [linuxNUS] Internship offer >>> >>> Hi >> >> I am a research scientist at Temasek Lab (eng. faculty), working >>> in >> cognitive science. I am looking for a programmer for a short job. >>> I >> haven't talked to the management yet if this could be paid, but there >> is >>> a good chance that some money can be awarded for it. In any case >> that would >>> be a valuable experience and it will definitely improve >> your resume. And who >>> knows, that might result in a longer contract >> (Research assistant for >>> instance). Here is the task to achieve. >> >> We have several linux >>> workstations with matlab installed. However we >> do not have the distributed >>> computing toolbox. So I want to create a >> workaround. Typically, I want to >>> test one function on several data >> sets with several parameters, and combine >>> the result to find the best >> combination of parameters. This is a very >>> parallel task by nature. I >> believe that it would not be too difficult to >>> write a script that >> would manage connections with all available workstations >>> and feed them >> with data. This script could be written in any language you >>> are >> familiar with. C might be a good choice because of Matlab's >>> C >> interface (search doc on the engopen function for more info). If >>> not >> using the C API, the program could rely on rsh to launch remote >>> matlab >> sessions. >> >> Don't hesitate to contact me for more info, or send >>> me your resume. >> Also, forward this mail to your friends who might be >>> interested. >> >> Regards >> Brice >>> Rebsamen >> >> >> ------------------------------------ >> >> Find us in IRC >>> on #linuxnus @ >>> href="http://irc.freenode.net";>irc.freenode.net: >>> http://linuxnus.org/chat >> >> Our website and wiki: http://linuxnus.org/ | >>> http://opensource.nus.edu.sg >> >> Follow us on twitter: >>> http://twitter.com/linuxnusYahoo! Groups Links >> >> <*> To visit your >>> group on the web, go to: >> >>> http://groups.yahoo.com/group/linuxNUS/ >> >> <*> Your email >>> settings: >> Individual Email | Traditional >> >> <*> To >>> change settings online go to: >> >>> http://groups.yahoo.com/group/linuxNUS/join >> (Yahoo! ID >>> required) >> >> <*> To change settings via email: >> >>> ymailto="mailto:[email protected]"; >>> >>> href="mailto:[email protected]";>[email protected] >>> >> >>> >>> href="mailto:[email protected]";>[email protected] >> >> <*> >>> To unsubscribe from this group, send an email to: >> >>> ymailto="mailto:[email protected]"; >>> >>> href="mailto:[email protected]";>[email protected] >> >> <*> >>> Your use of Yahoo! Groups is subject to: >> >>> http://docs.yahoo.com/info/terms/ >> >> >> >> >> _______________________________________________ >> LUGS Mailing list - [email protected] >> List FAQ: http://wiki.lugs.org.sg/LugsMailingListFaq >> Info page: http://www.lugs.org.sg/mailman/listinfo/slugnet >> To unsubscribe send an empty email to: [email protected] >> >> _______________________________________________ >> LUGS Mailing list - [email protected] >> List FAQ: http://wiki.lugs.org.sg/LugsMailingListFaq >> Info page: http://www.lugs.org.sg/mailman/listinfo/slugnet >> To unsubscribe send an empty email to: [email protected] >> >> > _______________________________________________ LUGS Mailing list - [email protected] List FAQ: http://wiki.lugs.org.sg/LugsMailingListFaq Info page: http://www.lugs.org.sg/mailman/listinfo/slugnet To unsubscribe send an empty email to: [email protected]
