Hi Kellen, Great point. Yes this would be a nice incremental step with BigTranslate and easily supportable. In Apache OODT we have a resource manager, with a BatchStub (worker) that runs on each node. That stub, can easily talk to e.g., a loaded Joshua server (via Tika Translate) with model already loaded, and really pump through the data. A few config options to set and we can easily support this.
Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Director, Information Retrieval and Data Science Group (IRDS) Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA WWW: http://irds.usc.edu/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ On 7/19/16, 4:12 AM, "kellen sunderland" <[email protected]> wrote: >Hey Chris, > >I'm also interested in batching jobs with Joshua and had a few questions >for you. I'm a little curious about how the http calls would work in a >clustered environment. Would they simply call a server running the Joshua >http service? Would waiting on the translations then become a bottleneck? > >I was wondering if you've considered the option of just loading the entire >model on each worker and handling the translate task as a map call? > >-Kellen > >On Tue, Jul 19, 2016 at 7:02 AM, Matt Post <[email protected]> wrote: > >> Yes — after the first week of August. This would be useful to factor into >> discussions about pulling the server out of the main code. >> >> >> > On Jul 15, 2016, at 1:11 AM, Mattmann, Chris A (3980) < >> [email protected]> wrote: >> > >> > Hey Matt, >> > >> > I’d love some help. Yes I would like to add a connection via >> > Tika Translate to Joshua - probably via the REST server. >> > Wanna help? >> > >> > Cheers, >> > Chris >> > >> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> > Chris Mattmann, Ph.D. >> > Chief Architect >> > Instrument Software and Science Data Systems Section (398) >> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> > Office: 168-519, Mailstop: 168-527 >> > Email: [email protected] >> > WWW: http://sunset.usc.edu/~mattmann/ >> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> > Director, Information Retrieval and Data Science Group (IRDS) >> > Adjunct Associate Professor, Computer Science Department >> > University of Southern California, Los Angeles, CA 90089 USA >> > WWW: http://irds.usc.edu/ >> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On 7/13/16, 7:49 AM, "Matt Post" <[email protected]> wrote: >> > >> >> Chris, >> >> >> >> This looks cool. How are you planning to get this to work with Joshua? >> Do you need help with the API piece? >> >> >> >> matt >> >> >> >> >> >>> On Jul 12, 2016, at 6:40 PM, Mattmann, Chris A (3980) < >> [email protected]> wrote: >> >>> >> >>> I will see about registering as well :) >> >>> >> >>> I have BigTranslate up and working if anyone is interested. I am >> >>> currently evaluating it on the XDATA employment corpus with Lingo24 >> >>> but next is Joshua (and hoping to use Bing Translate too). If anyone >> >>> has an Amazon unlimited key for translation to send my way would >> >>> love to add it to the mix too :) >> >>> >> >>> http://github.com/chrismattmann/bigtranslate/ >> >>> >> >>> Cheers, >> >>> Chris >> >>> >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> Chris Mattmann, Ph.D. >> >>> Chief Architect >> >>> Instrument Software and Science Data Systems Section (398) >> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> >>> Office: 168-519, Mailstop: 168-527 >> >>> Email: [email protected] >> >>> WWW: http://sunset.usc.edu/~mattmann/ >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> Director, Information Retrieval and Data Science Group (IRDS) >> >>> Adjunct Associate Professor, Computer Science Department >> >>> University of Southern California, Los Angeles, CA 90089 USA >> >>> WWW: http://irds.usc.edu/ >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On 7/12/16, 5:12 PM, "kellen sunderland" <[email protected]> >> wrote: >> >>> >> >>>> Thanks for forwarding Matt. I think a fair number of people from my >> team >> >>>> will want to attend. I'll pass around the registration link. >> >>>> >> >>>> -Kellen >> >>>> On Jul 12, 2016 11:01 PM, "Matt Post" <[email protected]> wrote: >> >>>> >> >>>>> Hi everyone, >> >>>>> >> >>>>> We had talked a while ago about Joshua projects for MT Marathon in >> Prague. >> >>>>> Registration (free) is now open. Let me know if you're planning to >> go and >> >>>>> we can make some plans! >> >>>>> >> >>>>> http://ufal.mff.cuni.cz/mtm16/registration >> >>>>> >> >>>>> matt >> >>>>> >> >>>>> >> >> >> >>
