Re: Multithreading ThreadJobFactory

2015-10-28 Thread Yi Pan
Hi, Lukas, Sorry to reply late. My comments below On Tue, Oct 20, 2015 at 7:14 AM, Lukas Steiblys wrote: > > >> I have been thinking lately about the most non-invasive way to add > >> multithreading capabilities to ThreadJobFactory, as that is the main > >> method > >> we run our jobs in produc

Re: Multithreading ThreadJobFactory

2015-10-20 Thread Lukas Steiblys
Kartik, This is for the case when you don't use YARN. ThreadJob runs locally and simply spins up a single thread for all tasks right now. Lukas On 10/20/15, Kartik Paramasivam wrote: > We have been wanting to do something similar at LinkedIn. We however > haven't thought through the details. >

Re: Multithreading ThreadJobFactory

2015-10-20 Thread Kartik Paramasivam
We have been wanting to do something similar at LinkedIn. We however haven't thought through the details. if container == thread.. then we would need to change the AppMaster to request the appropriate number of Yarn 'containers' (processes) .. i.e. we would have to decouple the process count from