Hi, Lukas,
Sorry to reply late. My comments below
On Tue, Oct 20, 2015 at 7:14 AM, Lukas Steiblys
wrote:
>
> >> I have been thinking lately about the most non-invasive way to add
> >> multithreading capabilities to ThreadJobFactory, as that is the main
> >> method
> >> we run our jobs in produc
Kartik,
This is for the case when you don't use YARN. ThreadJob runs locally
and simply spins up a single thread for all tasks right now.
Lukas
On 10/20/15, Kartik Paramasivam wrote:
> We have been wanting to do something similar at LinkedIn. We however
> haven't thought through the details.
>
We have been wanting to do something similar at LinkedIn. We however
haven't thought through the details.
if container == thread.. then we would need to change the AppMaster to
request the appropriate number of Yarn 'containers' (processes) .. i.e. we
would have to decouple the process count from
I have been thinking lately about the most non-invasive way to add
multithreading capabilities to ThreadJobFactory, as that is the main method we
run our jobs in production. Looking at the master branch code in Git, I have
found the following:
a.. The best way would be to simply spin up a new
