THREAD_LOCAL - operators share thread CONTAINER_LOCAL - each operator has its own thread
So as long as operators utilize the CPU sufficiently (compete), the latter will perform better. There will be cases where a single thread can accommodate multiple operators. For example, a socket reader (mostly waiting for IO) and a decompress (CPU hungry) can share a thread. But to get back to the original question, stream locality does generally not reduce the total memory requirement. If you add multiple operators into one container, that container will also require more memory and that's how the container size is calculated in the physical plan. You may get some extra mileage when multiple operators share the same heap but the need to identify the memory requirement per operator does not go away. Thomas On Sat, Sep 26, 2015 at 12:41 PM, Munagala Ramanath <[email protected]> wrote: > Would CONTAINER_LOCAL achieve the same thing and perform a little better on > a multi-core box ? > > Ram > > On Sat, Sep 26, 2015 at 12:18 PM, Sandeep Deshmukh < > [email protected]> > wrote: > > > Yes, with this approach only two containers are required: one for stram > and > > another for all operators. You can easily fit around 10 operators in less > > than 1GB. > > On 27 Sep 2015 00:32, "Timothy Farkas" <[email protected]> wrote: > > > > > Hi Ram, > > > > > > You could make all the operators thread local. This cuts down on the > > > overhead of separate containers and maximizes the memory available to > > each > > > operator. > > > > > > Tim > > > > > > On Sat, Sep 26, 2015 at 10:07 AM, Munagala Ramanath < > [email protected] > > > > > > wrote: > > > > > > > Hi, > > > > > > > > I was running into memory issues when deploying my app on the > sandbox > > > > where all the operators were stuck forever in the PENDING state > because > > > > they were being continually aborted and restarted because of the > > limited > > > > memory on the sandbox. After some experimentation, I found that the > > > > following config values seem to work: > > > > ------------------------------------------ > > > > < > https://datatorrent.slack.com/archives/engineering/p1443263607000010> > > > > > > > > > > > > > > > > *<property> <name>dt.attr.MASTER_MEMORY_MB</name> > > > <value>500</value> > > > > </property> <property> <name>dt.application..operator.* > > > > > > > > > > > > > > > > > > > > > > > > *.attr.MEMORY_MB</name> <value>200</value> </property> > <property> > > > > > > > > > > > > > > <name>dt.application.TopNWordsWithQueries.operator.fileWordCount.attr.MEMORY_MB</name> > > > > <value>512</value> </property>* > > > > ------------------------------------------------ > > > > Are these reasonable values ? Is there a more systematic way of > coming > > up > > > > with these values than trial-and-error ? Most of my operators -- with > > the > > > > exception of fileWordCount -- need very little memory; is there a way > > to > > > > cut all values down to the bare minimum and maximize available memory > > for > > > > this one operator ? > > > > > > > > Thanks. > > > > > > > > Ram > > > > > > > > > >
