A good read - http://preshing.com/20111118/locks-arent-slow-lock-contention-is/
Though it does not explain order of magnitude difference. Amol On Sat, Sep 26, 2015 at 4:25 PM, Vlad Rozov <[email protected]> wrote: > In the benchmark test THREAD_LOCAL outperforms CONTAINER_LOCAL by an order > of magnitude and both operators compete for CPU. I'll take a closer look > why. > > Thank you, > > Vlad > > > On 9/26/15 14:52, Thomas Weise wrote: > >> THREAD_LOCAL - operators share thread >> CONTAINER_LOCAL - each operator has its own thread >> >> So as long as operators utilize the CPU sufficiently (compete), the latter >> will perform better. >> >> There will be cases where a single thread can accommodate multiple >> operators. For example, a socket reader (mostly waiting for IO) and a >> decompress (CPU hungry) can share a thread. >> >> But to get back to the original question, stream locality does generally >> not reduce the total memory requirement. If you add multiple operators >> into >> one container, that container will also require more memory and that's how >> the container size is calculated in the physical plan. You may get some >> extra mileage when multiple operators share the same heap but the need to >> identify the memory requirement per operator does not go away. >> >> Thomas >> >> >> On Sat, Sep 26, 2015 at 12:41 PM, Munagala Ramanath <[email protected]> >> wrote: >> >> Would CONTAINER_LOCAL achieve the same thing and perform a little better >>> on >>> a multi-core box ? >>> >>> Ram >>> >>> On Sat, Sep 26, 2015 at 12:18 PM, Sandeep Deshmukh < >>> [email protected]> >>> wrote: >>> >>> Yes, with this approach only two containers are required: one for stram >>>> >>> and >>> >>>> another for all operators. You can easily fit around 10 operators in >>>> less >>>> than 1GB. >>>> On 27 Sep 2015 00:32, "Timothy Farkas" <[email protected]> wrote: >>>> >>>> Hi Ram, >>>>> >>>>> You could make all the operators thread local. This cuts down on the >>>>> overhead of separate containers and maximizes the memory available to >>>>> >>>> each >>>> >>>>> operator. >>>>> >>>>> Tim >>>>> >>>>> On Sat, Sep 26, 2015 at 10:07 AM, Munagala Ramanath < >>>>> >>>> [email protected] >>> >>>> wrote: >>>>> >>>>> Hi, >>>>>> >>>>>> I was running into memory issues when deploying my app on the >>>>>> >>>>> sandbox >>> >>>> where all the operators were stuck forever in the PENDING state >>>>>> >>>>> because >>> >>>> they were being continually aborted and restarted because of the >>>>>> >>>>> limited >>>> >>>>> memory on the sandbox. After some experimentation, I found that the >>>>>> following config values seem to work: >>>>>> ------------------------------------------ >>>>>> < >>>>>> >>>>> https://datatorrent.slack.com/archives/engineering/p1443263607000010> >>> >>>> >>>>>> >>>>>> *<property> <name>dt.attr.MASTER_MEMORY_MB</name> >>>>>> >>>>> <value>500</value> >>>>> >>>>>> </property> <property> <name>dt.application..operator.* >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *.attr.MEMORY_MB</name> <value>200</value> </property> >>>>>> >>>>> <property> >>> >>>> >>>>>> >>> <name>dt.application.TopNWordsWithQueries.operator.fileWordCount.attr.MEMORY_MB</name> >>> >>>> <value>512</value> </property>* >>>>>> ------------------------------------------------ >>>>>> Are these reasonable values ? Is there a more systematic way of >>>>>> >>>>> coming >>> >>>> up >>>> >>>>> with these values than trial-and-error ? Most of my operators -- with >>>>>> >>>>> the >>>> >>>>> exception of fileWordCount -- need very little memory; is there a way >>>>>> >>>>> to >>>> >>>>> cut all values down to the bare minimum and maximize available memory >>>>>> >>>>> for >>>> >>>>> this one operator ? >>>>>> >>>>>> Thanks. >>>>>> >>>>>> Ram >>>>>> >>>>>> >
