In the benchmark test THREAD_LOCAL outperforms CONTAINER_LOCAL by an
order of magnitude and both operators compete for CPU. I'll take a
closer look why.
Thank you,
Vlad
On 9/26/15 14:52, Thomas Weise wrote:
THREAD_LOCAL - operators share thread
CONTAINER_LOCAL - each operator has its own thread
So as long as operators utilize the CPU sufficiently (compete), the latter
will perform better.
There will be cases where a single thread can accommodate multiple
operators. For example, a socket reader (mostly waiting for IO) and a
decompress (CPU hungry) can share a thread.
But to get back to the original question, stream locality does generally
not reduce the total memory requirement. If you add multiple operators into
one container, that container will also require more memory and that's how
the container size is calculated in the physical plan. You may get some
extra mileage when multiple operators share the same heap but the need to
identify the memory requirement per operator does not go away.
Thomas
On Sat, Sep 26, 2015 at 12:41 PM, Munagala Ramanath <[email protected]>
wrote:
Would CONTAINER_LOCAL achieve the same thing and perform a little better on
a multi-core box ?
Ram
On Sat, Sep 26, 2015 at 12:18 PM, Sandeep Deshmukh <
[email protected]>
wrote:
Yes, with this approach only two containers are required: one for stram
and
another for all operators. You can easily fit around 10 operators in less
than 1GB.
On 27 Sep 2015 00:32, "Timothy Farkas" <[email protected]> wrote:
Hi Ram,
You could make all the operators thread local. This cuts down on the
overhead of separate containers and maximizes the memory available to
each
operator.
Tim
On Sat, Sep 26, 2015 at 10:07 AM, Munagala Ramanath <
[email protected]
wrote:
Hi,
I was running into memory issues when deploying my app on the
sandbox
where all the operators were stuck forever in the PENDING state
because
they were being continually aborted and restarted because of the
limited
memory on the sandbox. After some experimentation, I found that the
following config values seem to work:
------------------------------------------
<
https://datatorrent.slack.com/archives/engineering/p1443263607000010>
*<property> <name>dt.attr.MASTER_MEMORY_MB</name>
<value>500</value>
</property> <property> <name>dt.application..operator.*
*.attr.MEMORY_MB</name> <value>200</value> </property>
<property>
<name>dt.application.TopNWordsWithQueries.operator.fileWordCount.attr.MEMORY_MB</name>
<value>512</value> </property>*
------------------------------------------------
Are these reasonable values ? Is there a more systematic way of
coming
up
with these values than trial-and-error ? Most of my operators -- with
the
exception of fileWordCount -- need very little memory; is there a way
to
cut all values down to the bare minimum and maximize available memory
for
this one operator ?
Thanks.
Ram
