Instead of tweak memory settings I think you can tweak memory setting for yarn. In this way, you can launch applications with more operators with less risk of outofmemory error and also show different localities
Sent from my iPhone > On Sep 26, 2015, at 12:18, Sandeep Deshmukh <[email protected]> wrote: > > Yes, with this approach only two containers are required: one for stram and > another for all operators. You can easily fit around 10 operators in less > than 1GB. >> On 27 Sep 2015 00:32, "Timothy Farkas" <[email protected]> wrote: >> >> Hi Ram, >> >> You could make all the operators thread local. This cuts down on the >> overhead of separate containers and maximizes the memory available to each >> operator. >> >> Tim >> >> On Sat, Sep 26, 2015 at 10:07 AM, Munagala Ramanath <[email protected]> >> wrote: >> >>> Hi, >>> >>> I was running into memory issues when deploying my app on the sandbox >>> where all the operators were stuck forever in the PENDING state because >>> they were being continually aborted and restarted because of the limited >>> memory on the sandbox. After some experimentation, I found that the >>> following config values seem to work: >>> ------------------------------------------ >>> <https://datatorrent.slack.com/archives/engineering/p1443263607000010> >>> >>> >>> >>> *<property> <name>dt.attr.MASTER_MEMORY_MB</name> >> <value>500</value> >>> </property> <property> <name>dt.application..operator.* >>> >>> >>> >>> >>> >>> *.attr.MEMORY_MB</name> <value>200</value> </property> <property> >> <name>dt.application.TopNWordsWithQueries.operator.fileWordCount.attr.MEMORY_MB</name> >>> <value>512</value> </property>* >>> ------------------------------------------------ >>> Are these reasonable values ? Is there a more systematic way of coming up >>> with these values than trial-and-error ? Most of my operators -- with the >>> exception of fileWordCount -- need very little memory; is there a way to >>> cut all values down to the bare minimum and maximize available memory for >>> this one operator ? >>> >>> Thanks. >>> >>> Ram >>
