Container & memory resource allocation

[Pradeep A. Dalvi]

Container & memory resource allocation has been a common question around
and so I thought it would be good to explain related configuration
parameters.

Please feel free to let me know your thoughts.

Also I'm planning to add following set of information under Apex Docs. How
could one add this to Apex Docs?

=-=-=-=

"Container is running beyond physical memory limits. Current usage: X GB of
Y GB physical memory used; A GB of B GB virtual memory used. Killing
container."

This is basically for some better understanding on Application Master's
container requests & Resource Manager's memory resource allocation. Please
note that these are individual container request params. All these
parameters are in MB i.e. 1024 => 1GB.

- AM's container requests to RM shall contain memory in the multiples of
*yarn.scheduler.minimum-**allocation-mb* & not exceeding
*yarn.scheduler.maximum-**allocation-mb*
   - If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024 and
container memory requirement is 1025 ( <= 2048 ), container will be
allocated with 2048 memory.

- With Apex applications, operator memory can be specified by property
*dt.application..operator..attr.MEMORY_MB*
   - Please note this parameter is at Operator level and container memory
is calculated based on number of Operators deployed in a container +
additional memory required depending on physical deployment requirements
e.g. unifier or bufferserver
   - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME

- If container memory is not specified, then AM would request for 1 unit of
*yarn.scheduler.minimum-**allocation-mb*, RM would provision container
taking that into consideration.

Node Manager monitors memory usage of each of these containers and kills
the ones crossing the configured limit.

Almost similar stuff is applicable for CPUs.

--prad

Re: Force Fail Application

[Sandesh Hegde]

Platform logs that an Operator has requested shutdown, below is the message.
 logger.debug("Shutdown requested by the operator when alive = {}.", alive);

Specific message is left to the business logic of an operator, it can log
before throwing the ShutdownException.

On Mon, Jul 18, 2016 at 11:56 AM Silver, Michael <
michael.sil...@capitalone.com> wrote:

> I would like the error to show up in the logs as an exception.
> Additionally I would like the error to be displayed with some helpful
> information such as why the application is being shutdown.
>
>
>
> On 2016-07-18 13:14 (-0400), Sandesh Hegde  wrote:
>
> > What is the outcome you are expecting from the "true exception" ?>
>
> >
>
> > On Mon, Jul 18, 2016 at 10:11 AM Silver, Michael <>
>
> > michael.sil...@capitalone.com> wrote:>
>
> >
>
> > > I looked into the ShutdownException. My only concern is that I want
> the>
>
> > > exception to be treated like a true error/exception and according to
> the>
>
> > > docs it is not treated as such.>
>
> > >>
>
> > > /**>
>
> > > * The operator should throw the following exception if it wants to>
>
> > > gracefully conclude its operation.>
>
> > > * This exception is not treated as an error by the engine. It's
> considered>
>
> > > a request by the operator>
>
> > > * to deactivate itself. Upon receiving this, the engine would let the>
>
> > > operator finish its in progress>
>
> > > * window and then call deactivate method on it if present.>
>
> > > *>
>
> > > */>
>
> > >>
>
> > >>
>
> > >>
>
> > >>
>
> > >>
>
> > > On 2016-07-18 13:08 (-0400), "h...@gmail.com" 
> wrote:>
>
> > >>
>
> > > > He Michael,>>
>
> > >>
>
> > > >>
>
> > >>
>
> > > > You can throw a ShutdownException.>>
>
> > >>
>
> > > >>
>
> > >>
>
> > > > Siyuan>>
>
> > >>
>
> > > >>
>
> > >>
>
> > > > On Mon, Jul 18, 2016 at 10:06 AM, Silver, Michael <>>
>
> > >>
>
> > > > michael.sil...@capitalone.com> wrote:>>
>
> > >>
>
> > > >>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > > Hello,>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > > I am looking for a solution to force shutdown or fail my
> application.>
>
> > > I>>
>
> > >>
>
> > > > > have an operator that checks that a file (which is needed for
> the>>
>
> > >>
>
> > > > > application to run) is present during setup. If the file is not>
>
> > > present I>>
>
> > >>
>
> > > > > want the entire application to fail. How would I do this in
> apex?>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > > Thank you,>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > > Michael>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > > -->>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > > > The information contained in this e-mail is confidential and/or>>
>
> > >>
>
> > > > > proprietary to Capital One and/or its affiliates and may only be
> used>>
>
> > >>
>
> > > > > solely in performance of work or services for Capital One. The>
>
> > > information>>
>
> > >>
>
> > > > > transmitted herewith is intended only for use by the individual
> or>
>
> > > entity>>
>
> > >>
>
> > > > > to which it is addressed. If the reader of this message is not
> the>
>
> > > intended>>
>
> > >>
>
> > > > > recipient, you are hereby notified that any review,
> retransmission,>>
>
> > >>
>
> > > > > dissemination, distribution, copying or other use of, or taking
> of>
>
> > > any>>
>
> > >>
>
> > > > > action in reliance upon this information is strictly prohibited.
> If>
>
> > > you>>
>
> > >>
>
> > > > > have received this communication in error, please contact the
> sender>
>
> > > and>>
>
> > >>
>
> > > > > delete the material from your computer.>>
>
> > >>
>
> > > > >>>
>
> > >>
>
> > > >>
>
> > >>
>
> > > -->
>
> > >>
>
> > > The information contained in this e-mail is confidential and/or>
>
> > > proprietary to Capital One and/or its affiliates and may only be used>
>
> > > solely in performance of work or services for Capital One. The
> information>
>
> > > transmitted herewith is intended only for use by the individual or
> entity>
>
> > > to which it is addressed. If the reader of this message is not the
> intended>
>
> > > recipient, you are hereby notified that any review, retransmission,>
>
> > > dissemination, distribution, copying or other use of, or taking of
> any>
>
> > > action in reliance upon this information is strictly prohibited. If
> you>
>
> > > have received this communication in error, please contact the sender
> and>
>
> > > delete the material from your computer.>
>
> > >>
>
> >
>
> --
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> 

Re: Force Fail Application

[Sandesh Hegde]

What is the outcome you are expecting from the "true exception" ?

On Mon, Jul 18, 2016 at 10:11 AM Silver, Michael <
michael.sil...@capitalone.com> wrote:

> I looked into the ShutdownException. My only concern is that I want the
> exception to be treated like a true error/exception and according to the
> docs it is not treated as such.
>
> /**
> * The operator should throw the following exception if it wants to
> gracefully conclude its operation.
> * This exception is not treated as an error by the engine. It's considered
> a request by the operator
> * to deactivate itself. Upon receiving this, the engine would let the
> operator finish its in progress
> * window and then call deactivate method on it if present.
> *
> */
>
>
>
>
>
> On 2016-07-18 13:08 (-0400), "h...@gmail.com"  wrote:
>
> > He Michael,>
>
> >
>
> > You can throw a ShutdownException.>
>
> >
>
> > Siyuan>
>
> >
>
> > On Mon, Jul 18, 2016 at 10:06 AM, Silver, Michael <>
>
> > michael.sil...@capitalone.com> wrote:>
>
> >
>
> > >>
>
> > >>
>
> > >>
>
> > >>
>
> > > Hello,>
>
> > >>
>
> > >>
>
> > >>
>
> > > I am looking for a solution to force shutdown or fail my application.
> I>
>
> > > have an operator that checks that a file (which is needed for the>
>
> > > application to run) is present during setup. If the file is not
> present I>
>
> > > want the entire application to fail. How would I do this in apex?>
>
> > >>
>
> > >>
>
> > >>
>
> > > Thank you,>
>
> > >>
>
> > >>
>
> > >>
>
> > > Michael>
>
> > >>
>
> > > -->
>
> > >>
>
> > > The information contained in this e-mail is confidential and/or>
>
> > > proprietary to Capital One and/or its affiliates and may only be used>
>
> > > solely in performance of work or services for Capital One. The
> information>
>
> > > transmitted herewith is intended only for use by the individual or
> entity>
>
> > > to which it is addressed. If the reader of this message is not the
> intended>
>
> > > recipient, you are hereby notified that any review, retransmission,>
>
> > > dissemination, distribution, copying or other use of, or taking of
> any>
>
> > > action in reliance upon this information is strictly prohibited. If
> you>
>
> > > have received this communication in error, please contact the sender
> and>
>
> > > delete the material from your computer.>
>
> > >>
>
> >
>
> --
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Re: Force Fail Application

[Silver, Michael]

I looked into the ShutdownException. My only concern is that I want the 
exception to be treated like a true error/exception and according to the docs 
it is not treated as such.
/**
* The operator should throw the following exception if it wants to gracefully 
conclude its operation.
* This exception is not treated as an error by the engine. It's considered a 
request by the operator
* to deactivate itself. Upon receiving this, the engine would let the operator 
finish its in progress
* window and then call deactivate method on it if present.
*
*/


On 2016-07-18 13:08 (-0400), "h...@gmail.com" 
> wrote:
> He Michael,>
>
> You can throw a ShutdownException.>
>
> Siyuan>
>
> On Mon, Jul 18, 2016 at 10:06 AM, Silver, Michael <>
> michael.sil...@capitalone.com> wrote:>
>
> >>
> >>
> >>
> >>
> > Hello,>
> >>
> >>
> >>
> > I am looking for a solution to force shutdown or fail my application. I>
> > have an operator that checks that a file (which is needed for the>
> > application to run) is present during setup. If the file is not present I>
> > want the entire application to fail. How would I do this in apex?>
> >>
> >>
> >>
> > Thank you,>
> >>
> >>
> >>
> > Michael>
> >>
> > -->
> >>
> > The information contained in this e-mail is confidential and/or>
> > proprietary to Capital One and/or its affiliates and may only be used>
> > solely in performance of work or services for Capital One. The information>
> > transmitted herewith is intended only for use by the individual or entity>
> > to which it is addressed. If the reader of this message is not the intended>
> > recipient, you are hereby notified that any review, retransmission,>
> > dissemination, distribution, copying or other use of, or taking of any>
> > action in reliance upon this information is strictly prohibited. If you>
> > have received this communication in error, please contact the sender and>
> > delete the material from your computer.>
> >>
>


The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

Re: Force Fail Application

[Devendra Tagare]

Hi,

Please see a sample here,

https://github.com/apache/apex-malhar/blob/3ce83708f795b081d564be357a8333928154398e/library/src/test/java/com/datatorrent/lib/testbench/SeedEventGeneratorTest.java

Thanks,
Dev

On Mon, Jul 18, 2016 at 10:08 AM, hsy...@gmail.com  wrote:

> He Michael,
>
> You can throw a ShutdownException.
>
> Siyuan
>
> On Mon, Jul 18, 2016 at 10:06 AM, Silver, Michael <
> michael.sil...@capitalone.com> wrote:
>
>>
>>
>>
>>
>> Hello,
>>
>>
>>
>> I am looking for a solution to force shutdown or fail my application. I
>> have an operator that checks that a file (which is needed for the
>> application to run) is present during setup. If the file is not present I
>> want the entire application to fail. How would I do this in apex?
>>
>>
>>
>> Thank you,
>>
>>
>>
>> Michael
>>
>> --
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>
>

Re: Force Fail Application

[hsy...@gmail.com]

He Michael,

You can throw a ShutdownException.

Siyuan

On Mon, Jul 18, 2016 at 10:06 AM, Silver, Michael <
michael.sil...@capitalone.com> wrote:

>
>
>
>
> Hello,
>
>
>
> I am looking for a solution to force shutdown or fail my application. I
> have an operator that checks that a file (which is needed for the
> application to run) is present during setup. If the file is not present I
> want the entire application to fail. How would I do this in apex?
>
>
>
> Thank you,
>
>
>
> Michael
>
> --
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Force Fail Application

[Silver, Michael]

Hello,

I am looking for a solution to force shutdown or fail my application. I have an 
operator that checks that a file (which is needed for the application to run) 
is present during setup. If the file is not present I want the entire 
application to fail. How would I do this in apex?

Thank you,

Michael


The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

CsvFormatter schema for enum property

[Yogi Devendra]

Hi,

We have a java object which has a enum field. We wish to send these objects
to CSVFormatter.
What should I specify for "type" of this field?

For example,

My Java Object is :
https://github.com/yogidevendra/examples/blob/13cd0ec8d24b5e5b125fe7567c18197b0a9a8f7e/tutorials/filter/src/main/java/com/datatorrent/tutorial/filter/TransactionPOJO.java

I am using schema as follows:
{"separator": "|","quoteChar":"\"","lineDelimiter":"\n","fields": [{"name":
"trasactionId","type": "long"},{"name": "amount","type": "double"},{"name":
"type","type":
"com.datatorrent.tutorial.filter.TransactionPOJO.TRANSACTION_TYPE"}]}

But, it gives following exception:

Abandoning deployment due to setup failure.
java.lang.IllegalArgumentException: No enum constant
com.datatorrent.contrib.parser.DelimitedSchema.FieldType.COM.DATATORRENT.TUTORIAL.FILTER.TRANSACTIONPOJO.TRANSACTION_TYPE
at java.lang.Enum.valueOf(Enum.java:236)
at
com.datatorrent.contrib.parser.DelimitedSchema$FieldType.valueOf(DelimitedSchema.java:161)
at
com.datatorrent.contrib.parser.DelimitedSchema$Field.(DelimitedSchema.java:289)
at
com.datatorrent.contrib.parser.DelimitedSchema.initialize(DelimitedSchema.java:199)
at
com.datatorrent.contrib.parser.DelimitedSchema.(DelimitedSchema.java:169)
at
com.datatorrent.contrib.formatter.CsvFormatter.setup(CsvFormatter.java:128)
at
com.datatorrent.contrib.formatter.CsvFormatter.setup(CsvFormatter.java:67)
at com.datatorrent.stram.engine.Node.setup(Node.java:187)
at
com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1309)
at
com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:130)
at
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1388)

~ Yogi