Awesome - will try this and report results. Thanks!
On 3/21/16, 11:15 AM, "Munagala Ramanath" <r...@datatorrent.com> wrote: >Please add the "*-XXMaxJavaStackTraceDepth=-1*" to the JVM options and >regenerate the stack trace. >Please note that the argument is a negative 1 which forces unlimited stack >trace depth. > >For example: > ><property> > <name>dt.attr.CONTAINER_JVM_OPTIONS</name> > <value>-XXMaxJavaStackTraceDepth=-1</value> ></property> > >Ram > >On Mon, Mar 21, 2016 at 11:06 AM, Ganelin, Ilya <ilya.gane...@capitalone.com >> wrote: > >> Ram - that is the complete log. I have nothing else available, either >> through YARN or through the DT UI. >> >> >> >> >> On 3/21/16, 10:33 AM, "Munagala Ramanath" <r...@datatorrent.com> wrote: >> >> >The call chain is not complete; it ends abruptly with: >> > >> >at java.util.ArrayList.writeObject(ArrayList.java:742) >> >at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) >> >at >> >> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >at java.lang.reflect.Method.invoke(Method.java:606) >> >at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988) >> >at >> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495) >> > >> > >> >We need to see the point of origin. >> > >> >Ram >> > >> >On Mon, Mar 21, 2016 at 10:02 AM, Ganelin, Ilya < >> ilya.gane...@capitalone.com >> >> wrote: >> > >> >> I uploaded the complete stack trace to the gist in the issue: >> >> https://gist.github.com/ilganeli/7f770374113b40ffa18a >> >> >> >> >> >> >> >> >> >> >> >> On 3/21/16, 9:38 AM, "Munagala Ramanath" <r...@datatorrent.com> wrote: >> >> >> >> >Ilya, could you upload a full stack trace of the failure so we can see >> >> >where the call chain >> >> >originated ? >> >> > >> >> >Ram >> >> > >> >> >On Mon, Mar 21, 2016 at 9:21 AM, Ganelin, Ilya < >> >> ilya.gane...@capitalone.com> >> >> >wrote: >> >> > >> >> >> Chandni- my application fails when launching in YARN, not in local >> mode. >> >> >> There is no custom partitioning - the code in the example is complete >> >> for >> >> >> both the input and output classes. >> >> >> >> >> >> >> >> >> >> >> >> Sent with Good (www.good.com) >> >> >> ________________________________ >> >> >> From: Chandni Singh <chan...@datatorrent.com> >> >> >> Sent: Monday, March 21, 2016 3:45:46 AM >> >> >> To: dev@apex.incubator.apache.org >> >> >> Subject: Re: Stack overflow errors when launching job >> >> >> >> >> >> >> >> >> debug.zip >> >> >> < >> >> >> >> >> >> https://drive.google.com/a/datatorrent.com/file/d/0BxX8sOLG8CxHLXFjUjBxM0hIZDg/view?usp=drive_web >> >> >> > >> >> >> Hi Ilya, >> >> >> >> >> >> Attached is the debug application with 20 partitions of input and >> output >> >> >> operators. I changed the default locality. This application doesn't >> >> fail in >> >> >> local mode. >> >> >> >> >> >> I am using the Stateless Partitioner for both Input and Output. >> >> >> Test configuration is in ApplicationTest and cluster configuration >> is in >> >> >> my-app-conf1.xml >> >> >> >> >> >> Have you added custom partitioning? They maybe causing the stack >> >> overflow >> >> >> in the app master. >> >> >> >> >> >> Can you modify this application so that the ApplicationTest throws >> this >> >> >> stack overflow? >> >> >> >> >> >> - Chandni >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Sun, Mar 20, 2016 at 11:30 AM, Chandni Singh < >> >> chan...@datatorrent.com> >> >> >> wrote: >> >> >> >> >> >> > Hi Ilya, >> >> >> > As Ram mentioned that we don't know the beginning of the stack >> track >> >> from >> >> >> > where this is triggered. We can add jvm options in the >> configuration >> >> file >> >> >> > so that app master is deployed with those configurations. >> >> >> > >> >> >> > Anyways I will look into creating this application (with 20 >> >> partitions) >> >> >> > and run it in local mode to find out where the problem is. >> >> >> > >> >> >> > Will get back to you today or tomorrow. >> >> >> > >> >> >> > Chandni >> >> >> > >> >> >> > On Sun, Mar 20, 2016 at 9:54 AM, Amol Kekre <a...@datatorrent.com> >> >> >> wrote: >> >> >> > >> >> >> >> Can we get on a webex to take a look? >> >> >> >> >> >> >> >> thks >> >> >> >> Amol >> >> >> >> >> >> >> >> >> >> >> >> On Sat, Mar 19, 2016 at 7:27 PM, Ganelin, Ilya < >> >> >> >> ilya.gane...@capitalone.com> >> >> >> >> wrote: >> >> >> >> >> >> >> >> > I don't think I have any time really to connect to the >> container. >> >> The >> >> >> >> > application launches and crashes almost immediately. Total >> runtime >> >> is >> >> >> 50 >> >> >> >> > seconds. >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > Sent with Good (www.good.com<http://www.good.com>) >> >> >> >> > ________________________________ >> >> >> >> > From: Munagala Ramanath <r...@datatorrent.com> >> >> >> >> > Sent: Saturday, March 19, 2016 5:39:11 PM >> >> >> >> > To: dev@apex.incubator.apache.org >> >> >> >> > Subject: Re: Stack overflow errors when launching job >> >> >> >> > >> >> >> >> > There is some info here, near the end of the page: >> >> >> >> > >> >> >> >> > http://docs.datatorrent.com/troubleshooting/ >> >> >> >> > >> >> >> >> > under the heading "How do I get a heap dump when a container >> gets >> >> an >> >> >> >> > OutOfMemoryError ?" >> >> >> >> > >> >> >> >> > However since you're blowing the stack, you may need to manually >> >> run >> >> >> >> jmap >> >> >> >> > on the running container >> >> >> >> > which may be difficult if it doesn't stay up for very long. >> There >> >> is a >> >> >> >> way >> >> >> >> > to dump the heap programmatically >> >> >> >> > as described, for instance, here: >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> https://blogs.oracle.com/sundararajan/entry/programmatically_dumping_heap_from_java >> >> >> >> > >> >> >> >> > Ram >> >> >> >> > >> >> >> >> > On Sat, Mar 19, 2016 at 2:07 PM, Ganelin, Ilya < >> >> >> >> > ilya.gane...@capitalone.com> >> >> >> >> > wrote: >> >> >> >> > >> >> >> >> > > How would we go about getting a heap dump? >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > Sent with Good (www.good.com<http://www.good.com< >> >> >> http://www.good.com<http://www.good.com>>) >> >> >> >> > > ________________________________ >> >> >> >> > > From: Yogi Devendra <yogideven...@apache.org> >> >> >> >> > > Sent: Saturday, March 19, 2016 12:19:26 AM >> >> >> >> > > To: dev@apex.incubator.apache.org >> >> >> >> > > Subject: Re: Stack overflow errors when launching job >> >> >> >> > > >> >> >> >> > > Stack trace in the gist shows some symptoms of infinite >> >> recursion. >> >> >> >> > > But, I could not figure out exact cause for it. >> >> >> >> > > >> >> >> >> > > Can you please check your heap dump to see if there are any >> >> cycles >> >> >> in >> >> >> >> the >> >> >> >> > > object hierarchy? >> >> >> >> > > >> >> >> >> > > ~ Yogi >> >> >> >> > > >> >> >> >> > > On 19 March 2016 at 00:36, Ashwin Chandra Putta < >> >> >> >> > ashwinchand...@gmail.com> >> >> >> >> > > wrote: >> >> >> >> > > >> >> >> >> > > > In the example you posted, do you have any locality >> constraint >> >> >> >> applied? >> >> >> >> > > > >> >> >> >> > > > From what I see, you have two operators - hdfs input >> operator >> >> and >> >> >> >> hdfs >> >> >> >> > > > output operator. Each of them have 40 partitions each and >> you >> >> >> don't >> >> >> >> > have >> >> >> >> > > > any other constraints on them. And the partitioner >> >> implementation >> >> >> >> you >> >> >> >> > are >> >> >> >> > > > using is >> >> com.datatorrent.common.partitioner.StatelessPartitioner >> >> >> >> > > > >> >> >> >> > > > Please confirm. >> >> >> >> > > > >> >> >> >> > > > Regards, >> >> >> >> > > > Ashwin. >> >> >> >> > > > >> >> >> >> > > > On Thu, Mar 17, 2016 at 5:00 PM, Ganelin, Ilya < >> >> >> >> > > > ilya.gane...@capitalone.com> >> >> >> >> > > > wrote: >> >> >> >> > > > >> >> >> >> > > > > I’ve updated the gist with a more complete example, and >> >> updated >> >> >> >> the >> >> >> >> > > > > associated JIRA that I’ve created. >> >> >> >> > > > > https://issues.apache.org/jira/browse/APEXCORE-392 >> >> >> >> > > > > >> >> >> >> > > > > >> >> >> >> > > > > >> >> >> >> > > > > >> >> >> >> > > > > >> >> >> >> > > > > On 3/17/16, 4:33 AM, "Tushar Gosavi" < >> tus...@datatorrent.com >> >> > >> >> >> >> wrote: >> >> >> >> > > > > >> >> >> >> > > > > >Hi, >> >> >> >> > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >I created a sample application with operators from the >> given >> >> >> >> link. >> >> >> >> > > just >> >> >> >> > > > a >> >> >> >> > > > > >simple input and output and created 32 partitions of >> each. >> >> >> Could >> >> >> >> not >> >> >> >> > > > > >reproduce the >> >> >> >> > > > > >stack overflow issue. Do you have a small sample >> application >> >> >> >> which >> >> >> >> > > could >> >> >> >> > > > > >reproduce this issue? >> >> >> >> > > > > > >> >> >> >> > > > > > @Override >> >> >> >> > > > > > public void populateDAG(DAG dag, Configuration >> >> configuration) >> >> >> >> > > > > > { >> >> >> >> > > > > > NewlineFileInputOperator in = >> dag.addOperator("Input", >> >> new >> >> >> >> > > > > >NewlineFileInputOperator()); >> >> >> >> > > > > > in.setDirectory("/user/tushar/data"); >> >> >> >> > > > > > in.setPartitionCount(32); >> >> >> >> > > > > > >> >> >> >> > > > > > HdfsFileOutputOperator out = >> dag.addOperator("Output", >> >> new >> >> >> >> > > > > >HdfsFileOutputOperator()); >> >> >> >> > > > > > out.setFilePath("/user/tushar/outdata"); >> >> >> >> > > > > > >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >dag.getMeta(out).getAttributes().put(Context.OperatorContext.PARTITIONER, >> >> >> >> > > > > >new StatelessPartitioner<HdfsFileOutputOperator>(32)); >> >> >> >> > > > > > >> >> >> >> > > > > > dag.addStream("s1", in.output, out.input); >> >> >> >> > > > > > } >> >> >> >> > > > > > >> >> >> >> > > > > >-Tushar. >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > > >> >> >> >> > > > > >On Thu, Mar 17, 2016 at 12:30 AM, Ganelin, Ilya < >> >> >> >> > > > > ilya.gane...@capitalone.com >> >> >> >> > > > > >> wrote: >> >> >> >> > > > > > >> >> >> >> > > > > >> Hi guys – I’m running into a very frustrating issue >> where >> >> >> >> certain >> >> >> >> > > DAG >> >> >> >> > > > > >> configurations cause the following error log >> (attached). >> >> When >> >> >> >> this >> >> >> >> > > > > happens, >> >> >> >> > > > > >> my application even fails to launch. This does not >> seem to >> >> >> be a >> >> >> >> > YARN >> >> >> >> > > > > issue >> >> >> >> > > > > >> since this occurs even with a relatively small number >> of >> >> >> >> > > > > partitions/memory. >> >> >> >> > > > > >> >> >> >> >> > > > > >> I’ve attached the input and output operators in >> question: >> >> >> >> > > > > >> https://gist.github.com/ilganeli/7f770374113b40ffa18a >> >> >> >> > > > > >> >> >> >> >> > > > > >> I can get this to occur predictable by >> >> >> >> > > > > >> >> >> >> >> > > > > >> 1. Increasing the partition count on my input >> operator >> >> >> >> (reads >> >> >> >> > > from >> >> >> >> > > > > >> HDFS) - values above 20 cause this error >> >> >> >> > > > > >> 2. Increase the partition count on my output >> operator >> >> >> >> (writes >> >> >> >> > to >> >> >> >> > > > > HDFS) >> >> >> >> > > > > >> - values above 20 cause this error >> >> >> >> > > > > >> 3. Set stream locality from the default to either >> >> thread >> >> >> >> local, >> >> >> >> > > > node >> >> >> >> > > > > >> local, or container_local on the output operator >> >> >> >> > > > > >> >> >> >> >> > > > > >> This behavior is very frustrating as it’s preventing me >> >> from >> >> >> >> > > > > partitioning >> >> >> >> > > > > >> my HDFS I/O appropriately, thus allowing me to scale to >> >> >> higher >> >> >> >> > > > > throughputs. >> >> >> >> > > > > >> >> >> >> >> > > > > >> Do you have any thoughts on what’s going wrong? I would >> >> love >> >> >> >> your >> >> >> >> > > > > feedback. >> >> >> >> > > > > >> >> ________________________________________________________ >> >> >> >> > > > > >> >> >> >> >> > > > > >> The information contained in this e-mail is >> confidential >> >> >> and/or >> >> >> >> > > > > >> proprietary to Capital One and/or its affiliates and >> may >> >> only >> >> >> >> be >> >> >> >> > > used >> >> >> >> > > > > >> solely in performance of work or services for Capital >> One. >> >> >> The >> >> >> >> > > > > information >> >> >> >> > > > > >> transmitted herewith is intended only for use by the >> >> >> >> individual or >> >> >> >> > > > > entity >> >> >> >> > > > > >> to which it is addressed. If the reader of this >> message is >> >> >> not >> >> >> >> the >> >> >> >> > > > > intended >> >> >> >> > > > > >> recipient, you are hereby notified that any review, >> >> >> >> > retransmission, >> >> >> >> > > > > >> dissemination, distribution, copying or other use of, >> or >> >> >> >> taking of >> >> >> >> > > any >> >> >> >> > > > > >> action in reliance upon this information is strictly >> >> >> >> prohibited. >> >> >> >> > If >> >> >> >> > > > you >> >> >> >> > > > > >> have received this communication in error, please >> contact >> >> the >> >> >> >> > sender >> >> >> >> > > > and >> >> >> >> > > > > >> delete the material from your computer. >> >> >> >> > > > > >> >> >> >> >> > > > > ________________________________________________________ >> >> >> >> > > > > >> >> >> >> > > > > The information contained in this e-mail is confidential >> >> and/or >> >> >> >> > > > > proprietary to Capital One and/or its affiliates and may >> >> only be >> >> >> >> used >> >> >> >> > > > > solely in performance of work or services for Capital One. >> >> The >> >> >> >> > > > information >> >> >> >> > > > > transmitted herewith is intended only for use by the >> >> individual >> >> >> or >> >> >> >> > > entity >> >> >> >> > > > > to which it is addressed. If the reader of this message is >> >> not >> >> >> the >> >> >> >> > > > intended >> >> >> >> > > > > recipient, you are hereby notified that any review, >> >> >> >> retransmission, >> >> >> >> > > > > dissemination, distribution, copying or other use of, or >> >> taking >> >> >> of >> >> >> >> > any >> >> >> >> > > > > action in reliance upon this information is strictly >> >> prohibited. >> >> >> >> If >> >> >> >> > you >> >> >> >> > > > > have received this communication in error, please contact >> the >> >> >> >> sender >> >> >> >> > > and >> >> >> >> > > > > delete the material from your computer. >> >> >> >> > > > > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> > > > -- >> >> >> >> > > > >> >> >> >> > > > Regards, >> >> >> >> > > > Ashwin. >> >> >> >> > > > >> >> >> >> > > ________________________________________________________ >> >> >> >> > > >> >> >> >> > > The information contained in this e-mail is confidential >> and/or >> >> >> >> > > proprietary to Capital One and/or its affiliates and may only >> be >> >> >> used >> >> >> >> > > solely in performance of work or services for Capital One. The >> >> >> >> > information >> >> >> >> > > transmitted herewith is intended only for use by the >> individual >> >> or >> >> >> >> entity >> >> >> >> > > to which it is addressed. If the reader of this message is not >> >> the >> >> >> >> > intended >> >> >> >> > > recipient, you are hereby notified that any review, >> >> retransmission, >> >> >> >> > > dissemination, distribution, copying or other use of, or >> taking >> >> of >> >> >> any >> >> >> >> > > action in reliance upon this information is strictly >> prohibited. >> >> If >> >> >> >> you >> >> >> >> > > have received this communication in error, please contact the >> >> sender >> >> >> >> and >> >> >> >> > > delete the material from your computer. >> >> >> >> > > >> >> >> >> > ________________________________________________________ >> >> >> >> > >> >> >> >> > The information contained in this e-mail is confidential and/or >> >> >> >> > proprietary to Capital One and/or its affiliates and may only be >> >> used >> >> >> >> > solely in performance of work or services for Capital One. The >> >> >> >> information >> >> >> >> > transmitted herewith is intended only for use by the individual >> or >> >> >> >> entity >> >> >> >> > to which it is addressed. If the reader of this message is not >> the >> >> >> >> intended >> >> >> >> > recipient, you are hereby notified that any review, >> retransmission, >> >> >> >> > dissemination, distribution, copying or other use of, or taking >> of >> >> any >> >> >> >> > action in reliance upon this information is strictly >> prohibited. If >> >> >> you >> >> >> >> > have received this communication in error, please contact the >> >> sender >> >> >> and >> >> >> >> > delete the material from your computer. >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> > >> >> >> ________________________________________________________ >> >> >> >> >> >> The information contained in this e-mail is confidential and/or >> >> >> proprietary to Capital One and/or its affiliates and may only be used >> >> >> solely in performance of work or services for Capital One. The >> >> information >> >> >> transmitted herewith is intended only for use by the individual or >> >> entity >> >> >> to which it is addressed. If the reader of this message is not the >> >> intended >> >> >> recipient, you are hereby notified that any review, retransmission, >> >> >> dissemination, distribution, copying or other use of, or taking of >> any >> >> >> action in reliance upon this information is strictly prohibited. If >> you >> >> >> have received this communication in error, please contact the sender >> and >> >> >> delete the material from your computer. >> >> >> >> >> ________________________________________________________ >> >> >> >> The information contained in this e-mail is confidential and/or >> >> proprietary to Capital One and/or its affiliates and may only be used >> >> solely in performance of work or services for Capital One. The >> information >> >> transmitted herewith is intended only for use by the individual or >> entity >> >> to which it is addressed. If the reader of this message is not the >> intended >> >> recipient, you are hereby notified that any review, retransmission, >> >> dissemination, distribution, copying or other use of, or taking of any >> >> action in reliance upon this information is strictly prohibited. If you >> >> have received this communication in error, please contact the sender and >> >> delete the material from your computer. >> >> >> ________________________________________________________ >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
