Fix in pull request, please review https://github.com/apache/incubator-apex-malhar/pull/178
On Tue, Jan 26, 2016 at 10:58 AM, Sandesh Hegde <[email protected]> wrote: > Siyuan, > > This needs to be fixed in local mode. App runs successfully on Yarn, but > fails in local mode. > > Thoughts ? > > Thanks > Sandesh > > On Tue, Jan 26, 2016 at 9:55 AM Shubham Pathak <[email protected]> > wrote: > > > Thanks Siyuan for the explanation ! > > > > On Tue, Jan 26, 2016 at 11:11 PM, Pramod Immaneni < > [email protected]> > > wrote: > > > > > That is a good explanation, can you add it to the JIRA as well. > > > > > > Thanks > > > > > > On Tue, Jan 26, 2016 at 9:39 AM, Siyuan Hua <[email protected]> > > > wrote: > > > > > > > @Pramod > > > > > > > > My understanding of local mode is the application is run in the same > > > > process as dtcli, so the default classloader can only load the > classes > > in > > > > dtcli classpath. > > > > When it tries to launch the application that has other dependencies, > in > > > > StramAppLauncher.loadDependencies() method, it tries to create a new > > > > ClassLoader to include those libs and set it as context classloader. > > > > So if the operator needs to load classes in it's own dependencies, it > > > needs > > > > to get that context classloader. > > > > > > > > On Tue, Jan 26, 2016 at 8:46 AM, Pramod Immaneni < > > [email protected] > > > > > > > > wrote: > > > > > > > > > The JIRA does not explain why it happens. Can you add the > > explanation. > > > It > > > > > might be useful if similar situations arise in future in other > > > scenarios. > > > > > > > > > > On Tue, Jan 26, 2016 at 7:17 AM, Sandesh Hegde < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Here is the jira for the issue > > > > > > https://issues.apache.org/jira/browse/APEXMALHAR-1984 > > > > > > > > > > > > It affects only local mode not on the cluster. I was trying it on > > > > > > DataTorrent Sandbox, in that machine local mode failed and passed > > on > > > > > yarn. > > > > > > > > > > > > On Tue, Jan 26, 2016 at 7:13 AM Shubham Pathak < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Could we have some more details as to what exactly is the > issue > > > and > > > > > why > > > > > > is > > > > > > > it failing in local mode ? > > > > > > > Also, i couldn't get why only local mode is mentioned > > specifically > > > ? > > > > > Does > > > > > > > it not affect when operator runs on the cluster ? > > > > > > > > > > > > > > Thanks, > > > > > > > Shubham > > > > > > > > > > > > > > On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni < > > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Siyuan can you explain the statement some more "It would fail > > in > > > > > local > > > > > > > mode > > > > > > > > if it tries to load class that only in app package > classpath." > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Sandesh and I found an issue in almost all the operators > that > > > use > > > > > > Kryo > > > > > > > to > > > > > > > > > clone operator itself in definePartition method. They all > > use > > > > > > default > > > > > > > > > Kryo instance without setting context classloader from the > > > thread > > > > > > that > > > > > > > > > execute the code. It would fail in local mode if it tries > to > > > load > > > > > > class > > > > > > > > > that only in app package classpath. > > > > > > > > > > > > > > > > > > When I'm trying to fix it in all the places, I found > repeated > > > > > pattern > > > > > > > in > > > > > > > > > definePartitions. And it's very error-prone since no one > > > actually > > > > > did > > > > > > > it > > > > > > > > in > > > > > > > > > right way until we found the issue. I'm thinking to make a > > > Util > > > > > > > functin > > > > > > > > to > > > > > > > > > create instance from operator itself and also hide the Kryo > > > > > > dependency > > > > > > > > from > > > > > > > > > the operator. Do you have any better suggestions to do > this? > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > Siyuan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
