Thanks Siyuan for the explanation ! On Tue, Jan 26, 2016 at 11:11 PM, Pramod Immaneni <[email protected]> wrote:
> That is a good explanation, can you add it to the JIRA as well. > > Thanks > > On Tue, Jan 26, 2016 at 9:39 AM, Siyuan Hua <[email protected]> > wrote: > > > @Pramod > > > > My understanding of local mode is the application is run in the same > > process as dtcli, so the default classloader can only load the classes in > > dtcli classpath. > > When it tries to launch the application that has other dependencies, in > > StramAppLauncher.loadDependencies() method, it tries to create a new > > ClassLoader to include those libs and set it as context classloader. > > So if the operator needs to load classes in it's own dependencies, it > needs > > to get that context classloader. > > > > On Tue, Jan 26, 2016 at 8:46 AM, Pramod Immaneni <[email protected] > > > > wrote: > > > > > The JIRA does not explain why it happens. Can you add the explanation. > It > > > might be useful if similar situations arise in future in other > scenarios. > > > > > > On Tue, Jan 26, 2016 at 7:17 AM, Sandesh Hegde < > [email protected]> > > > wrote: > > > > > > > Here is the jira for the issue > > > > https://issues.apache.org/jira/browse/APEXMALHAR-1984 > > > > > > > > It affects only local mode not on the cluster. I was trying it on > > > > DataTorrent Sandbox, in that machine local mode failed and passed on > > > yarn. > > > > > > > > On Tue, Jan 26, 2016 at 7:13 AM Shubham Pathak < > > [email protected]> > > > > wrote: > > > > > > > > > Could we have some more details as to what exactly is the issue > and > > > why > > > > is > > > > > it failing in local mode ? > > > > > Also, i couldn't get why only local mode is mentioned specifically > ? > > > Does > > > > > it not affect when operator runs on the cluster ? > > > > > > > > > > Thanks, > > > > > Shubham > > > > > > > > > > On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni < > > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > Siyuan can you explain the statement some more "It would fail in > > > local > > > > > mode > > > > > > if it tries to load class that only in app package classpath." > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Sandesh and I found an issue in almost all the operators that > use > > > > Kryo > > > > > to > > > > > > > clone operator itself in definePartition method. They all use > > > > default > > > > > > > Kryo instance without setting context classloader from the > thread > > > > that > > > > > > > execute the code. It would fail in local mode if it tries to > load > > > > class > > > > > > > that only in app package classpath. > > > > > > > > > > > > > > When I'm trying to fix it in all the places, I found repeated > > > pattern > > > > > in > > > > > > > definePartitions. And it's very error-prone since no one > actually > > > did > > > > > it > > > > > > in > > > > > > > right way until we found the issue. I'm thinking to make a > Util > > > > > functin > > > > > > to > > > > > > > create instance from operator itself and also hide the Kryo > > > > dependency > > > > > > from > > > > > > > the operator. Do you have any better suggestions to do this? > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Siyuan > > > > > > > > > > > > > > > > > > > > > > > > > > > >
