That is a good explanation, can you add it to the JIRA as well. Thanks
On Tue, Jan 26, 2016 at 9:39 AM, Siyuan Hua <[email protected]> wrote: > @Pramod > > My understanding of local mode is the application is run in the same > process as dtcli, so the default classloader can only load the classes in > dtcli classpath. > When it tries to launch the application that has other dependencies, in > StramAppLauncher.loadDependencies() method, it tries to create a new > ClassLoader to include those libs and set it as context classloader. > So if the operator needs to load classes in it's own dependencies, it needs > to get that context classloader. > > On Tue, Jan 26, 2016 at 8:46 AM, Pramod Immaneni <[email protected]> > wrote: > > > The JIRA does not explain why it happens. Can you add the explanation. It > > might be useful if similar situations arise in future in other scenarios. > > > > On Tue, Jan 26, 2016 at 7:17 AM, Sandesh Hegde <[email protected]> > > wrote: > > > > > Here is the jira for the issue > > > https://issues.apache.org/jira/browse/APEXMALHAR-1984 > > > > > > It affects only local mode not on the cluster. I was trying it on > > > DataTorrent Sandbox, in that machine local mode failed and passed on > > yarn. > > > > > > On Tue, Jan 26, 2016 at 7:13 AM Shubham Pathak < > [email protected]> > > > wrote: > > > > > > > Could we have some more details as to what exactly is the issue and > > why > > > is > > > > it failing in local mode ? > > > > Also, i couldn't get why only local mode is mentioned specifically ? > > Does > > > > it not affect when operator runs on the cluster ? > > > > > > > > Thanks, > > > > Shubham > > > > > > > > On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni < > > [email protected] > > > > > > > > wrote: > > > > > > > > > Siyuan can you explain the statement some more "It would fail in > > local > > > > mode > > > > > if it tries to load class that only in app package classpath." > > > > > > > > > > Thanks > > > > > > > > > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua < > [email protected]> > > > > > wrote: > > > > > > > > > > > Sandesh and I found an issue in almost all the operators that use > > > Kryo > > > > to > > > > > > clone operator itself in definePartition method. They all use > > > default > > > > > > Kryo instance without setting context classloader from the thread > > > that > > > > > > execute the code. It would fail in local mode if it tries to load > > > class > > > > > > that only in app package classpath. > > > > > > > > > > > > When I'm trying to fix it in all the places, I found repeated > > pattern > > > > in > > > > > > definePartitions. And it's very error-prone since no one actually > > did > > > > it > > > > > in > > > > > > right way until we found the issue. I'm thinking to make a Util > > > > functin > > > > > to > > > > > > create instance from operator itself and also hide the Kryo > > > dependency > > > > > from > > > > > > the operator. Do you have any better suggestions to do this? > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Siyuan > > > > > > > > > > > > > > > > > > > > >
