Could we have some more details as to what exactly is the issue and why is it failing in local mode ? Also, i couldn't get why only local mode is mentioned specifically ? Does it not affect when operator runs on the cluster ?
Thanks, Shubham On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni <[email protected]> wrote: > Siyuan can you explain the statement some more "It would fail in local mode > if it tries to load class that only in app package classpath." > > Thanks > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua <[email protected]> > wrote: > > > Sandesh and I found an issue in almost all the operators that use Kryo to > > clone operator itself in definePartition method. They all use default > > Kryo instance without setting context classloader from the thread that > > execute the code. It would fail in local mode if it tries to load class > > that only in app package classpath. > > > > When I'm trying to fix it in all the places, I found repeated pattern in > > definePartitions. And it's very error-prone since no one actually did it > in > > right way until we found the issue. I'm thinking to make a Util functin > to > > create instance from operator itself and also hide the Kryo dependency > from > > the operator. Do you have any better suggestions to do this? > > > > Thanks! > > > > Siyuan > > >
