Hi, I'm currently in the process of porting an Apache project for Big-Data analysis (Apache Drill) to YARN. I'm using Twill for that manner and it almost works.
The last problem, I can't seem to figure out, is an exception from janino. Apache Drill uses the compiler to create classes on the fly and that works fine when I run Drill directly. But when run in Apache Twill I get the exception from Janino that a class couldn't be loaded. I've added this specific class as a dependency and also could find it in the container jar-file which was created by Twill. And the class really is there, because the class which can't be found, is the same as the exception which is thrown. - Yes that same class to which the original CompileException is transformed in the end, is the same as the class which is imported as first and can't be found. I suspect that there is a problem with how twill loads the jars and that's the reason why janino can't find the class - since it works running it directly from command line - but I don't know where to look anymore. Any ideas? Thanks for your help The stacktrace from the exception is here: https://gist.github.com/pgrm/d29d33f356601b1a1a3f The class definition which can't be compiled alone is here: https://gist.github.com/pgrm/a589edb64317f5cb1207 And my Twill-Application definition can be found here: https://github.com/pgrm/incubator-drill/blob/YARN-support/yarn-integration/src/main/java/org/apache/drill/yarn/integration/RunDrill.java And here also one more the the exception trace: 2014-06-09T18:30:40,093Z ERROR o.a.d.e.p.i.p.ProjectRecordBatch [zk1] [37daf04b-7d82-4d2f-987c-59851f2aeafe:frag:0:0] AbstractSingleRecordBatch:next(AbstractSingleRecordBatch.java:60) - Failure during query org.apache.drill.exec.exception.SchemaChangeException: Failure while attempting to load generated class at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:243) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.drill.exec.exception.ClassTransformationException: Failure generating transformation classes for value: package org.apache.drill.exec.test.generated; import org.apache.drill.exec.exception.SchemaChangeException; import org.apache.drill.exec.expr.holders.BitHolder; import org.apache.drill.exec.expr.holders.VarCharHolder; import org.apache.drill.exec.ops.FragmentContext; import org.apache.drill.exec.record.RecordBatch; import org.apache.drill.exec.vector.RepeatedVarCharVector; import org.apache.drill.exec.vector.VarCharVector; import org.apache.drill.exec.vector.complex.impl.RepeatedVarCharReaderImpl; public class ProjectorGen0 { RepeatedVarCharVector vv0; RepeatedVarCharReaderImpl reader4; VarCharVector vv5; public boolean doEval(int inIndex, int outIndex) throws SchemaChangeException { { VarCharHolder out3 = new VarCharHolder(); complex: vv0 .getAccessor().getReader().setPosition((inIndex)); reader4 .read(0, out3); BitHolder out8 = new BitHolder(); out8 .value = 1; if (!vv5 .getMutator().setSafe((outIndex), out3)) { out8 .value = 0; } if (out8 .value == 0) { return false; } } { return true; } } public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing) throws SchemaChangeException { { int[] fieldIds1 = new int[ 1 ] ; fieldIds1 [ 0 ] = 0; Object tmp2 = (incoming).getValueAccessorById(RepeatedVarCharVector.class, fieldIds1).getValueVector(); if (tmp2 == null) { throw new SchemaChangeException("Failure while loading vector vv0 with id: org.apache.drill.exec.record.TypedFieldId@1cf4a5a0."); } vv0 = ((RepeatedVarCharVector) tmp2); reader4 = ((RepeatedVarCharReaderImpl) vv0 .getAccessor().getReader()); int[] fieldIds6 = new int[ 1 ] ; fieldIds6 [ 0 ] = 0; Object tmp7 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds6).getValueVector(); if (tmp7 == null) { throw new SchemaChangeException("Failure while loading vector vv5 with id: org.apache.drill.exec.record.TypedFieldId@1ce776c0."); } vv5 = ((VarCharVector) tmp7); } } } at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:302) at org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.codehaus.commons.compiler.CompileException: Line 4, Column 8: Imported class "org.apache.drill.exec.exception.SchemaChangeException" could not be loaded at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014) at org.codehaus.janino.UnitCompiler.import2(UnitCompiler.java:192) at org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:104) at org.codehaus.janino.UnitCompiler$1.visitSingleTypeImportDeclaration(UnitCompiler.java:166) at org.codehaus.janino.Java$CompilationUnit$SingleTypeImportDeclaration.accept(Java.java:171) at org.codehaus.janino.UnitCompiler.<init>(UnitCompiler.java:164) at org.apache.drill.exec.compile.JaninoClassCompiler.getClassByteCode(JaninoClassCompiler.java:53) at org.apache.drill.exec.compile.QueryClassLoader.getClassByteCode(QueryClassLoader.java:69) at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:256) at org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-06-09T18:30:40,109Z ERROR o.a.d.e.p.i.ScreenCreator$ScreenRoot [zk1] [37daf04b-7d82-4d2f-987c-59851f2aeafe:frag:0:0] ErrorHelper:logAndConvertError(ErrorHelper.java:60) - Error 5625373a-729d-4f3f-a262-6f46f2f879d9: Screen received stop request sent. org.codehaus.commons.compiler.CompileException: Line 4, Column 8: Imported class "org.apache.drill.exec.exception.SchemaChangeException" could not be loaded at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014) at org.codehaus.janino.UnitCompiler.import2(UnitCompiler.java:192) at org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:104) at org.codehaus.janino.UnitCompiler$1.visitSingleTypeImportDeclaration(UnitCompiler.java:166) at org.codehaus.janino.Java$CompilationUnit$SingleTypeImportDeclaration.accept(Java.java:171) at org.codehaus.janino.UnitCompiler.<init>(UnitCompiler.java:164) at org.apache.drill.exec.compile.JaninoClassCompiler.getClassByteCode(JaninoClassCompiler.java:53) at org.apache.drill.exec.compile.QueryClassLoader.getClassByteCode(QueryClassLoader.java:69) at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:256) at org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)
