Yeah, my test was entirely focused on compile time. In the case of simple queries, the compile time differences would be important. In larger queries, this would be less of an issue.
j On Mon, May 20, 2013 at 8:56 AM, Lisen Mu <[email protected]> wrote: > Jacques, > > I've tried Janino now, with your > org.apache.drill.exec.compile.TestClassCompilationTypes, > > One thing is that the compilation time dominates the benchmark. If we > compile once and benchmark evaluate() only, the performance of class > compiled by JDK compiler is comparable with the class compiled by Janino's > embed compiler. JDK compiler seems to invoke javac which is more > heavyweight than Janino. > > Another thing is that evalute() of ExpressionEvaluator by both Janino and > JDK implementation need to do reflect invocation once. I assume that in the > future utilization in drill, we would have several limited type of > evaluate() method signature; so by using SimpleCompiler to compile the > expression directly into a subclass of our Evaluator interface, we can > fully avoid reflection. > > Here is my benchmark result: > > Janino: 882 micros. JDK: 1641 micros. Direct: 41 micros. > > Please see the > https://github.com/immars/incubator-drill/commit/34c0c3db7287904dbfdef9751e41e252b264dc88 > > > > > On Sun, Apr 28, 2013 at 10:25 AM, Ted Dunning <[email protected]> wrote: > >> Is the actual execution time after warming up the JIT the same? >> >> >> On Fri, Apr 26, 2013 at 8:16 PM, Jacques Nadeau <[email protected]> >> wrote: >> >> > I'm working on some of the bytecode stuff for Drill and found some >> > interesting random testing data: >> > >> > Java6 adds a new JavaCompiler API. Janino supports utilizing that >> > instead of its internal parsing. The benefit of this is that all the >> > JDK 5 stuff like generics are fully supported. Since that seemed >> > nice, i figured we could use that. >> > >> > In initial testing, I've actually found it substantially slower than >> > the Janino compiler. I ran some crappy microbenchmarks just to see >> > why my stuff was going so slow. Running a basic comparison test of >> > the simple ExpressionEvaluator example utilizing Janino versus JDK >> > showed an average compile and single invocation time of ~18000 micros >> > on JDK versus ~950 micros on Janino. >> > >> > Has anyone else experimented with this/experienced this? >> > >> > Example block below (straight from Janino docs). >> > >> > ExpressionEvaluator ee = new ExpressionEvaluator("c > d ? c : d", >> > // expression >> > int.class, // expressionType >> > new String[] { "c", "d" }, // parameterNames >> > new Class[] { int.class, int.class } // parameterTypes >> > ); >> > >> > // Evaluate it with varying parameter values; very fast. >> > return (Integer) ee.evaluate(new Object[] { // parameterValues >> > new Integer(10), new Integer(11), }); >> > >>
