Hi Qiaoyi, In general, optimal performance occurs when a system knows the schema at the start and can fully optimize based on that schema. Think C or C++ compilers compared with Java or Python.
On the other hand, the JVM HotSpot optimizer has shown that one can achieve very good performance via incremental optimization, but at the cost of extreme runtime complexity. The benefit is that Java is much more flexible, machine independent, and simpler than C or C++ (at least for non-system applications.) Python is the other extreme: it is so dynamic that the literature has shown that it is very difficult to optimize Python at the compiler or runtime level. (Though, there is some interesting research on this topic. See [1], [2].) Drill is somewhere in the middle. Drill does not do code generation at the start like Impala or Spark do. Nor is it fully interpreted. Rather, Drill is roughly like Java: code generation is done at runtime based on the observed data types. (The JVM does machine code generation based on observed execution patterns.) The advantage is that Drill is able to achieve its pretty-good performance without the cost of a metadata system to provide schema at plan time. So, to get the absolute fastest performance (think Impala), you must pay the cost of a metadata system for all queries. Drill gets nearly as good performance without the complexity of the Hive metastore -- a pretty good tradeoff. If I may ask, what is your area of interest? Are you looking to use Drill for some project? Or, just interested in how Drill works? Thanks, - Paul [1] Towards Practical Gradual Typing: https://blog.acolyer.org/2015/08/03/towards-practical-gradual-typing/ [2] Is Sound Gradual Typing Dead? https://blog.acolyer.org/2016/02/05/is-sound-gradual-typing-dead/ On Sunday, August 5, 2018, 11:36:52 PM PDT, 丁乔毅(智乔) <[email protected]> wrote: Thanks Paul, good to know the design principals of the Drill query execution process model. I am very new to Drill, please bear with me. One more question. As you mentioned, the schema-free processing is the key feature to be advantage over Spark, is there any performance consideration behind this design except the techniques of the dynamic codegen and vectorization computation? Regards, Qiaoyi ------------------------------------------------------------------ 发件人：Paul Rogers <[email protected]> 发送时间：2018年8月4日(星期六) 02:27 收件人：dev <[email protected]> 主 题：Re: Is Drill query execution processing model just the same idea with the Spark whole-stage codegen improvement Hi Qiaoyi, As you noted, Drill and Spark have similar models -- but with important differences. Drill is schema-on-read (also called "schema less"). In particular, this means that Drill does not know the schema of the data until the first row (actually "record batch") arrives at each operator. Once Drill sees that first batch, it has a data schema, and can generate the corresponding code; but only for that one operator. The above process repeats up the fragment ("fragment" is Drill's term for a Spark stage.) I believe that Spark requires (or at least allows) the user to define a schema up front. This is particularly true for the more modern data frame APIs. Do you think the Spark improvement would apply to Drill's case of determining the schema operator-by-opeartor up the DAG? Thanks, - Paul On Friday, August 3, 2018, 8:57:29 AM PDT, 丁乔毅(智乔) <[email protected]> wrote: Hi, all. I'm very new to Apache Drill. I'm quite interest in Drill query execution's implementation. After a little bit of source code reading, I found it is built on a processing model quite like a data-centric pushed-based style, which is very similar with the idea behind the Spark whole-stage codegen improvement(jira ticket https://issues.apache.org/jira/browse/SPARK-12795) And I wonder is there any detailed documentation about this? What's the consideration behind of our design in the Drill project. : ) Regards, Qiaoyi
