On Wed, Jan 30, 2013 at 11:53 AM, Brian O'Neill <[email protected]>wrote:
> ... > How do we intend to distribute the execution engine across a set of > machines? > There are a variety of thoughts. These include: - custom built execution controller similar to Storm's Nimbus - use Storm's Nimbus - use the custom built controller via Yarn. Or Mesos. Or the MapR warden - start them by hand. Obviously the last option will be the one that is used in initial testing. Any thought to deploying the engine as a Storm topology? > Using Storm probably limits the performance that we can get. Storm's performance is creditable but not super awesomely impressive. Some of the performance issues with Storm include: - limited to Java. This may or may not make a difference in the end in terms of performance, but we definitely want flexibility here. Java can be awesomely fast (see LMAX and Disruptor), but C++ may be more predictable. We definitely *don't* want to decide for all time right now which option we take and we definitely *do* want to have the C++ option in our hip-pocket later regardless of how we build execution engines now. Part of Storm's limitations here have to do with the use of Kryo instead of a portable serializer like protobufs. - the designs I have seen or heard batting around tend to deal with batches of records represented in an ephemeral column oriented design. It will also be important for records to be kept in unmaterialized, virtual form to minimize copying, especially when flattening arrays and such. Storm allows tuples to be kept in memory when bolts are on the same machine, but insists on serializing and deserializing them at the frontier. To control this, we would have to nest serializations which seems a bit like incipient insanity. Other issues include: - Drill execution engines will need access to a considerable amount of out-of-band information such as schemas and statistics. How do we manage that in a restrictive paradigm like Storm - Storm hides location from Bolts. Drill needs to make decisions based on location of execution engines and data.
