[
https://issues.apache.org/jira/browse/DRILL-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5376:
-------------------------------
Priority: Minor (was: Major)
Issue Type: Improvement (was: Bug)
> Rationalize Drill's row metadata for simpler code, better performance
> ---------------------------------------------------------------------
>
> Key: DRILL-5376
> URL: https://issues.apache.org/jira/browse/DRILL-5376
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.10.0
> Reporter: Paul Rogers
> Priority: Minor
>
> Drill is a columnar system, but data is ultimately represented as rows (AKA
> records or tuples.) The way that Drill represents rows leads to excessive
> code complexity and runtime cost.
> Data in Drill is stored in vectors: one (or more) per column. Vectors do not
> stand alone, however, they are "bundled" into various forms of grouping: the
> {{VectorContainer}}, {{RecordBatch}}, {{VectorAccessible}},
> {{VectorAccessibleSerializable}}, and more. Each has slightly different
> semantics, requiring large amounts of code to bridge between the
> representations.
> Consider only a simple row: one with only scalar columns. In classic
> relational theory, such a row is a tuple:
> {code}
> R = (a, b, c, d, ...)
> {code}
> A tuple is defined as an ordered list of column values. Unlike a list or
> array, the column values also have names and may have varying data types.
> In SQL, columns are referenced by either position or name. In most execution
> engines, columns are referenced by position (since positions, in most
> systems, cannot change.) A 1:1 mapping is provided between names and
> positions. (See the JDBC {{RecordSet}} interface.)
> This allows code to be very fast: code references columns by index, not by
> name, avoiding name lookups for each column reference.
> Drill provides a murky, hybrid approach. Some structures ({{BatchSchema}},
> for example) appear to provide a fixed column ordering, allowing indexed
> column access. But, other abstractions provide only an iterator. Others (such
> as {{VectorContainer}}) provides name-based access or, by clever programming,
> indexed access.
> As a result, it is never clear exactly how to quickly access a column: by
> name, by name to multi-part index to vector?
> Of course, Drill also supports maps, which add to the complexity. First, we
> must understand that a "map" in Drill is not a "map" in the classic sense: it
> is not a collection of (name, value) pairs in the JSON sense: a collection in
> which each instance may have a different set of pairs.
> Instead, in Drill, a "map" is really a nested tuple: a map has the same
> structure as a Drill record: a collection of names and values in which all
> rows have the same structure. (This is so because maps are really a
> collection of value vectors, and the vectors cut across all rows.)
> Drill, however, does not reflect this symmetry: that a row and a map are both
> tuples. There are no common abstractions for the two. Instead, maps are
> represented as a {{MapVector}} that contains a (name, vector) map for its
> children.
> Because of this name-based mapping, high-speed indexed access to vectors is
> not provided "out of the box." Certainly each consumer of a map can build its
> own indexing mechanism. But, this leads to code complexity and redundancy.
> This ticket asks to rationalize Drill's row, map and schema abstractions
> around the tuple concept. A schema is a description of a tuple and should (as
> in JDBC) provide both name and index based access. That is, provide methods
> of the form:
> {code}
> MaterializedField getField(int index);
> MaterializedField getField(String name);
> ...
> ValueVector getVector(int index);
> ValueVector getVector(String name);
> {code}
> Provide a common abstraction for rows and maps, recognizing their structural
> similarity.
> There is an obvious issue with indexing columns in a row when the row
> contains maps. Should indexing be multi-part (index into row, then into map)
> as today? A better alternative is to provide a flattened interface:
> {code}
> 0: a, 1: b.x, 2: b.y, 3: c, ...
> {code}
> Use this change to simplify client code, over time, to use a simple
> indexed-based column access.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
