>I'm working on a columnar data store that supports hierarchical schemas
FYI Apache Drill is a query engine built on top of Calcite that supports hierarchical data and columnar storage using Parquet. Depending on where your development is, you may want to take a look at it. On Fri, Dec 12, 2014 at 12:46 PM, Hartman, Trevor <[email protected]> wrote: > I'm working on a columnar data store that supports hierarchical schemas. > It's not open source (yet), so I've created a demo repo that mirrors > exactly what I'm trying to do. > > In MapProjectRel, I get some data from `getNamedProjects` and store it on > the implementor [1]. This data structure (Seq[String]) is not sufficient to > capture nested projections, but I haven't fixed that yet because my main > problem is the linq4j translation like I mentioned, which I documented in > MapToEnumerableConverter [2]. Even getting the raw List[RexNode] from > `getProjections` would have the data required for my MapEnumerator to get > the right fields while it scans the dataset. > > I don't know whether it makes more sense to use the Enumerable or > Interpreter calling convention. Performance does matter here as this will > hopefully become the primary way in which live apps will get data out of > the system. Enumerable sounds nice in that I can over time pushdown each > piece as I get to it, and still have Calcite provide the other operation > in the meantime. Would Interpreter be able to do the same? > > What I've been trying to do is get all the necessary data structures > containing projections, sorting and aggregation info into my MapEnumerator, > because that's where the query interface to the underlying database lives > in my actual project. > > Thanks, > Trevor > > [1] > https://github.com/devth/calcite-map-demo/blob/master/src%2Fmain%2Fscala%2Fdevth%2Fcalcite%2FMapProjectRel.scala#L36 > > [2] > https://github.com/devth/calcite-map-demo/blob/master/src%2Fmain%2Fscala%2Fdevth%2Fcalcite%2FMapToEnumerableConverter.scala#L45-L55 >
