Il mer 22 nov 2017, 22:28 Iñigo Mediavilla <[email protected]> ha scritto:
> Hello, > > I'm using Calcite to provide an SQL interface (Read-Only) for a Java > Service that contains its data in memory. However most of the fields are > primitive types and exposing them in a ProjectableFilterableTable forces > boxing of the fields values since the scan method returns > Enumerable<Object[]>. > I see another issue, maybe from a different perspective. Having to deal with Object[] means that you always have to unpack your record to have this particular representation. For instance in my system I have a page of data loaded in memory and it contains a bunch of records/tuples. If I want to feed Calcite I always have to unpack each record to Object[], and as there is a projection to apply very often a new Object[] is to be created. This is not efficient and not GC friendly. A great enhancement maybe it would be to have some interface instead of the bare array. This will enable the underlying implementation not to deserialize cells Object[] As far as my limited experience of Calcite suggests me this change will be very difficult to introduce as it will break binary compatibility, as most of the Enumerable stuff is done with reflection. Just my 2 cents Enrico > > > > > When I look at how ProjectableFilterableTable is used, it seems that for > every Object array calcite is generating a Row object that relies on an > array of Object internally. > > I was wondering if an UnsafeRow similar to what Spark has implemented could > be considered for Calcite given the possible savings that it could bring in > terms of memory and how it could in some cases like mine help avoiding > unnecessary boxing / unboxing. > > > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java > > Kind regards, > > Inigo Mediavilla > -- -- Enrico Olivelli
