Sorry, I misinterpreted your last message. As for Dremio/Drill, the model offered by CalciteSchema is not scalable as it requires all subschemas and tables to be loaded, even if only one table is accessed: fetching all metadata might be an expensive operation depending on the type and number of sources, especially compared to the previous model where the validator would not try to resolve a table by querying all subschemas in the path vs directly asking the catalog reader for it.
I'm trying to see if it is possible to figure out a model where both approaches can be satisfied. Laurent On Mon, Apr 9, 2018 at 11:07 PM, Laurent Goujon <laur...@dremio.com> wrote: > That was my understanding, but it seems usage of CalciteSchema is > prevalent outside the scope of the JDBC adapter: > > $ grep -rl import.*CalciteSchema core/src/main/java|grep -v calcite/jdbc > core/src/main/java/org/apache/calcite/materialize/Lattice.java > core/src/main/java/org/apache/calcite/materialize/ > MaterializationActor.java > core/src/main/java/org/apache/calcite/materialize/ > MaterializationService.java > core/src/main/java/org/apache/calcite/model/ModelHandler.java > core/src/main/java/org/apache/calcite/plan/RelOptLattice.java > core/src/main/java/org/apache/calcite/prepare/CalciteCatalogReader.java > core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java > core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java > core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java > core/src/main/java/org/apache/calcite/prepare/Prepare.java > core/src/main/java/org/apache/calcite/prepare/QueryableRelBuilder.java > core/src/main/java/org/apache/calcite/prepare/RelOptTableImpl.java > core/src/main/java/org/apache/calcite/rel/rules/ > AggregateStarTableRule.java > core/src/main/java/org/apache/calcite/schema/impl/ > MaterializedViewTable.java > core/src/main/java/org/apache/calcite/schema/impl/ViewTable.java > core/src/main/java/org/apache/calcite/schema/impl/ViewTableMacro.java > core/src/main/java/org/apache/calcite/schema/Schemas.java > core/src/main/java/org/apache/calcite/sql/validate/EmptyScope.java > core/src/main/java/org/apache/calcite/sql/validate/ > SqlValidatorCatalogReader.java > core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java > core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorUtil.java > core/src/main/java/org/apache/calcite/tools/Frameworks.java > > So should we go the opposite way and remove usage of CalciteSchema in > calcite core, outside of the JDBC adapter? It looks like some interfaces > would have to be extended to replace its use. > > On Mon, Apr 9, 2018 at 3:23 PM, Julian Hyde <jh...@apache.org> wrote: > >> The policy that CalciteSchema should be private to Calcite hasn’t changed >> (it is a public interface only because… Java). >> >> It has never been private to the JDBC adapter. (The JDBC adapter readers >> from JDBC sources and is in org.apache.calcite.adapter.jdbc, whereas >> CalciteSchema is in org.apache.calcite.jdbc, which makes Calcite look like >> a JDBC database.) >> >> CalciteSchema is an abstract class. There are two concrete >> implementations: CachingCalciteSchema which remembers what it has read from >> the underlying source, and SimpleCalciteSchema that caches nothing. I know >> Drill (and I guess now Dremio) has a different policy, so how about adding >> an implementation of CalciteSchema that has that policy? I have been asking >> for this for a long time. >> >> If we did what you suggest, and make CalciteSchema (or a base interface >> of it) a public API then people will go and create implementations of that >> interface that will get broken all the time. >> >> Julian >> >> >> > On Apr 3, 2018, at 4:10 PM, Laurent Goujon <laur...@dremio.com> wrote: >> > >> > Hi, >> > >> > While working on rebasing our code on top of Calcite 1.16, I noticed >> that a >> > new method was introduced to SqlValidatorCatalogReader: CalciteSchema >> > `SqlValidatorCatalogReader#getRootSchema()` (not recently, it was >> added for >> > Calcite 1.12) >> > >> > My understanding of CalciteSchema was that this class is private and >> > specific to the JDBC adapter, but looking at the code base, I see it >> used >> > in lots of various places. Has the policy changed? if so, would it make >> > sense to change it to an interface or to relax the class? The current >> model >> > where all informations are stored in maps doesn't match our own model of >> > navigating our catalog. >> > >> > Cheers, >> > >> > Laurent >> >> >
