Re: Use of CalciteSchema across Calcite

[Laurent Goujon]

Sorry, I misinterpreted your last message. As for Dremio/Drill, the model
offered by CalciteSchema is not scalable as it requires all subschemas and
tables to be loaded, even if only one table is accessed: fetching all
metadata might be an expensive operation depending on the type and number
of sources, especially compared to the previous model where the validator
would not try to resolve a table by querying all subschemas in the path vs
directly asking the catalog reader for it.

I'm trying to see if it is possible to figure out a model where both
approaches can be satisfied.

Laurent

On Mon, Apr 9, 2018 at 11:07 PM, Laurent Goujon  wrote:

> That was my understanding, but it seems usage of CalciteSchema is
> prevalent outside the scope of the JDBC adapter:
>
> $ grep -rl import.*CalciteSchema core/src/main/java|grep -v calcite/jdbc
> core/src/main/java/org/apache/calcite/materialize/Lattice.java
> core/src/main/java/org/apache/calcite/materialize/
> MaterializationActor.java
> core/src/main/java/org/apache/calcite/materialize/
> MaterializationService.java
> core/src/main/java/org/apache/calcite/model/ModelHandler.java
> core/src/main/java/org/apache/calcite/plan/RelOptLattice.java
> core/src/main/java/org/apache/calcite/prepare/CalciteCatalogReader.java
> core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java
> core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java
> core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java
> core/src/main/java/org/apache/calcite/prepare/Prepare.java
> core/src/main/java/org/apache/calcite/prepare/QueryableRelBuilder.java
> core/src/main/java/org/apache/calcite/prepare/RelOptTableImpl.java
> core/src/main/java/org/apache/calcite/rel/rules/
> AggregateStarTableRule.java
> core/src/main/java/org/apache/calcite/schema/impl/
> MaterializedViewTable.java
> core/src/main/java/org/apache/calcite/schema/impl/ViewTable.java
> core/src/main/java/org/apache/calcite/schema/impl/ViewTableMacro.java
> core/src/main/java/org/apache/calcite/schema/Schemas.java
> core/src/main/java/org/apache/calcite/sql/validate/EmptyScope.java
> core/src/main/java/org/apache/calcite/sql/validate/
> SqlValidatorCatalogReader.java
> core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java
> core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorUtil.java
> core/src/main/java/org/apache/calcite/tools/Frameworks.java
>
> So should we go the opposite way and remove usage of CalciteSchema in
> calcite core, outside of the JDBC adapter? It looks like some interfaces
> would have to be extended to replace its use.
>
> On Mon, Apr 9, 2018 at 3:23 PM, Julian Hyde  wrote:
>
>> The policy that CalciteSchema should be private to Calcite hasn’t changed
>> (it is a public interface only because… Java).
>>
>> It has never been private to the JDBC adapter. (The JDBC adapter readers
>> from JDBC sources and is in org.apache.calcite.adapter.jdbc, whereas
>> CalciteSchema is in org.apache.calcite.jdbc, which makes Calcite look like
>> a JDBC database.)
>>
>> CalciteSchema is an abstract class. There are two concrete
>> implementations: CachingCalciteSchema which remembers what it has read from
>> the underlying source, and SimpleCalciteSchema that caches nothing. I know
>> Drill (and I guess now Dremio) has a different policy, so how about adding
>> an implementation of CalciteSchema that has that policy? I have been asking
>> for this for a long time.
>>
>> If we did what you suggest, and make CalciteSchema (or a base interface
>> of it) a public API then people will go and create implementations of that
>> interface that will get broken all the time.
>>
>> Julian
>>
>>
>> > On Apr 3, 2018, at 4:10 PM, Laurent Goujon  wrote:
>> >
>> > Hi,
>> >
>> > While working on rebasing our code on top of Calcite 1.16, I noticed
>> that a
>> > new method was introduced to SqlValidatorCatalogReader: CalciteSchema
>> > `SqlValidatorCatalogReader#getRootSchema()` (not recently, it was
>> added for
>> > Calcite 1.12)
>> >
>> > My understanding of CalciteSchema was that this class is private and
>> > specific to the JDBC adapter, but looking at the code base, I see it
>> used
>> > in lots of various places. Has the policy changed? if so, would it make
>> > sense to change it to an interface or to relax the class? The current
>> model
>> > where all informations are stored in maps doesn't match our own model of
>> > navigating our catalog.
>> >
>> > Cheers,
>> >
>> > Laurent
>>
>>
>

Re: Use of CalciteSchema across Calcite

[Laurent Goujon]

That was my understanding, but it seems usage of CalciteSchema is prevalent
outside the scope of the JDBC adapter:

$ grep -rl import.*CalciteSchema core/src/main/java|grep -v calcite/jdbc
core/src/main/java/org/apache/calcite/materialize/Lattice.java
core/src/main/java/org/apache/calcite/materialize/MaterializationActor.java
core/src/main/java/org/apache/calcite/materialize/MaterializationService.java
core/src/main/java/org/apache/calcite/model/ModelHandler.java
core/src/main/java/org/apache/calcite/plan/RelOptLattice.java
core/src/main/java/org/apache/calcite/prepare/CalciteCatalogReader.java
core/src/main/java/org/apache/calcite/prepare/CalciteMaterializer.java
core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java
core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java
core/src/main/java/org/apache/calcite/prepare/Prepare.java
core/src/main/java/org/apache/calcite/prepare/QueryableRelBuilder.java
core/src/main/java/org/apache/calcite/prepare/RelOptTableImpl.java
core/src/main/java/org/apache/calcite/rel/rules/AggregateStarTableRule.java
core/src/main/java/org/apache/calcite/schema/impl/MaterializedViewTable.java
core/src/main/java/org/apache/calcite/schema/impl/ViewTable.java
core/src/main/java/org/apache/calcite/schema/impl/ViewTableMacro.java
core/src/main/java/org/apache/calcite/schema/Schemas.java
core/src/main/java/org/apache/calcite/sql/validate/EmptyScope.java
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorCatalogReader.java
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorUtil.java
core/src/main/java/org/apache/calcite/tools/Frameworks.java

So should we go the opposite way and remove usage of CalciteSchema in
calcite core, outside of the JDBC adapter? It looks like some interfaces
would have to be extended to replace its use.

On Mon, Apr 9, 2018 at 3:23 PM, Julian Hyde  wrote:

> The policy that CalciteSchema should be private to Calcite hasn’t changed
> (it is a public interface only because… Java).
>
> It has never been private to the JDBC adapter. (The JDBC adapter readers
> from JDBC sources and is in org.apache.calcite.adapter.jdbc, whereas
> CalciteSchema is in org.apache.calcite.jdbc, which makes Calcite look like
> a JDBC database.)
>
> CalciteSchema is an abstract class. There are two concrete
> implementations: CachingCalciteSchema which remembers what it has read from
> the underlying source, and SimpleCalciteSchema that caches nothing. I know
> Drill (and I guess now Dremio) has a different policy, so how about adding
> an implementation of CalciteSchema that has that policy? I have been asking
> for this for a long time.
>
> If we did what you suggest, and make CalciteSchema (or a base interface of
> it) a public API then people will go and create implementations of that
> interface that will get broken all the time.
>
> Julian
>
>
> > On Apr 3, 2018, at 4:10 PM, Laurent Goujon  wrote:
> >
> > Hi,
> >
> > While working on rebasing our code on top of Calcite 1.16, I noticed
> that a
> > new method was introduced to SqlValidatorCatalogReader: CalciteSchema
> > `SqlValidatorCatalogReader#getRootSchema()` (not recently, it was added
> for
> > Calcite 1.12)
> >
> > My understanding of CalciteSchema was that this class is private and
> > specific to the JDBC adapter, but looking at the code base, I see it used
> > in lots of various places. Has the policy changed? if so, would it make
> > sense to change it to an interface or to relax the class? The current
> model
> > where all informations are stored in maps doesn't match our own model of
> > navigating our catalog.
> >
> > Cheers,
> >
> > Laurent
>
>

Re: Use of CalciteSchema across Calcite

[Julian Hyde]

The policy that CalciteSchema should be private to Calcite hasn’t changed (it 
is a public interface only because… Java). 

It has never been private to the JDBC adapter. (The JDBC adapter readers from 
JDBC sources and is in org.apache.calcite.adapter.jdbc, whereas CalciteSchema 
is in org.apache.calcite.jdbc, which makes Calcite look like a JDBC database.)

CalciteSchema is an abstract class. There are two concrete implementations: 
CachingCalciteSchema which remembers what it has read from the underlying 
source, and SimpleCalciteSchema that caches nothing. I know Drill (and I guess 
now Dremio) has a different policy, so how about adding an implementation of 
CalciteSchema that has that policy? I have been asking for this for a long time.

If we did what you suggest, and make CalciteSchema (or a base interface of it) 
a public API then people will go and create implementations of that interface 
that will get broken all the time.

Julian


> On Apr 3, 2018, at 4:10 PM, Laurent Goujon  wrote:
> 
> Hi,
> 
> While working on rebasing our code on top of Calcite 1.16, I noticed that a
> new method was introduced to SqlValidatorCatalogReader: CalciteSchema
> `SqlValidatorCatalogReader#getRootSchema()` (not recently, it was added for
> Calcite 1.12)
> 
> My understanding of CalciteSchema was that this class is private and
> specific to the JDBC adapter, but looking at the code base, I see it used
> in lots of various places. Has the policy changed? if so, would it make
> sense to change it to an interface or to relax the class? The current model
> where all informations are stored in maps doesn't match our own model of
> navigating our catalog.
> 
> Cheers,
> 
> Laurent