[
https://issues.apache.org/jira/browse/CALCITE-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977322#comment-15977322
]
Jinfeng Ni commented on CALCITE-1748:
-------------------------------------
Thanks for the very helpful comments, [~maryannxue]!
[~cshi] and I discussed yesterday. We agreed with what [~julianhyde] suggested.
We do not have to change {{CalciteCatalogReader}}. Rather, we would leverage
the {{Schema}} SPI, to dynamically load both subschemas and tables.
Today, same as Phoenix, Drill uses {{SimpleCalciteSchema}}, which wraps a
{{Schema}} instance. All the tables are processed as implicit objects, and are
dynamically loaded through {{Schema.getTable()}} call. However, the subschemas
under root schema are not processed as implicit objects, and are pre-loaded as
explicit objects for all the storage plugins. We should apply the way how
tables are loaded to subschemas, by leveraging {{Schema.getSubSchema()}} call.
[~cshi] will try this idea.
Drill builds a schema tree for each query statements. That's probably one
reason we did not run into issue as CALCITE-1742. On the other hand, it also
makes worse if we do not dynamically load the subschemas.
Your proposed solution regarding explicit objects makes sense to me. I'm not
clear about the idea to use map in {{CalciteSchema}} for implicit objects. The
implicit objects are provided / maintained by the underlying {{Schema}}
instance. I thought it's the underlying {{Schema}} instance's responsibility to
decide whether an implicit object is updated/deleted, rather than
{{CalciteSchema}}.
> Make CalciteCatalogReader.getSchema extendable to support dynamically load
> schema tree - getSchema need to be set to protected to allow overriding
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CALCITE-1748
> URL: https://issues.apache.org/jira/browse/CALCITE-1748
> Project: Calcite
> Issue Type: Bug
> Reporter: Chunhui Shi
> Assignee: Julian Hyde
>
> In system like Drill, there is a need to load partial schema (e.g. for only
> one storage plugin) only when needed. Since Drill has no way to get a full
> available schema tree before hand, nor could Drill cache available schema for
> a storage plugin(e.g. Hive, MongoDB) since the storage plugin may not have
> notification mechanism to update Schema tree timely.
>
> The proposed fix is to load schema dynamically as shown in
> https://issues.apache.org/jira/browse/DRILL-5089
> To achieve this, we need to make CalciteCatalogReader.getSchema to be
> protected so it could be overridden by derived class while the derived class
> can reuse other functionalities in CalciteCatalogReader class
> private CalciteSchema getSchema(Iterable<String> schemaNames,
> SqlNameMatcher nameMatcher)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
