kramerul opened a new pull request, #4100:
URL: https://github.com/apache/calcite/pull/4100
### Motivation
For databases with a huge set of schemas and tables it takes quite long to
prepare queries. Currently all tables/schemas are loaded into memory.
Caching all these schemas and tables is not an option
1. It will require a lot of memory
2. The eviction of the cache must happen quite often since it's likely that
every second one of these table is changed.
Therefore, we tried to find a way to load only those tables/schemas, which
are required to prepare a query.
### API Changes
This PR introduces a new mechanism to lookup tables and schemas within a
schema. For this purpose a new interface is introduced
```java
public interface Lookup<T> {
@Nullable T get(String name);
@Nullable Named<T> getIgnoreCase(String name);
Set<String> getNames(LikePattern pattern);
}
```
The `LikePattern` was extracted from `CalciteMetaImpl` to hold a pattern,
which can be used to query tables and schemas inside a JDBC database using the
`LIKE` operator. Additionally, it also supports the conversion to a
`Predicate1<String>` which can be used to implement filters in plain java.
The `Schema` is now using this `Lookup` interface to find schemas and
tables. It could be also extended to functions and types.
```java
public interface Schema {
default Lookup<Table> tables() {
...
}
default Lookup<? extends Schema> subSchemas() {
...
}
...
}
```
### Implementation
The case insensitive search is now directly implemented in the specific
`Schema` using matching implementation of the `Lookup` interface. Formerly, it
was done in the `CalciteSchema`.
`Jdbc..Schema` is using a special implementation of `Lookup`:
`LoadingCacheLookup`. This implementation is using a `LoadingCache` inside to
speed up things. If only case sensitive schema/table lookup is required, this
can be done quite fast since `DatabaseMetaData#getTables` can be used to query
a single table. The result is cached inside the `LoadingCache` for one minute.
Unfortunately `DatabaseMetaData#getTables` doesn't support case insensitive
queries. In this case, it's still required to load all database tables to
perform case insensitive lookups.
The performance gain for huge sets of tables/schemas in database schemas can
only be achieved if caching is turned off in Calcite (`SimpleCalciteSchema` is
used instead of `CachingCalciteSchema`).
I tried to keep the behavior of `CachingCalciteSchema` exactly the same.
This behavior includes that all tables/schemas are loaded into memory.
`CachedLookup` is used to achieve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
