[
https://issues.apache.org/jira/browse/DRILL-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941151#comment-15941151
]
ASF GitHub Bot commented on DRILL-5089:
---------------------------------------
GitHub user chunhui-shi opened a pull request:
https://github.com/apache/drill/pull/795
DRILL-5089: Get only partial schemas of relevant storage plugins inst…
…ead of all storages ahead.
1. For each query, rootSchema is empty, add schemas of a storage plugin
only when needed -- when asked for a schemaPath.
2. Allow mock environments to provide dynamic schemas.
3. SchemaUtils.findSchema used in many Sql handlers now is handled by
SqlConverter.catalog to allow expand schema dynamically
4. Temp table resolve also could not take a schema tree and assume it is
complete.
NOTE: pom.xml in this pull request is pointing to temporary calcite versoin
'r20-test', this should change to 'r20' once this pull request is ready to
commit
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chunhui-shi/drill DRILL-5089
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/795.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #795
----
commit 4814e37eefd16ee7aa1ba6d21d66211e41922dbf
Author: chunhui-shi <[email protected]>
Date: 2017-01-06T09:13:02Z
DRILL-5089: Get only partial schemas of relevant storage plugins instead of
all storages ahead.
1. For each query, rootSchema is empty, add schemas of a storage plugin
only when needed -- when asked for a schemaPath.
2. Allow mock environments to provide dynamic schemas.
3. SchemaUtils.findSchema used in many Sql handlers now is handled by
SqlConverter.catalog to allow expand schema dynamically
4. Temp table resolve also could not take a schema tree and assume it is
complete.
NOTE: pom.xml in this pull request is pointing to temporary calcite versoin
'r20-test', this should change to 'r20' once this pull request is ready to
commit
----
> Skip initializing all enabled storage plugins for every query
> -------------------------------------------------------------
>
> Key: DRILL-5089
> URL: https://issues.apache.org/jira/browse/DRILL-5089
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
> Affects Versions: 1.9.0
> Reporter: Abhishek Girish
> Assignee: Chunhui Shi
> Priority: Critical
>
> In a query's lifecycle, at attempt is made to initialize each enabled storage
> plugin, while building the schema tree. This is done regardless of the actual
> plugins involved within a query.
> Sometimes, when one or more of the enabled storage plugins have issues -
> either due to misconfiguration or the underlying datasource being slow or
> being down, the overall query time taken increases drastically. Most likely
> due the attempt being made to register schemas from a faulty plugin.
> For example, when a jdbc plugin is configured with SQL Server, and at one
> point the underlying SQL Server db goes down, any Drill query starting to
> execute at that point and beyond begin to slow down drastically.
> We must skip registering unrelated schemas (& workspaces) for a query.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
