Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/701
@paul-rogers as we discussed to have tried to find the way to preserve
lazy-init approach.
I have renamed PR to reflect latest changes. Please find new solution and
description of changes below:
Lazy-init was performed only when function was not found during Calcite
parsing but DRILL-4963 shows different cases when Calcite parsing can pass
(usually during function overloading) but still function is not found. To
handle such cases, we need to enhance lazy-init process:
1. Lazy-init process should be more light-weight. Currently when function
is not found, we load all jars from remote function registry and compare with
jars from local function registry. It's not optimal especially when both
registries are in sync. To improve performance I have introduced remove
function registry version which can be used to check if we need to sync remote
and local registries prior to checking jars and functions.
2. During parsing stage we were only catching Calcite parsing exception but
function not found can be also indicated by Drill function error and so on. Not
to be engaged into enumerating possible exceptions (which can added in the code
later), we are checking if remote and local function registries are in sync on
any error. Such check may only affect on queries that will fail anyway. So if
failure time will take a little bit longer, it won't make significant
difference.
3. During execution stage Drill attempts to find matching function from the
list of functions with the same names. `DefaultFunctionResolver.getBestMatch()`
does not do exact match, it may return function with different input parameters
types. Best match is found according to rules described in
`TypeCastRules.class`. Currently we attempt to sync remote and local function
registries only if best matching function was found but it is not correct since
even if Drill finds the best matching function among current functions but does
not mean that remote function registry does not hold even better matching
function. To fix this issue we would first try to find function using
`ExactFunctionResolver.getBestMatch()` and if exactly matching function is not
found, we'll check if remote and local function registries are in sync and then
use `DefaultFunctionResolver.getBestMatch()` to find the best matching
function. But if exactly matching function is found, we'll return it right away
without
any registries sync checks.
Changes:
1. Add `consists` method to PersistentStore interface which can return true
if key exists in store, false otherwise. This method is needed to return only
remote function registry version without its content (unlike method `get`).
We'll pull remote function registry content only if versions are different.
2. Added check if remote and local function registries are in sync on any
failure during planning stage
and on exact matching function not found during execution stage.
3. Added additional debug messages for `CreateFunctionHandler` and
`DropFunctionHandler`.
4. Updated unit tests to reflect new changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
