[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524292#comment-15524292
]
ASF GitHub Bot commented on DRILL-4726:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r80559233
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
---
@@ -0,0 +1,360 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p/>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p/>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.registry;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Queues;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * Function registry holder stores function implementations by jar name,
function name.
+ * Contains two maps that hold data by jars and functions respectively.
+ * Jars map contains each jar as a key and map of all its functions with
collection of function signatures as value.
+ * Functions map contains function name as key and map of its signatures
and function holder as value.
+ * All maps and collections used are concurrent to guarantee memory
consistency effects.
+ * Such structure is chosen to achieve maximum speed while retrieving data
by jar or by function name,
+ * since we expect infrequent registry changes.
+ * Holder is designed to allow concurrent reads and single writes to keep
data consistent.
+ * This is achieved by {@link ReadWriteLock} implementation usage.
+ * Holder has number version which changes every time new jars are added
or removed. Initial version number is 0.
+ * Also version is used when user needs data from registry with version it
is based on.
+ *
+ * Structure example:
+ *
+ * JARS
+ * built-in -> upper -> upper(VARCHAR-REQUIRED)
+ * -> lower -> lower(VARCHAR-REQUIRED)
+ *
+ * First.jar -> upper -> upper(VARCHAR-OPTIONAL)
+ * -> custom_upper -> custom_upper(VARCHAR-REQUIRED)
+ * -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * Second.jar -> lower -> lower(VARCHAR-OPTIONAL)
+ * -> custom_upper -> custom_upper(VARCHAR-REQUIRED)
+ * -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * FUNCTIONS
+ * upper -> upper(VARCHAR-REQUIRED) -> function holder for
upper(VARCHAR-REQUIRED)
+ * -> upper(VARCHAR-OPTIONAL) -> function holder for
upper(VARCHAR-OPTIONAL)
+ *
+ * lower -> lower(VARCHAR-REQUIRED) -> function holder for
lower(VARCHAR-REQUIRED)
+ * -> lower(VARCHAR-OPTIONAL) -> function holder for
lower(VARCHAR-OPTIONAL)
+ *
+ * custom_upper -> custom_upper(VARCHAR-REQUIRED) -> function holder for
custom_upper(VARCHAR-REQUIRED)
+ * -> custom_upper(VARCHAR-OPTIONAL) -> function holder for
custom_upper(VARCHAR-OPTIONAL)
+ *
+ * custom_lower -> custom_lower(VARCHAR-REQUIRED) -> function holder for
custom_lower(VARCHAR-REQUIRED)
+ * -> custom_lower(VARCHAR-OPTIONAL) -> function holder for
custom_lower(VARCHAR-OPTIONAL)
+ *
+ * where
+ * First.jar is jar name represented by String
+ * upper is function name represented by String
+ * upper(VARCHAR-REQUIRED) is signature name represented by String which
consist of function name, list of input parameters
+ * function holder for upper(VARCHAR-REQUIRED) is {@link DrillFuncHolder}
initiated for each function.
+ *
+ */
+public class FunctionRegistryHolder {
+
+ private final ReadWriteLock readWriteLock = new ReentrantReadWriteLock();
+ private final AutoCloseableLock readLock = new
AutoCloseableLock(readWriteLock.readLock());
+ private final AutoCloseableLock writeLock = new
AutoCloseableLock(readWriteLock.writeLock());
+ private final AtomicLong version = new AtomicLong();
+
+ // jar name, Map<function name, Queue<function signature>
+ private final Map<String, Map<String, Queue<String>>> jars;
+
+ // function name, Map<function signature, function holder>
+ private final Map<String, Map<String, DrillFuncHolder>> functions;
+
+ public FunctionRegistryHolder() {
+ this.functions = Maps.newConcurrentMap();
+ this.jars = Maps.newConcurrentMap();
+ }
+
+ /**
+ * This is read operation, so several users at a time can get this data.
+ * @return local function registry version number
+ */
+ public long getVersion() {
+ try (AutoCloseableLock lock = readLock.open()) {
+ return version.get();
+ }
+ }
+
+ /**
+ * Adds jars to the function registry.
+ * If jar with the same name already exists, it and its functions will
be removed.
+ * Then jar will be added to {@link #jars}
+ * and each function will be added using {@link #addFunctions(Map,
List)}.
+ * Function version registry will be incremented by 1 if at least one
jar was added but not for each jar.
+ * This is write operation, so one user at a time can call perform such
action,
+ * others will wait till first user completes his action.
+ *
+ * @param newJars jars and list of their function holders, each contains
function name, signature and holder
+ */
+ public void addJars(Map<String, List<FunctionHolder>> newJars) {
+ try (AutoCloseableLock lock = writeLock.open()) {
+ for (Map.Entry<String, List<FunctionHolder>> newJar :
newJars.entrySet()) {
+ String jarName = newJar.getKey();
+ removeAllByJar(jarName);
+ Map<String, Queue<String>> jar = Maps.newConcurrentMap();
+ jars.put(jarName, jar);
+ addFunctions(jar, newJar.getValue());
+ }
+ if (!newJars.isEmpty()) {
+ version.incrementAndGet();
+ }
+ }
+ }
+
+ /**
+ * Removes jar from {@link #jars} and all associated with jar functions
from {@link #functions}
+ * If jar was removed, function registry version will be incremented by
1.
+ * This is write operation, so one user at a time can call perform such
action,
+ * others will wait till first user completes his action.
+ *
+ * @param jarName jar name to be removed
+ */
+ public void removeJar(String jarName) {
+ try (AutoCloseableLock lock = writeLock.open()) {
+ if (removeAllByJar(jarName)) {
+ version.incrementAndGet();
+ }
+ }
+ }
+
+ /**
+ * Retrieves list of all jars name present in {@link #jars}
+ * This is read operation, so several users can get this data.
+ *
+ * @return list of all jar names
+ */
+ public List<String> getAllJarNames() {
+ try (AutoCloseableLock lock = readLock.open()) {
+ return Lists.newArrayList(jars.keySet());
+ }
+ }
+
+ /**
+ * Retrieves all function names associated with the jar from {@link
#jars}.
+ * Returns empty list if jar is not registered.
+ * This is read operation, so several users can perform this operation
at the same time.
+ *
+ * @param jarName jar name
+ * @return list of functions names associated from the jar
+ */
+ public List<String> getFunctionNamesByJar(String jarName) {
--- End diff --
The list of function names is lock-protected. Good. Let's consider the use
of the list. How will the list be used?
Suppose the caller gets the list. The caller finds a function in the list.
The caller then tries to do something with the function. Could it be the case
that, since retrieving the list, the function was removed from the registry?
What happens?
If we use this list to parse a SQL statement (and resolve functions), how
do we prevent the function from being unregistered before the statement
executes?
Or, is this list used just for non-critical purposes, such as logging?
> Dynamic UDFs support
> --------------------
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
> Affects Versions: 1.6.0
> Reporter: Arina Ielchiieva
> Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
