[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact
[
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632769#comment-16632769
]
ASF GitHub Bot commented on DRILL-6246:
---
krystaln commented on issue #1168: DRILL-6246: Reduced the size of the jdbc-all
jar file
URL: https://github.com/apache/drill/pull/1168#issuecomment-425613110
cat git.properties
#Generated by Git-Commit-Id-Plugin
#Fri Sep 28 10:10:32 PDT 2018
git.commit.id.abbrev=527ce72
git.commit.user.email=sachouc...@gmail.com
git.commit.message.full=DRILL-6246\: Reduced the size of the jdbc-all jar
file\n
git.commit.id=527ce7261b6fc6bf374a3eb0d6c40da6dc34a6dd
git.commit.message.short=DRILL-6246\: Reduced the size of the jdbc-all jar
file
git.commit.user.name=Salim Achouche
Tested the drill-jdbc-all-1.14.0-SNAPSHOT.jar from the above branch with
SQuirrel. Tested the following items:
- text files
-parquet files
-json files
-hive
- complex data
-ctas
-create views
-information schema
I did not see any issues.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Labels: pull-request-available
>
> * {color:#00}It was noticed that the build was failing because of the
> jdbc-all artifact{color}
> * {color:#00}The maximum compressed jar size was set to 32MB but we are
> currently creating a JAR a bit larger than 32MB {color}
> * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and
> drill-1.13.0 (on my MacOS){color}
> * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
> * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
> * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is
> roughly 32MB){color}
> * {color:#00}Compared then in more details jdbc-all-1.12.0 and
> jdbc-all-1.13.0{color}
> * {color:#00}The bulk of the increase is attributed to the calcite
> artifact{color}
> * {color:#00}Used to be 2MB (uncompressed) and now 22MB
> (uncompressed){color}
> * {color:#00}It is likely an exclusion problem {color}
> * {color:#00}The jdbc-all-1.12.0 version has only two top packages
> calcite/avatica/utils and calcite/avatica/remote{color}
> * {color:#00}The jdbc-all-1.13.0 includes new packages (within
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6246) Build Failing in jdbc-all artifact
[
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
salim achouche updated DRILL-6246:
--
Labels: pull-request-available (was: )
> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Labels: pull-request-available
>
> * {color:#00}It was noticed that the build was failing because of the
> jdbc-all artifact{color}
> * {color:#00}The maximum compressed jar size was set to 32MB but we are
> currently creating a JAR a bit larger than 32MB {color}
> * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and
> drill-1.13.0 (on my MacOS){color}
> * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
> * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
> * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is
> roughly 32MB){color}
> * {color:#00}Compared then in more details jdbc-all-1.12.0 and
> jdbc-all-1.13.0{color}
> * {color:#00}The bulk of the increase is attributed to the calcite
> artifact{color}
> * {color:#00}Used to be 2MB (uncompressed) and now 22MB
> (uncompressed){color}
> * {color:#00}It is likely an exclusion problem {color}
> * {color:#00}The jdbc-all-1.12.0 version has only two top packages
> calcite/avatica/utils and calcite/avatica/remote{color}
> * {color:#00}The jdbc-all-1.13.0 includes new packages (within
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (DRILL-6246) Build Failing in jdbc-all artifact
[
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
salim achouche reopened DRILL-6246:
---
Re-opening this issue as PR #1168 has been successfully tested and thus should
provide a more optimal solution.
> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the
> jdbc-all artifact{color}
> * {color:#00}The maximum compressed jar size was set to 32MB but we are
> currently creating a JAR a bit larger than 32MB {color}
> * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and
> drill-1.13.0 (on my MacOS){color}
> * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
> * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
> * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is
> roughly 32MB){color}
> * {color:#00}Compared then in more details jdbc-all-1.12.0 and
> jdbc-all-1.13.0{color}
> * {color:#00}The bulk of the increase is attributed to the calcite
> artifact{color}
> * {color:#00}Used to be 2MB (uncompressed) and now 22MB
> (uncompressed){color}
> * {color:#00}It is likely an exclusion problem {color}
> * {color:#00}The jdbc-all-1.12.0 version has only two top packages
> calcite/avatica/utils and calcite/avatica/remote{color}
> * {color:#00}The jdbc-all-1.13.0 includes new packages (within
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6661) Need a configurable parameter to stop Long running queries
[ https://issues.apache.org/jira/browse/DRILL-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632615#comment-16632615 ] Kunal Khatua edited comment on DRILL-6661 at 9/28/18 10:50 PM: --- [~adityaar] there is a JDBC `Statement.setQueryTimeout(int timeInSeconds)` that does this. Any JDBC application should be able to do that. DRILL-3640 introduced this in 1.13.0 was (Author: kkhatua): [~adityaar] there is a JDBC `Statement.setQueryTimeout(int timeInSeconds)` that does this. Any JDBC application should be able to do that. > Need a configurable parameter to stop Long running queries > -- > > Key: DRILL-6661 > URL: https://issues.apache.org/jira/browse/DRILL-6661 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Monitoring >Affects Versions: 1.13.0 >Reporter: Aditya Allamraju >Priority: Major > > I am looking for a way to stop any long running queries that run beyond a > certain time. > This is not to be confused with queue timeout which does not trigger if the > query has started executing. Other database vendors do this via resource > management or a simple timeout. > Currently, the default behavior is to allow any query in execution to > continue till arbitrarily large amount of time until completion. > There is a genuine need for this case. For instance, i want to stop queries > running beyond 15mins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6661) Need a configurable parameter to stop Long running queries
[ https://issues.apache.org/jira/browse/DRILL-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632615#comment-16632615 ] Kunal Khatua commented on DRILL-6661: - [~adityaar] there is a JDBC `Statement.setQueryTimeout(int timeInSeconds)` that does this. Any JDBC application should be able to do that. > Need a configurable parameter to stop Long running queries > -- > > Key: DRILL-6661 > URL: https://issues.apache.org/jira/browse/DRILL-6661 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Monitoring >Affects Versions: 1.13.0 >Reporter: Aditya Allamraju >Priority: Major > > I am looking for a way to stop any long running queries that run beyond a > certain time. > This is not to be confused with queue timeout which does not trigger if the > query has started executing. Other database vendors do this via resource > management or a simple timeout. > Currently, the default behavior is to allow any query in execution to > continue till arbitrarily large amount of time until completion. > There is a genuine need for this case. For instance, i want to stop queries > running beyond 15mins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6720) Unable to read more than 1000 records from DB2 table
[
https://issues.apache.org/jira/browse/DRILL-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632607#comment-16632607
]
Kunal Khatua commented on DRILL-6720:
-
[~kshitij@infy] this seems to be because of the driver closing the resultSet
prematurely and not specific to Drill. The resource pointed by [~appler]
mentions the driver as the cause.
> Unable to read more than 1000 records from DB2 table
>
>
> Key: DRILL-6720
> URL: https://issues.apache.org/jira/browse/DRILL-6720
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Storage - JDBC
>Affects Versions: 1.9.0
>Reporter: kshitij
>Priority: Critical
> Fix For: 1.9.0
>
> Attachments: drill_DB2_query_error_log.txt, drill_db2_query_status.PNG
>
>
> We have created a storage plugin in drill for DB2 database, PFB the details:
> {
> "type": "jdbc",
> "driver": "com.ibm.db2.jcc.DB2Driver",
> "url": "jdbc:db2://server:port/databasename",
> "username": "user",
> "password": "password",
> "enabled": true
> }
> Version of DB2 is 10.1. We are using a type 4 JDBC driver (db2jcc4.jar).
> When we try to read the data in any of the tables, the query fails with
> following error from drill:
> org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR:
> Failure while attempting to read from database. sql SELECT * FROM
> schema.table plugin DB2_PLUGIN Fragment 0:0 [Error Id:
> 1404544f-bb5e-439b-b1a8-679388bb344d on server:port]
> The error logs from drill have been attached.
> One interesting observation - when we put a LIMIT clause of <=1000 to the
> query, the query works and returns the data. Anything more than 1000 in LIMIT
> clause throws back the same error as above.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6704) Alter Session won't run on MapR 1.9(but runs on 1.8)
[ https://issues.apache.org/jira/browse/DRILL-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632604#comment-16632604 ] Kunal Khatua commented on DRILL-6704: - [~marcmcirl] are you trying to use Tableau to query a MapR node using Apache Drill? What is the ODBC driver you are using? If there is a protocol change in the client-server communication, you might be using a client that is compatible with MapR Drill 1.8 but not MapR Drill 1.9. Can you retry with the next version of the ODBC driver (*not* the latest, since that is compatible with Drill 1.13/1.14) ? > Alter Session won't run on MapR 1.9(but runs on 1.8) > > > Key: DRILL-6704 > URL: https://issues.apache.org/jira/browse/DRILL-6704 > Project: Apache Drill > Issue Type: Bug > Components: Client - ODBC >Affects Versions: 1.9.0 >Reporter: Marc McElhinney >Priority: Minor > > Have an issue whereby Tableau Desktop is trying to run an initial SQL on > connection to MapR through apache and won't allow it in 1.9, same sql works > fine in 1.8 , any thoughts? > running > ALTER SESSION SET `planner.memory.max_query_memory_per_node` = figure> > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6668) In Web Console, highlight options that are different from default values
[
https://issues.apache.org/jira/browse/DRILL-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632599#comment-16632599
]
Kunal Khatua commented on DRILL-6668:
-
[~paul-rogers] providing a button for 'ALTER SYSTEM RESET ' should be
possible. But it might confuse a user who changes a field (with a non-default
value) and tries to undo the change.
> In Web Console, highlight options that are different from default values
>
>
> Key: DRILL-6668
> URL: https://issues.apache.org/jira/browse/DRILL-6668
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.14.0
>Reporter: Paul Rogers
>Priority: Minor
>
> Suppose you inherit a Drill setup created by someone else (or by you, some
> time in the past). Or, suppose you are a support person. You want to know
> which Drill options have been changed from the defaults.
> The Web UI conveniently displays all options. But, there is no indication of
> which might have non-default values.
> After the improvements of the last year, the information needed to detect
> non-default values is now available. Would be great to mark these values.
> Perhaps using colors, perhaps with words.
> For example:
> *planner.width.max_per_node* 200 \[Update]
> Or
> planner.width.max_per_node (system) 200 \[Update]
> (The Web UI does not, I believe, show session settings, since the Web UI has
> no sessions. I believe the custom values are all set by {{ALTER SYSTEM}}.
> Otherwise, we could also have a "(session)" suffix above.)
> Then, in addition to the {{[Update]}} button, for non default values, also
> provide a {{[Reset]}} button that does the same as {{ALTER SESSION RESET}}.
> planner.width.max_per_node (session) 200 \[Update] \[Reset]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632558#comment-16632558
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r221377000
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.pojo.NonNullable;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.shaded.guava.com.google.common.collect.ListMultimap;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private LocalFunctionRegistry localFunctionRegistry;
+ private RemoteFunctionRegistry remoteFunctionRegistry;
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+
+this.localFunctionRegistry = context.getLocalFunctionRegistry();
+listLocalFunctionRegistry(functionMap);
+
+this.remoteFunctionRegistry = context.getRemoteFunctionRegistry();
+listRemoteFunctionRegistry(functionMap);
+
+//Extract list & sort via comparator
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(initFunctionComparator());
+sortedIterator = functionList.iterator();
+ }
+
+ //Provide a function comparator
+ private Comparator initFunctionComparator() {
+//Initialize the comparator
+return new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+};
+ }
+
+ //Loads local-registry functions into functionMap
+ private void listLocalFunctionRegistry(Map
functionMap) {
+ListMultimap builtInFuncHolderList =
localFunctionRegistry.getRegistryHolder().getAllFunctionsWithHolders();
+logger.debug("{} functions are currently initialized",
builtInFuncHolderList.size());
+
+Iterator builtInFuncIter =
builtInFuncHolderList.asMap().keySet().iterator();
+while (builtInFuncIter.hasNext()) {
+ String s = builtInFuncIter.next();
+ for (DrillFuncHolder drillFuncHolder : builtInFuncHolderList.get(s)) {
+String registeredNames[] = drillFuncHolder.getRegisteredNames();
+String signature = drillFuncHolder.getInputParameters();
+String returnType =
drillFuncHolder.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ FunctionInfo fnInfo = new FunctionInfo(name, signature, returnType,
LocalFunctionRegistry.BUILT_IN);
+ functionMap.put(generateKey(name, signature), fnInfo);
+}
+ }
+}
+ }
+
+ //Loads remote-registry functions into functionMap (updates if
+ private void listRemoteFunctionRegistry(Map
functionMap) {
Review comment:
@kkhatua
To avoid this unpleasant parsing I would suggest you add method to
`FunctionRegistryHolder` which would return `ListMultimap`, where key would be jar_name,
[jira] [Commented] (DRILL-6761) Documentation - broken table of contents on the REST-API page
[
https://issues.apache.org/jira/browse/DRILL-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632556#comment-16632556
]
ASF GitHub Bot commented on DRILL-6761:
---
bbevens closed pull request #1479: DRILL-6761: Updated table of contents on the
REST-API page
URL: https://github.com/apache/drill/pull/1479
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/_docs/developer-information/rest-api/010-rest-api-introduction.md
b/_docs/developer-information/rest-api/010-rest-api-introduction.md
index 7b0f90f2cdc..7db18ad4a65 100644
--- a/_docs/developer-information/rest-api/010-rest-api-introduction.md
+++ b/_docs/developer-information/rest-api/010-rest-api-introduction.md
@@ -18,17 +18,17 @@ Several examples in the document use the donuts.json file.
To download this file
This documentation presents HTTP methods in the same order as functions appear
in the Web Console:
-[**Query**]({{site.baseurl}}/docs/rest-api/#query)
+[**Query**]({{site.baseurl}}/docs/rest-api-introduction/#query)
Submit a query and return results.
-[**Profiles**](({{site.baseurl}}/docs/rest-api/#profiles))
+[**Profiles**]({{site.baseurl}}/docs/rest-api-introduction/#profiles)
* Get the profiles of running and completed queries.
* Get the profile of the query that has the given queryid.
* Cancel the query that has the given queryid.
-[**Storage**]({{site.baseurl}}/docs/rest-api/#storage)
+[**Storage**]({{site.baseurl}}/docs/rest-api-introduction/#storage)
* Get the list of storage plugin names and configurations.
* Get the definition of the named storage plugin.
@@ -38,11 +38,11 @@ Submit a query and return results.
* Get Drillbit information, such as ports numbers.
* Get the current memory metrics.
-[**Threads**](({{site.baseurl}}/docs/rest-api/#threads))
+[**Threads**]({{site.baseurl}}/docs/rest-api-introduction/#threads)
Get the status of threads.
-[**Options**]({{site.baseurl}}/docs/rest-api/#options)
+[**Options**]({{site.baseurl}}/docs/rest-api-introduction/#options)
List information about system/session options.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Documentation - broken table of contents on the REST-API page
> -
>
> Key: DRILL-6761
> URL: https://issues.apache.org/jira/browse/DRILL-6761
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Denys Ordynskiy
>Assignee: Denys Ordynskiy
>Priority: Minor
>
> Page [https://drill.apache.org/docs/rest-api-introduction/] has a broken
> links on the table of contents:
> [https://drill.apache.org/docs/rest-api/#query]
> [https://drill.apache.org/docs/rest-api-introduction/(/docs/rest-api/#profiles)]
> [https://drill.apache.org/docs/rest-api/#storage]
> [https://drill.apache.org/docs/rest-api-introduction/(/docs/rest-api/#threads)]
> [https://drill.apache.org/docs/rest-api/#options]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6749) Fixing broken links to CTAS and Explain Commands
[
https://issues.apache.org/jira/browse/DRILL-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632555#comment-16632555
]
ASF GitHub Bot commented on DRILL-6749:
---
bbevens closed pull request #1474: DRILL-6749: Fixing broken links to CTAS and
Explain Commands
URL: https://github.com/apache/drill/pull/1474
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/_docs/performance-tuning/026-parquet-filter-pushdown.md
b/_docs/performance-tuning/026-parquet-filter-pushdown.md
index 466f0830a85..0afdcee561f 100644
--- a/_docs/performance-tuning/026-parquet-filter-pushdown.md
+++ b/_docs/performance-tuning/026-parquet-filter-pushdown.md
@@ -24,7 +24,7 @@ Parquet filter pushdown is similar to partition pruning in
that it reduces the a
The query planner looks at the minimum and maximum values in each row group
for an intersection. If no intersection exists, the planner can prune the row
group in the table. If the minimum and maximum value range is too large, Drill
does not apply Parquet filter pushdown. The query planner can typically prune
more data when the tables in the Parquet file are sorted by row groups.
##Using Parquet Filter Pushdown
-Currently, Parquet filter pushdown only supports filters that reference
columns from a single table (local filters). Parquet filter pushdown requires
the minimum and maximum values in the Parquet file metadata. All Parquet files
created in Drill using the CTAS statement contain the necessary metadata. If
your Parquet files were created using another tool, you may need to use Drill
to read and rewrite the files using the [CTAS
command]({{site.baseurl}}/docs/create-table-as-ctas-command/).
+Currently, Parquet filter pushdown only supports filters that reference
columns from a single table (local filters). Parquet filter pushdown requires
the minimum and maximum values in the Parquet file metadata. All Parquet files
created in Drill using the CTAS statement contain the necessary metadata. If
your Parquet files were created using another tool, you may need to use Drill
to read and rewrite the files using the [CTAS
command]({{site.baseurl}}/docs/create-table-as-ctas/).
Parquet filter pushdown works best if you presort the data. You do not have to
sort the entire data set at once. You can sort a subset of the data set, sort
another subset, and so on.
@@ -39,7 +39,7 @@ The following table lists the Parquet filter pushdown options
with their descrip
| "planner.store.parquet.rowgroup.filter.pushdown.threshold" | Sets the number
of row groups that a table can have. You can increase the threshold if the
filter can prune many row groups. However, if this setting is too high, the
filter evaluation overhead increases. Base this setting on the data set.
Reduce this setting if the planning time is significant, or you do not see
any benefit at runtime. | 10,000|
###Viewing the Query Plan
-Because Drill applies Parquet filter pushdown during the query planning phase,
you can view the query execution plan to see if Drill pushes down the filter
when a query on a Parquet file contains a filter expression. You can run the
[EXPLAIN PLAN command]({{site.baseurl}}/docs/explain-commands/) to see the
execution plan for the query, as shown in the following example.
+Because Drill applies Parquet filter pushdown during the query planning phase,
you can view the query execution plan to see if Drill pushes down the filter
when a query on a Parquet file contains a filter expression. You can run the
[EXPLAIN PLAN command]({{site.baseurl}}/docs/explain/) to see the execution
plan for the query, as shown in the following example.
**Example**
@@ -79,4 +79,4 @@ The following table lists the supported and unsupported
clauses, operators, data
- a dynamic star in the sub-query or queries that include the WITH statement.
- several filter predicates with the OR logical operator.
- more than one EXISTS operator (instead of JOIN operators).
-- INNER JOIN and local filtering with a several conditions.
\ No newline at end of file
+- INNER JOIN and local filtering with a several conditions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fixing broken links to CTAS and Explain Commands
>
>
> Key: DRILL-6749
> URL: https://issues.apache.org/jira/browse/DRILL-6749
>
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632553#comment-16632553
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r221377000
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.pojo.NonNullable;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.shaded.guava.com.google.common.collect.ListMultimap;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private LocalFunctionRegistry localFunctionRegistry;
+ private RemoteFunctionRegistry remoteFunctionRegistry;
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+
+this.localFunctionRegistry = context.getLocalFunctionRegistry();
+listLocalFunctionRegistry(functionMap);
+
+this.remoteFunctionRegistry = context.getRemoteFunctionRegistry();
+listRemoteFunctionRegistry(functionMap);
+
+//Extract list & sort via comparator
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(initFunctionComparator());
+sortedIterator = functionList.iterator();
+ }
+
+ //Provide a function comparator
+ private Comparator initFunctionComparator() {
+//Initialize the comparator
+return new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+};
+ }
+
+ //Loads local-registry functions into functionMap
+ private void listLocalFunctionRegistry(Map
functionMap) {
+ListMultimap builtInFuncHolderList =
localFunctionRegistry.getRegistryHolder().getAllFunctionsWithHolders();
+logger.debug("{} functions are currently initialized",
builtInFuncHolderList.size());
+
+Iterator builtInFuncIter =
builtInFuncHolderList.asMap().keySet().iterator();
+while (builtInFuncIter.hasNext()) {
+ String s = builtInFuncIter.next();
+ for (DrillFuncHolder drillFuncHolder : builtInFuncHolderList.get(s)) {
+String registeredNames[] = drillFuncHolder.getRegisteredNames();
+String signature = drillFuncHolder.getInputParameters();
+String returnType =
drillFuncHolder.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ FunctionInfo fnInfo = new FunctionInfo(name, signature, returnType,
LocalFunctionRegistry.BUILT_IN);
+ functionMap.put(generateKey(name, signature), fnInfo);
+}
+ }
+}
+ }
+
+ //Loads remote-registry functions into functionMap (updates if
+ private void listRemoteFunctionRegistry(Map
functionMap) {
Review comment:
@kkhatua
To avoid this unpleasant parsing I would suggest you add method to
`FunctionRegistryHolder` which would return `ListMultimap`, where key would be jar_name,
[jira] [Commented] (DRILL-6762) Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
[
https://issues.apache.org/jira/browse/DRILL-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632539#comment-16632539
]
ASF GitHub Bot commented on DRILL-6762:
---
arina-ielchiieva opened a new pull request #1484: DRILL-6762: Fix dynamic UDFs
versioning issue
URL: https://github.com/apache/drill/pull/1484
1. Added UndefinedVersionDelegatingStore to serve as versioned wrapper for
those stores that do not support versioning.
2. Aligned remote and local function registries version type. Type will be
represented as int since ZK version is returned as int.
3. Added NOT_AVAILABLE and UNDEFINED versions to DataChangeVersion holder to
indicate proper registry state.
4. Added additional trace logging.
5. Minor refactoring and clean up.
Details in [DRILL-6762](https://issues.apache.org/jira/browse/DRILL-6762).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
> --
>
> Key: DRILL-6762
> URL: https://issues.apache.org/jira/browse/DRILL-6762
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.15.0
>
> Attachments: Dynamic UDFs issue.pdf
>
>
> Originally Reported :
> https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
> When using a 4-node Drill 1.14 cluster, UDF jars registered on one node are
> not usable on other nodes despite the {{/registry}} and ZK showing the UDFs
> as registered.
> This was previously working on 1.13.0
> *Root cause*
> 1. {{VersionedDelegatingStore}} was starting with version -1 and local
> function registry with version 0. This caused issues when
> {{LocalPersistentStore}} already existed on the file system. When adding jars
> into remote registry its versioned was bumped to 0 and synchronization did
> not happen since local registry had the same version.
> *Fix*: start {{VersionedDelegatingStore}} with version 0, local function
> registry with undefined version (-2) thus first sync will always happen.
> 2. {{PersistentStoreProvider.getOrCreateVersionedStore}} was wrapping stores
> into VersionedDelegatingStore for those store providers that did not override
> this method. Only Zookeeper store was overriding it. But
> {{VersionedDelegatingStore}} is only keeps versioning locally and thus can be
> applied only for local stores, i.e. Hbase, Mongo cannot use it.
> {{CachingPersistentStoreProvider}} did not override
> {{getOrCreateVersionedStore}} either. Mostly all stores in Drill are created
> using {{CachingPersistentStoreProvider}}. Thus all stores where wrapped into
> {{VersionedDelegatingStore}}, even Zookeeper one which caused function
> registries version synchronization issues.
> *Fix*: Add {{UndefinedVersionDelegatingStore}} for those stores that do not
> support versioning and wrap into it by default in
> {{PersistentStoreProvider.getOrCreateVersionedStore}} if this method is not
> overriden. {{UndefinedVersionDelegatingStore}} will return UNDEFINED version
> (-2). During sync between remote and local registries if remote registry has
> UNDEFINED version sync will be done immediately, on the contrary with
> NOT_AVAILABLE version (-1) which indicates that remote function registry is
> not accessible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632537#comment-16632537
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r221377000
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.pojo.NonNullable;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.shaded.guava.com.google.common.collect.ListMultimap;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private LocalFunctionRegistry localFunctionRegistry;
+ private RemoteFunctionRegistry remoteFunctionRegistry;
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+
+this.localFunctionRegistry = context.getLocalFunctionRegistry();
+listLocalFunctionRegistry(functionMap);
+
+this.remoteFunctionRegistry = context.getRemoteFunctionRegistry();
+listRemoteFunctionRegistry(functionMap);
+
+//Extract list & sort via comparator
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(initFunctionComparator());
+sortedIterator = functionList.iterator();
+ }
+
+ //Provide a function comparator
+ private Comparator initFunctionComparator() {
+//Initialize the comparator
+return new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+};
+ }
+
+ //Loads local-registry functions into functionMap
+ private void listLocalFunctionRegistry(Map
functionMap) {
+ListMultimap builtInFuncHolderList =
localFunctionRegistry.getRegistryHolder().getAllFunctionsWithHolders();
+logger.debug("{} functions are currently initialized",
builtInFuncHolderList.size());
+
+Iterator builtInFuncIter =
builtInFuncHolderList.asMap().keySet().iterator();
+while (builtInFuncIter.hasNext()) {
+ String s = builtInFuncIter.next();
+ for (DrillFuncHolder drillFuncHolder : builtInFuncHolderList.get(s)) {
+String registeredNames[] = drillFuncHolder.getRegisteredNames();
+String signature = drillFuncHolder.getInputParameters();
+String returnType =
drillFuncHolder.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ FunctionInfo fnInfo = new FunctionInfo(name, signature, returnType,
LocalFunctionRegistry.BUILT_IN);
+ functionMap.put(generateKey(name, signature), fnInfo);
+}
+ }
+}
+ }
+
+ //Loads remote-registry functions into functionMap (updates if
+ private void listRemoteFunctionRegistry(Map
functionMap) {
Review comment:
@kkhatua
To avoid this unpleasant parsing I would suggest you add method to
`FunctionRegistryHolder` which would return `ListMultimap`, where key would be jar_name,
[jira] [Updated] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-3988: Labels: doc-impacting (was: newbie) > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6762) Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
[
https://issues.apache.org/jira/browse/DRILL-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6762:
Description:
Originally Reported :
https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
When using a 4-node Drill 1.14 cluster, UDF jars registered on one node are not
usable on other nodes despite the {{/registry}} and ZK showing the UDFs as
registered.
This was previously working on 1.13.0
*Root cause*
1. {{VersionedDelegatingStore}} was starting with version -1 and local function
registry with version 0. This caused issues when {{LocalPersistentStore}}
already existed on the file system. When adding jars into remote registry its
versioned was bumped to 0 and synchronization did not happen since local
registry had the same version.
*Fix*: start {{VersionedDelegatingStore}} with version 0, local function
registry with undefined version (-2) thus first sync will always happen.
2. {{PersistentStoreProvider.getOrCreateVersionedStore}} was wrapping stores
into VersionedDelegatingStore for those store providers that did not override
this method. Only Zookeeper store was overriding it. But
{{VersionedDelegatingStore}} is only keeps versioning locally and thus can be
applied only for local stores, i.e. Hbase, Mango cannot use it.
{{CachingPersistentStoreProvider}} did not override
{{getOrCreateVersionedStore}} either. Mostly all stores in Drill are created
using {{CachingPersistentStoreProvider}}. Thus all stores where wrapped into
{{VersionedDelegatingStore}}, even Zookeeper one which caused function
registries version synchronization issues.
*Fix*: Add {{UndefinedVersionDelegatingStore}} for those stores that do not
support versioning and wrap into it by default in
{{PersistentStoreProvider.getOrCreateVersionedStore}} if this method is not
overriden. {{UndefinedVersionDelegatingStore}} will return UNDEFINED version
(-2). During sync between remote and local registries if remote registry has
UNDEFINED version sync will be done immediately, on the contrary with
NOT_AVAILABLE version (-1) which indicates that remote function registry is not
accessible.
was:
Originally Reported :
https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
When using a 4-node Drill 1.14 cluster, UDF jars registered on one node are not
usable on other nodes despite the {{/registry}} and ZK showing the UDFs as
registered.
This was previously working on 1.13.0
*Root cause*
1. {{VersionedDelegatingStore}} was starting with version -1 and local function
registry with version 0. This caused issues when {{LocalPersistentStore}}
already existed on the file system. When adding jars into remote registry its
versioned was bumped to 0 and synchronization did not happen since local
registry had the same version.
*Fix*: start {{VersionedDelegatingStore}} with version 0, local function
registry with undefined version (-2) thus first sync will always happen.
2. {{PersistentStoreProvider.getOrCreateVersionedStore}} was wrapping stores
into VersionedDelegatingStore for those store providers that did not override
this method. Only Zookeeper store was overriding it. But
{{VersionedDelegatingStore}} is only keeps versioning locally and thus can be
applied only for local stores, i.e. Hbase, Mango cannot use it.
{{CachingPersistentStoreProvider}} did not override
{{getOrCreateVersionedStore}} either. Mostly all stores in Drill are created
using {{CachingPersistentStoreProvider}}. Thus all stores where wrapped into
{{VersionedDelegatingStore}}, even Zookeeper one which caused function
registries version synchronization issues.
*Fix*: Add {{UnversionedDelegatingStore}} for those stores that do not support
versioning and wrap into it by default in
{{PersistentStoreProvider.getOrCreateVersionedStore}} if this method is not
overriden. {{UnversionedDelegatingStore}} will return UNDEFINED version (-2).
During sync between remote and local registries if remote registry has
UNDEFINED version sync will be done immediately, on the contrary with
NOT_AVAILABLE version (-1) which indicates that remote function registry is not
accessible.
> Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
> --
>
> Key: DRILL-6762
> URL: https://issues.apache.org/jira/browse/DRILL-6762
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.15.0
>
> Attachments: Dynamic UDFs issue.pdf
>
>
> Originally Reported :
> https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
> When using a 4-node Drill 1.14
[jira] [Updated] (DRILL-6762) Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
[
https://issues.apache.org/jira/browse/DRILL-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6762:
Description:
Originally Reported :
https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
When using a 4-node Drill 1.14 cluster, UDF jars registered on one node are not
usable on other nodes despite the {{/registry}} and ZK showing the UDFs as
registered.
This was previously working on 1.13.0
*Root cause*
1. {{VersionedDelegatingStore}} was starting with version -1 and local function
registry with version 0. This caused issues when {{LocalPersistentStore}}
already existed on the file system. When adding jars into remote registry its
versioned was bumped to 0 and synchronization did not happen since local
registry had the same version.
*Fix*: start {{VersionedDelegatingStore}} with version 0, local function
registry with undefined version (-2) thus first sync will always happen.
2. {{PersistentStoreProvider.getOrCreateVersionedStore}} was wrapping stores
into VersionedDelegatingStore for those store providers that did not override
this method. Only Zookeeper store was overriding it. But
{{VersionedDelegatingStore}} is only keeps versioning locally and thus can be
applied only for local stores, i.e. Hbase, Mongo cannot use it.
{{CachingPersistentStoreProvider}} did not override
{{getOrCreateVersionedStore}} either. Mostly all stores in Drill are created
using {{CachingPersistentStoreProvider}}. Thus all stores where wrapped into
{{VersionedDelegatingStore}}, even Zookeeper one which caused function
registries version synchronization issues.
*Fix*: Add {{UndefinedVersionDelegatingStore}} for those stores that do not
support versioning and wrap into it by default in
{{PersistentStoreProvider.getOrCreateVersionedStore}} if this method is not
overriden. {{UndefinedVersionDelegatingStore}} will return UNDEFINED version
(-2). During sync between remote and local registries if remote registry has
UNDEFINED version sync will be done immediately, on the contrary with
NOT_AVAILABLE version (-1) which indicates that remote function registry is not
accessible.
was:
Originally Reported :
https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
When using a 4-node Drill 1.14 cluster, UDF jars registered on one node are not
usable on other nodes despite the {{/registry}} and ZK showing the UDFs as
registered.
This was previously working on 1.13.0
*Root cause*
1. {{VersionedDelegatingStore}} was starting with version -1 and local function
registry with version 0. This caused issues when {{LocalPersistentStore}}
already existed on the file system. When adding jars into remote registry its
versioned was bumped to 0 and synchronization did not happen since local
registry had the same version.
*Fix*: start {{VersionedDelegatingStore}} with version 0, local function
registry with undefined version (-2) thus first sync will always happen.
2. {{PersistentStoreProvider.getOrCreateVersionedStore}} was wrapping stores
into VersionedDelegatingStore for those store providers that did not override
this method. Only Zookeeper store was overriding it. But
{{VersionedDelegatingStore}} is only keeps versioning locally and thus can be
applied only for local stores, i.e. Hbase, Mango cannot use it.
{{CachingPersistentStoreProvider}} did not override
{{getOrCreateVersionedStore}} either. Mostly all stores in Drill are created
using {{CachingPersistentStoreProvider}}. Thus all stores where wrapped into
{{VersionedDelegatingStore}}, even Zookeeper one which caused function
registries version synchronization issues.
*Fix*: Add {{UndefinedVersionDelegatingStore}} for those stores that do not
support versioning and wrap into it by default in
{{PersistentStoreProvider.getOrCreateVersionedStore}} if this method is not
overriden. {{UndefinedVersionDelegatingStore}} will return UNDEFINED version
(-2). During sync between remote and local registries if remote registry has
UNDEFINED version sync will be done immediately, on the contrary with
NOT_AVAILABLE version (-1) which indicates that remote function registry is not
accessible.
> Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
> --
>
> Key: DRILL-6762
> URL: https://issues.apache.org/jira/browse/DRILL-6762
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.15.0
>
> Attachments: Dynamic UDFs issue.pdf
>
>
> Originally Reported :
> https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
> When using a 4-node Drill
[jira] [Updated] (DRILL-6762) Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
[
https://issues.apache.org/jira/browse/DRILL-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6762:
Description:
Originally Reported :
https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
When using a 4-node Drill 1.14 cluster, UDF jars registered on one node are not
usable on other nodes despite the {{/registry}} and ZK showing the UDFs as
registered.
This was previously working on 1.13.0
*Root cause*
1. {{VersionedDelegatingStore}} was starting with version -1 and local function
registry with version 0. This caused issues when {{LocalPersistentStore}}
already existed on the file system. When adding jars into remote registry its
versioned was bumped to 0 and synchronization did not happen since local
registry had the same version.
*Fix*: start {{VersionedDelegatingStore}} with version 0, local function
registry with undefined version (-2) thus first sync will always happen.
2. {{PersistentStoreProvider.getOrCreateVersionedStore}} was wrapping stores
into VersionedDelegatingStore for those store providers that did not override
this method. Only Zookeeper store was overriding it. But
{{VersionedDelegatingStore}} is only keeps versioning locally and thus can be
applied only for local stores, i.e. Hbase, Mango cannot use it.
{{CachingPersistentStoreProvider}} did not override
{{getOrCreateVersionedStore}} either. Mostly all stores in Drill are created
using {{CachingPersistentStoreProvider}}. Thus all stores where wrapped into
{{VersionedDelegatingStore}}, even Zookeeper one which caused function
registries version synchronization issues.
*Fix*: Add {{UnversionedDelegatingStore}} for those stores that do not support
versioning and wrap into it by default in
{{PersistentStoreProvider.getOrCreateVersionedStore}} if this method is not
overriden. {{UnversionedDelegatingStore}} will return UNDEFINED version (-2).
During sync between remote and local registries if remote registry has
UNDEFINED version sync will be done immediately, on the contrary with
NOT_AVAILABLE version (-1) which indicates that remote function registry is not
accessible.
was:
Originally Reported :
https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
When using a 4-node Drill 1.14 cluster, UDF jars registered on one node are not
usable on other nodes despite the {{/registry}} and ZK showing the UDFs as
registered.
This was previously working on 1.13.0
> Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
> --
>
> Key: DRILL-6762
> URL: https://issues.apache.org/jira/browse/DRILL-6762
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.15.0
>
> Attachments: Dynamic UDFs issue.pdf
>
>
> Originally Reported :
> https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
> When using a 4-node Drill 1.14 cluster, UDF jars registered on one node are
> not usable on other nodes despite the {{/registry}} and ZK showing the UDFs
> as registered.
> This was previously working on 1.13.0
> *Root cause*
> 1. {{VersionedDelegatingStore}} was starting with version -1 and local
> function registry with version 0. This caused issues when
> {{LocalPersistentStore}} already existed on the file system. When adding jars
> into remote registry its versioned was bumped to 0 and synchronization did
> not happen since local registry had the same version.
> *Fix*: start {{VersionedDelegatingStore}} with version 0, local function
> registry with undefined version (-2) thus first sync will always happen.
> 2. {{PersistentStoreProvider.getOrCreateVersionedStore}} was wrapping stores
> into VersionedDelegatingStore for those store providers that did not override
> this method. Only Zookeeper store was overriding it. But
> {{VersionedDelegatingStore}} is only keeps versioning locally and thus can be
> applied only for local stores, i.e. Hbase, Mango cannot use it.
> {{CachingPersistentStoreProvider}} did not override
> {{getOrCreateVersionedStore}} either. Mostly all stores in Drill are created
> using {{CachingPersistentStoreProvider}}. Thus all stores where wrapped into
> {{VersionedDelegatingStore}}, even Zookeeper one which caused function
> registries version synchronization issues.
> *Fix*: Add {{UnversionedDelegatingStore}} for those stores that do not
> support versioning and wrap into it by default in
> {{PersistentStoreProvider.getOrCreateVersionedStore}} if this method is not
> overriden. {{UnversionedDelegatingStore}} will return UNDEFINED version
> (-2). During sync between
[jira] [Commented] (DRILL-6734) Unable to find value vector of path `EXPR$0`, returning null instance.
[
https://issues.apache.org/jira/browse/DRILL-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632277#comment-16632277
]
Hanumath Rao Maduri commented on DRILL-6734:
[~appler] Can you please post the plan for this query.
Please use the following queries to get the plan.
{code:java}
explain plan including all attributes for select count(*) from
ngmysql.information_schema.`PROCESSLIST`{code}
and also with alias name.
{code:java}
explain plan including all attributes for select count(*) as num from
ngmysql.information_schema.`PROCESSLIST`{code}
> Unable to find value vector of path `EXPR$0`, returning null instance.
> --
>
> Key: DRILL-6734
> URL: https://issues.apache.org/jira/browse/DRILL-6734
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.14.0
> Environment: Apache Drill version 1.14.0 running on CentOS 7.0.1406.
> MySQL version 5.5.43 running on CentOS 6.4.
> MySQL connector/j version 5.1.44.
>Reporter: Cheolgoo Kang
>Priority: Major
>
> Expressions in a query against JDBC without alias returns null as their value.
> I was trying to run this sample query to retrieve count of a MySQL table
> connected to Drill through the RDBMS storage plugin:
> {code}
> select count(*) from ngmysql.information_schema.`PROCESSLIST`
> {code}
> which returns
> {code}
> EXPR$0
> -
>
> {code}
> and you could find this warning from the log:
> {quote}
> Unable to find value vector of path `EXPR$0`, returning null instance.
> {quote}
> But it works fine if you give an alias to the expression like this:
> {code}
> select count(*) as num from ngmysql.information_schema.`PROCESSLIST`;
> {code}
> which would end up giving this:
> {code}
> num
> --
> 16
> {code}
> Here's the portion of logs regarding the sample query:
> {code}
> 2018-09-07 21:44:52,709 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f: select count(*) from
> ngmysql.information_schema.`PROCESSLIST`
> 2018-09-07 21:44:52,752 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2018-09-07 21:44:52,753 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f:0:0: State to report: RUNNING
> 2018-09-07 21:44:52,756 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] WARN
> o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path
> `EXPR$0`, returning null instance.
> 2018-09-07 21:44:52,759 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f:0:0: State change requested RUNNING -->
> FINISHED
> 2018-09-07 21:44:52,760 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f:0:0: State to report: FINISHED
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6734) Unable to find value vector of path `EXPR$0`, returning null instance.
[
https://issues.apache.org/jira/browse/DRILL-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632261#comment-16632261
]
Kunal Khatua commented on DRILL-6734:
-
[~hmaduri] can you provide some guidance?
> Unable to find value vector of path `EXPR$0`, returning null instance.
> --
>
> Key: DRILL-6734
> URL: https://issues.apache.org/jira/browse/DRILL-6734
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.14.0
> Environment: Apache Drill version 1.14.0 running on CentOS 7.0.1406.
> MySQL version 5.5.43 running on CentOS 6.4.
> MySQL connector/j version 5.1.44.
>Reporter: Cheolgoo Kang
>Priority: Major
>
> Expressions in a query against JDBC without alias returns null as their value.
> I was trying to run this sample query to retrieve count of a MySQL table
> connected to Drill through the RDBMS storage plugin:
> {code}
> select count(*) from ngmysql.information_schema.`PROCESSLIST`
> {code}
> which returns
> {code}
> EXPR$0
> -
>
> {code}
> and you could find this warning from the log:
> {quote}
> Unable to find value vector of path `EXPR$0`, returning null instance.
> {quote}
> But it works fine if you give an alias to the expression like this:
> {code}
> select count(*) as num from ngmysql.information_schema.`PROCESSLIST`;
> {code}
> which would end up giving this:
> {code}
> num
> --
> 16
> {code}
> Here's the portion of logs regarding the sample query:
> {code}
> 2018-09-07 21:44:52,709 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f: select count(*) from
> ngmysql.information_schema.`PROCESSLIST`
> 2018-09-07 21:44:52,752 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2018-09-07 21:44:52,753 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f:0:0: State to report: RUNNING
> 2018-09-07 21:44:52,756 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] WARN
> o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path
> `EXPR$0`, returning null instance.
> 2018-09-07 21:44:52,759 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f:0:0: State change requested RUNNING -->
> FINISHED
> 2018-09-07 21:44:52,760 [246d0eaa-f8e6-9536-af0c-1df3932cce9f:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 246d0eaa-f8e6-9536-af0c-1df3932cce9f:0:0: State to report: FINISHED
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6084) Expose list of Drill functions for consumption by JavaScript libraries
[
https://issues.apache.org/jira/browse/DRILL-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-6084:
Reviewer: Arina Ielchiieva
> Expose list of Drill functions for consumption by JavaScript libraries
> --
>
> Key: DRILL-6084
> URL: https://issues.apache.org/jira/browse/DRILL-6084
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Labels: javascript
> Fix For: 1.15.0
>
>
> DRILL-5868 introduces SQL highlighter and the auto-complete feature in the
> web-UI's query editor.
> For the backend Javascript to highlight the keywords correctly as SQL
> reserved words or functions, it needs a list of all the Drill functions to be
> already defined in a source JavaScript ( {{../mode-sql.js}} ) .
> While this works well for standard Drill functions, it means that any new
> function added to Drill needs to be explicitly hard-coded into the file,
> rather than be programatically discovered and loaded for the library to
> highlight.
> It would help if this can be done dynamically without contributors having to
> explicitly alter the script file to add new function names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6764) Query fails with IOB when Unnest has reference to deep nested field like (t.c_orders.o_lineitems)
Sorabh Hamirwasia created DRILL-6764:
Summary: Query fails with IOB when Unnest has reference to deep
nested field like (t.c_orders.o_lineitems)
Key: DRILL-6764
URL: https://issues.apache.org/jira/browse/DRILL-6764
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 1.14.0
Reporter: Sorabh Hamirwasia
Assignee: Hanumath Rao Maduri
Fix For: 1.15.0
When we run query with unnest referencing a list within a nested map type then
query fails with IOB. But if list if not nested in map it works fine. For
example:
*Failure Case: with schema*
{code:java}
customer {
element {
c_orders: [O1, O2]
}
}
{code}
{code:java}
SELECT tt.og, tt.ag
FROM dfs.testData.`/*` t,
LATERAL (SELECT l.o as og, l.o as ag FROM UNNEST(t.element.c_orders) l(o))
tt{code}
*Success Case:*
{code:java}
customer {
c_orders: [O1, O2]
}
{code}
{code:java}
SELECT tt.og, tt.ag
FROM dfs.testData.`/*` t,
LATERAL (SELECT l.o as og, l.o as ag FROM UNNEST(t.c_orders)
l(o)) tt{code}
*Exception:*
{code:java}
2018-09-28 11:13:03,633 [245190cf-da1c-02cd-977e-ded27e7abcff:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
245190cf-da1c-02cd-977e-ded27e7abcff issued by anonymous: SELECT tt.og, tt.ag
FROM dfs.testData.`/*` t,
LATERAL (SELECT l.o as og, l.o as ag FROM UNNEST(t.element.c_orders) l(o)) tt
2018-09-28 11:13:03,725 [245190cf-da1c-02cd-977e-ded27e7abcff:foreman] ERROR
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IndexOutOfBoundsException:
index (3) must be less than size (3)
[Error Id: d5a21cc8-581f-4c35-9950-c7cfbd7de194 on 172.30.8.49:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IndexOutOfBoundsException: index (3) must be less than size (3)
[Error Id: d5a21cc8-581f-4c35-9950-c7cfbd7de194 on 172.30.8.49:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:779)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.checkCommonStates(QueryStateProcessor.java:325)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.planning(QueryStateProcessor.java:221)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:83)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:299)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
exception during fragment initialization: Error while applying rule
DrillMergeProjectRule:force_mode, args
[rel#4423:LogicalProject.NONE.ANY([]).[](input=rel#4422:Subset#8.NONE.ANY([]).[],og=$2,ag=$3),
rel#4421:LogicalProject.NONE.ANY([]).[](input=rel#4420:Subset#7.NONE.ANY([]).[],**=$0,c_orders=$1,og=$3)]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:300)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
... 3 common frames omitted
Caused by: java.lang.RuntimeException: Error while applying rule
DrillMergeProjectRule:force_mode, args
[rel#4423:LogicalProject.NONE.ANY([]).[](input=rel#4422:Subset#8.NONE.ANY([]).[],og=$2,ag=$3),
rel#4421:LogicalProject.NONE.ANY([]).[](input=rel#4420:Subset#7.NONE.ANY([]).[],**=$0,c_orders=$1,og=$3)]
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:236)
~[calcite-core-1.17.0-drill-r0.jar:1.17.0-drill-r0]
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:648)
~[calcite-core-1.17.0-drill-r0.jar:1.17.0-drill-r0]
at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
~[calcite-core-1.17.0-drill-r0.jar:1.17.0-drill-r0]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:425)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:365)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:252)
[jira] [Updated] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-3988: Reviewer: Arina Ielchiieva > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: newbie > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632180#comment-16632180 ] ASF GitHub Bot commented on DRILL-3988: --- kkhatua opened a new pull request #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483 This commit exposes available SQL functions in Drill and also detects UDFs that have been dynamically loaded into Drill. An example is shown below for 2 UDFs dynamically loaded into the cluster, along side the existing built-in functions that come with Drill. ``` 0: jdbc:drill:schema=sys> select source, count(*) as functionCount from sys.functions group by source; +-++ | source | functionCount | +-++ | built-in| 2704 | | simple-drill-function-1.0-SNAPSHOT.jar | 12 | | drill-url-tools-1.0.jar | 1 | +-++ 1 row selected (0.211 seconds) ``` The system table exposes information as shown. Since UDFs are lazily initialized (i.e. only when a SQL query needs it), the `returnType` is not available and listed as `n/a`. Once the UDF is initialized, the `returnType` is also available. The `random(FLOAT8-REQUIRED,FLOAT8-REQUIRED)` function is an example of a lazily loaded UDF (see `returnType`). The `url_parse(VARCHAR-REQUIRED)` function is an example of an initialized UDF (see `returnType`). Rest are built-in functions that meet the query's filter criteria. ``` 0: jdbc:drill:schema=sys>select * from sys.functions where name like 'random' or `name` like '%url%'; +-+--+-+-+ |name |signature | returnType | source | +-+--+-+-+ | parse_url | VARCHAR-REQUIRED | LATE| built-in | | random | | FLOAT8 | built-in | | random | FLOAT8-REQUIRED,FLOAT8-REQUIRED | n/a | simple-drill-function-1.0-SNAPSHOT.jar | | url_decode | VARCHAR-REQUIRED | VARCHAR | built-in | | url_encode | VARCHAR-REQUIRED | VARCHAR | built-in | | url_parse | VARCHAR-REQUIRED | LATE| drill-url-tools-1.0.jar | +-+--+-+-+ 6 rows selected (0.221 seconds) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: newbie > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6731) JPPD:Move aggregating the BF from the Foreman to the RuntimeFilter
[
https://issues.apache.org/jira/browse/DRILL-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632166#comment-16632166
]
ASF GitHub Bot commented on DRILL-6731:
---
sohami commented on a change in pull request #1459: DRILL-6731: Move the BFs
aggregating work from the Foreman to the RuntimeFi…
URL: https://github.com/apache/drill/pull/1459#discussion_r221329807
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterSink.java
##
@@ -36,25 +40,63 @@
private RuntimeFilterWritable aggregated = null;
- private Queue rfQueue = new ConcurrentLinkedQueue<>();
+ private BlockingQueue rfQueue = new
LinkedBlockingQueue<>();
private AtomicBoolean running = new AtomicBoolean(true);
+ private ReentrantLock aggregatedRFLock = new ReentrantLock();
+
+ private Thread asyncAggregateThread;
+
+ private BufferAllocator bufferAllocator;
+
+ private static final Logger logger =
LoggerFactory.getLogger(RuntimeFilterSink.class);
+
+
+ public RuntimeFilterSink(BufferAllocator bufferAllocator) {
+this.bufferAllocator = bufferAllocator;
+AsyncAggregateWorker asyncAggregateWorker = new AsyncAggregateWorker();
+asyncAggregateThread = new
NamedThreadFactory("RFAggregating-").newThread(asyncAggregateWorker);
+asyncAggregateThread.start();
+ }
+
public void aggregate(RuntimeFilterWritable runtimeFilterWritable) {
-rfQueue.add(runtimeFilterWritable);
-if (currentBookId.get() == 0) {
- AsyncAggregateWorker asyncAggregateWorker = new AsyncAggregateWorker();
- Thread asyncAggregateThread = new
NamedThreadFactory("RFAggregating-").newThread(asyncAggregateWorker);
- asyncAggregateThread.start();
+if (running.get()) {
+ if (containOne()) {
+boolean same = aggregated.same(runtimeFilterWritable);
+if (!same) {
+ //This is to solve the only one fragment case that two
RuntimeFilterRecordBatchs
+ //share the same FragmentContext.
Review comment:
Yes I did consider the right deep tree case too and the logic looks fine to
me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> JPPD:Move aggregating the BF from the Foreman to the RuntimeFilter
> --
>
> Key: DRILL-6731
> URL: https://issues.apache.org/jira/browse/DRILL-6731
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
>Affects Versions: 1.15.0
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
>
> This PR is to move the BloomFilter aggregating work from the foreman to
> RuntimeFilter. Though this change, the RuntimeFilter can apply the incoming
> BF as soon as possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6749) Fixing broken links to CTAS and Explain Commands
[ https://issues.apache.org/jira/browse/DRILL-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632117#comment-16632117 ] ASF GitHub Bot commented on DRILL-6749: --- bbevens commented on issue #1474: DRILL-6749: Fixing broken links to CTAS and Explain Commands URL: https://github.com/apache/drill/pull/1474#issuecomment-425495739 Yes, I'm working on another release and need to finish that. I will try to get to this later today or tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fixing broken links to CTAS and Explain Commands > > > Key: DRILL-6749 > URL: https://issues.apache.org/jira/browse/DRILL-6749 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Reporter: Nitin Sharma >Assignee: Nitin Sharma >Priority: Major > Fix For: 1.15.0 > > > The parquet push down page references a few dead links to CTAS and Explain > Commands. They need to be fixed > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6749) Fixing broken links to CTAS and Explain Commands
[ https://issues.apache.org/jira/browse/DRILL-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632107#comment-16632107 ] ASF GitHub Bot commented on DRILL-6749: --- nitingithub commented on issue #1474: DRILL-6749: Fixing broken links to CTAS and Explain Commands URL: https://github.com/apache/drill/pull/1474#issuecomment-425494204 Hi, can you please take a look at this when you get a chance? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fixing broken links to CTAS and Explain Commands > > > Key: DRILL-6749 > URL: https://issues.apache.org/jira/browse/DRILL-6749 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Reporter: Nitin Sharma >Assignee: Nitin Sharma >Priority: Major > Fix For: 1.15.0 > > > The parquet push down page references a few dead links to CTAS and Explain > Commands. They need to be fixed > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6762) Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
[
https://issues.apache.org/jira/browse/DRILL-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631601#comment-16631601
]
ASF GitHub Bot commented on DRILL-6762:
---
arina-ielchiieva commented on a change in pull request #1482: DRILL-6762:
Dynamic UDFs registered on 1 Drillbit not visible on others
URL: https://github.com/apache/drill/pull/1482#discussion_r221192929
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/RemoteFunctionRegistry.java
##
@@ -84,6 +84,7 @@
private static final String registry_path = "registry";
private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(RemoteFunctionRegistry.class);
private static final ObjectMapper mapper = new
ObjectMapper().enable(INDENT_OUTPUT);
+ public static final int UNREACHABLE = Integer.MIN_VALUE;
Review comment:
I would suggest we have all versions in int since ZK versioning is always in
int.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
> --
>
> Key: DRILL-6762
> URL: https://issues.apache.org/jira/browse/DRILL-6762
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.15.0
>
> Attachments: Dynamic UDFs issue.pdf
>
>
> Originally Reported :
> https://stackoverflow.com/questions/52480160/dynamic-udf-in-apache-drill-cluster
> When using a 4-node Drill 1.14 cluster, UDF jars registered on one node are
> not usable on other nodes despite the {{/registry}} and ZK showing the UDFs
> as registered.
> This was previously working on 1.13.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6731) JPPD:Move aggregating the BF from the Foreman to the RuntimeFilter
[
https://issues.apache.org/jira/browse/DRILL-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631504#comment-16631504
]
ASF GitHub Bot commented on DRILL-6731:
---
weijietong commented on a change in pull request #1459: DRILL-6731: Move the
BFs aggregating work from the Foreman to the RuntimeFi…
URL: https://github.com/apache/drill/pull/1459#discussion_r221171069
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterSink.java
##
@@ -36,25 +40,63 @@
private RuntimeFilterWritable aggregated = null;
- private Queue rfQueue = new ConcurrentLinkedQueue<>();
+ private BlockingQueue rfQueue = new
LinkedBlockingQueue<>();
private AtomicBoolean running = new AtomicBoolean(true);
+ private ReentrantLock aggregatedRFLock = new ReentrantLock();
+
+ private Thread asyncAggregateThread;
+
+ private BufferAllocator bufferAllocator;
+
+ private static final Logger logger =
LoggerFactory.getLogger(RuntimeFilterSink.class);
+
+
+ public RuntimeFilterSink(BufferAllocator bufferAllocator) {
+this.bufferAllocator = bufferAllocator;
+AsyncAggregateWorker asyncAggregateWorker = new AsyncAggregateWorker();
+asyncAggregateThread = new
NamedThreadFactory("RFAggregating-").newThread(asyncAggregateWorker);
+asyncAggregateThread.start();
+ }
+
public void aggregate(RuntimeFilterWritable runtimeFilterWritable) {
-rfQueue.add(runtimeFilterWritable);
-if (currentBookId.get() == 0) {
- AsyncAggregateWorker asyncAggregateWorker = new AsyncAggregateWorker();
- Thread asyncAggregateThread = new
NamedThreadFactory("RFAggregating-").newThread(asyncAggregateWorker);
- asyncAggregateThread.start();
+if (running.get()) {
+ if (containOne()) {
+boolean same = aggregated.same(runtimeFilterWritable);
+if (!same) {
+ //This is to solve the only one fragment case that two
RuntimeFilterRecordBatchs
+ //share the same FragmentContext.
Review comment:
The directly example is this TPC-H sql:
```
select l.l_orderkey, sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate, o.o_shippriority
from dfs.`/tpch-parquet/customer` c, dfs.`/tpch-parquet/orders` o,
dfs.`/tpch-parquet/lineitem` l
where c.c_mktsegment = 'HOUSEHOLD' and c.c_custkey = o.o_custkey and
l.l_orderkey = o.o_orderkey and o.o_orderdate < date '1995-03-25' and
l.l_shipdate > date '1995-03-25'
group by l.l_orderkey, o.o_orderdate, o.o_shippriority
order by revenue desc, o.o_orderdate limit 10
```
The corresponding plan is:
```
00-00Screen : rowType = RecordType(ANY l_orderkey, ANY revenue, ANY
o_orderdate, ANY o_shippriority): rowcount = 10.0, cumulative cost =
{4051714.179997 rows, 3.094535517999211E7 cpu, 0.0 io, 0.0 network,
1.298667392002E7 memory}, id = 3423
00-01 Project(l_orderkey=[$0], revenue=[$1], o_orderdate=[$2],
o_shippriority=[$3]) : rowType = RecordType(ANY l_orderkey, ANY revenue, ANY
o_orderdate, ANY o_shippriority): rowcount = 10.0, cumulative cost =
{4051713.179997 rows, 3.094535417999211E7 cpu, 0.0 io, 0.0 network,
1.298667392002E7 memory}, id = 3422
00-02SelectionVectorRemover : rowType = RecordType(ANY l_orderkey,
ANY revenue, ANY o_orderdate, ANY o_shippriority): rowcount = 10.0, cumulative
cost = {4051703.179997 rows, 3.094531417999211E7 cpu, 0.0 io, 0.0 network,
1.298667392002E7 memory}, id = 3421
00-03 Limit(fetch=[10]) : rowType = RecordType(ANY l_orderkey, ANY
revenue, ANY o_orderdate, ANY o_shippriority): rowcount = 10.0, cumulative cost
= {4051693.179997 rows, 3.094530417999211E7 cpu, 0.0 io, 0.0 network,
1.298667392002E7 memory}, id = 3420
00-04SelectionVectorRemover : rowType = RecordType(ANY
l_orderkey, ANY revenue, ANY o_orderdate, ANY o_shippriority): rowcount =
3002.85997, cumulative cost = {4051683.179997 rows,
3.094526417999211E7 cpu, 0.0 io, 0.0 network, 1.298667392002E7 memory}, id
= 3419
00-05 TopN(limit=[10]) : rowType = RecordType(ANY l_orderkey,
ANY revenue, ANY o_orderdate, ANY o_shippriority): rowcount =
3002.85997, cumulative cost = {4048680.32 rows, 3.094226131999211E7
cpu, 0.0 io, 0.0 network, 1.298667392002E7 memory}, id = 3418
00-06Project(l_orderkey=[$0], revenue=[$3],
o_orderdate=[$1], o_shippriority=[$2]) : rowType = RecordType(ANY l_orderkey,
ANY revenue, ANY o_orderdate, ANY o_shippriority): rowcount =
3002.85997, cumulative cost = {4045677.46 rows, 3.086245904003E7
cpu, 0.0 io, 0.0 network, 1.298667392002E7 memory}, id = 3417
00-07 HashAgg(group=[{0, 1, 2}], revenue=[SUM($3)]) :
rowType = RecordType(ANY l_orderkey, ANY o_orderdate, ANY o_shippriority, ANY
revenue): rowcount = 3002.85997, cumulative cost = {4042674.6
