[jira] [Commented] (DRILL-7458) Base storage plugin framework
[ https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331650#comment-17331650 ] Paul Rogers commented on DRILL-7458: This ticket is paused. The general idea is still valid, but the simple solution proposed in the PR was perhaps a bit too simple. There is another set of tickets about improving the storage plugin mechanism itself, and work was started on that. These efforts should be combined to create a "version 2" of the storage plugin mechanism that is a bit easier to implement, easier to test, and easier to install, version and so on. > Base storage plugin framework > - > > Key: DRILL-7458 > URL: https://issues.apache.org/jira/browse/DRILL-7458 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Labels: doc-impacting > > The "Easy" framework allows third-parties to add format plugins to Drill with > moderate effort. (The process could be easier, but "Easy" makes it as simple > as possible given the current structure.) > At present, no such "starter" framework exists for storage plugins. Further, > multiple storage plugins have implemented filter push down, seemingly by > copying large blocks of code. > This ticket offers a "base" framework for storage plugins and for filter > push-downs. The framework builds on the EVF, allowing plugins to also support > project push down. > The framework has a "test mule" storage plugin to verify functionality, and > was used as the basis of an REST-like plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[ https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034998#comment-17034998 ] ASF GitHub Bot commented on DRILL-7458: --- paul-rogers commented on pull request #1914: DRILL-7458: Base framework for storage plugins URL: https://github.com/apache/drill/pull/1914 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Base storage plugin framework > - > > Key: DRILL-7458 > URL: https://issues.apache.org/jira/browse/DRILL-7458 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Labels: doc-impacting > Fix For: 1.18.0 > > > The "Easy" framework allows third-parties to add format plugins to Drill with > moderate effort. (The process could be easier, but "Easy" makes it as simple > as possible given the current structure.) > At present, no such "starter" framework exists for storage plugins. Further, > multiple storage plugins have implemented filter push down, seemingly by > copying large blocks of code. > This ticket offers a "base" framework for storage plugins and for filter > push-downs. The framework builds on the EVF, allowing plugins to also support > project push down. > The framework has a "test mule" storage plugin to verify functionality, and > was used as the basis of an REST-like plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[ https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034997#comment-17034997 ] ASF GitHub Bot commented on DRILL-7458: --- paul-rogers commented on issue #1914: DRILL-7458: Base framework for storage plugins URL: https://github.com/apache/drill/pull/1914#issuecomment-584990522 As noted earlier, the general consensus is that we should come up with a better overall solution than the stop-gap offered here. So, I'll separate this PR into (as least) two. One will generalize the projection push-down to avoid the copy/paste solution we use today. For the plugins themselves, I'm looking into creating a proper "SPI" (service provider interface) based on Java's `ServiceLoader` pattern. Doing that requires digging much deeper into the existing plugin mechanism and refactoring that code to add a third storage plugin source (beyond the current classpath and system mechanisms.) Because this work will take time, I'd suggest that other PRs don't wait for this work. Let's get the other PRs in. Then, we can convert them to the new mechanism at some future time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Base storage plugin framework > - > > Key: DRILL-7458 > URL: https://issues.apache.org/jira/browse/DRILL-7458 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Labels: doc-impacting > Fix For: 1.18.0 > > > The "Easy" framework allows third-parties to add format plugins to Drill with > moderate effort. (The process could be easier, but "Easy" makes it as simple > as possible given the current structure.) > At present, no such "starter" framework exists for storage plugins. Further, > multiple storage plugins have implemented filter push down, seemingly by > copying large blocks of code. > This ticket offers a "base" framework for storage plugins and for filter > push-downs. The framework builds on the EVF, allowing plugins to also support > project push down. > The framework has a "test mule" storage plugin to verify functionality, and > was used as the basis of an REST-like plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027752#comment-17027752
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373639771
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[ https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027746#comment-17027746 ] ASF GitHub Bot commented on DRILL-7458: --- paul-rogers commented on issue #1914: DRILL-7458: Base framework for storage plugins URL: https://github.com/apache/drill/pull/1914#issuecomment-580867493 Thanks everyone for the review comments. I think there are enough substantial suggestions that we should go ahead and pause this PR. I'll divide this effort into smaller pieces, each of which is more complete to address the issues raised in reviews. This is all good news: shows that there is demand for an improved storage plugin API and that the present work is too modest an improvement. Let me figure out an alternative approach. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Base storage plugin framework > - > > Key: DRILL-7458 > URL: https://issues.apache.org/jira/browse/DRILL-7458 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Labels: doc-impacting > Fix For: 1.18.0 > > > The "Easy" framework allows third-parties to add format plugins to Drill with > moderate effort. (The process could be easier, but "Easy" makes it as simple > as possible given the current structure.) > At present, no such "starter" framework exists for storage plugins. Further, > multiple storage plugins have implemented filter push down, seemingly by > copying large blocks of code. > This ticket offers a "base" framework for storage plugins and for filter > push-downs. The framework builds on the EVF, allowing plugins to also support > project push down. > The framework has a "test mule" storage plugin to verify functionality, and > was used as the basis of an REST-like plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027691#comment-17027691
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373600104
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027683#comment-17027683
]
ASF GitHub Bot commented on DRILL-7458:
---
cgivre commented on pull request #1914: DRILL-7458: Base framework for storage
plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373594369
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a separate
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027664#comment-17027664
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373582218
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027662#comment-17027662
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373579329
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027444#comment-17027444
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373451734
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027160#comment-17027160
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373276549
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027161#comment-17027161
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373278001
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027153#comment-17027153
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373262171
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownStrategy.java
##
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.planner.common.DrillRelOptUtil;
+import org.apache.drill.exec.planner.logical.DrillFilterRel;
+import org.apache.drill.exec.planner.logical.DrillOptiq;
+import org.apache.drill.exec.planner.logical.DrillParseContext;
+import org.apache.drill.exec.planner.logical.DrillProjectRel;
+import org.apache.drill.exec.planner.logical.DrillScanRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+import org.apache.drill.exec.planner.physical.FilterPrel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+
+/**
+ * Generalized filter push-down strategy which performs all the tree-walking
+ * and tree restructuring work, allowing a "listener" to do the work needed
+ * for a particular scan.
+ *
+ * General usage in a storage plugin:
+ * public Set getPhysicalOptimizerRules(
+ *OptimizerRulesContext optimizerRulesContext) {
+ * return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+ * new MyPushDownListener(...));
+ * }
+ *
+ */
+
+public class FilterPushDownStrategy {
+
+ private static final Collection BANNED_OPERATORS =
+ Lists.newArrayList("flatten");
+
+ /**
+ * Base rule that passes target information to the push-down strategy
+ */
+
+ private static abstract class AbstractFilterPushDownRule extends
StoragePluginOptimizerRule {
+
+protected final FilterPushDownStrategy strategy;
+
+public AbstractFilterPushDownRule(RelOptRuleOperand operand, String
description,
+FilterPushDownStrategy strategy) {
+ super(operand, description);
+ this.strategy = strategy;
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> PROJECT --> SCAN
+ */
+
+ private static class ProjectAndFilterRule extends AbstractFilterPushDownRule
{
+
+private ProjectAndFilterRule(FilterPushDownStrategy strategy) {
+ super(RelOptHelper.some(FilterPrel.class,
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
+strategy.namePrefix() + "PushDownFilter:Filter_On_Project",
+strategy);
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+ if (!super.matches(call)) {
+return false;
+ }
+ DrillScanRel scan = call.rel(2);
+ return strategy.isTargetScan(scan);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ DrillFilterRel filterRel = call.rel(0);
+ DrillProjectRel projectRel = call.rel(1);
+ DrillScanRel scanRel = call.rel(2);
+ strategy.onMatch(call, filterRel, projectRel, scanRel);
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> SCAN
+ */
+
+ private static class FilterWithoutProjectRule extends
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027154#comment-17027154
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373263379
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseScanFactory.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+
+/**
+ * Creates the multiple classes needed to plan and execute a scan. Centralizes
+ * most implementation-specific object creation in one location to avoid
needing
+ * to implement many methods just for type-specific creation.
+ *
+ * @param the storage plugin class
+ * @param the scan specification class
+ * @param the group scan class
+ * @param the sub scan class
+ */
+
+public abstract class BaseScanFactory<
+ PLUGIN extends BaseStoragePlugin,
+ SPEC,
+ GROUP extends BaseGroupScan,
+ SUB extends BaseSubScan> {
+
+ @SuppressWarnings("unchecked")
Review comment:
If it helps, it turns out that Presto does make heavy use of casting in its
API. Typical example:
```
class MyFoo implements PrestoFoo {
void somemethod(PrestoBar bar) {
MyBar myBar = (MyBar) bar;
// Do something
}
}
```
At least in this code, the casting is hidden rather than in every method
implementation.
Would love to improve this if anyone has suggestions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027155#comment-17027155
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373270515
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/ConstantHolder.java
##
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.joda.time.DateTimeZone;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Description of a constant argument of an expression.
+ */
+
+@JsonPropertyOrder({"type", "value"})
+public class ConstantHolder {
+ @JsonProperty("type")
+ public final MinorType type;
+ @JsonProperty("value")
+ public final Object value;
+
+ @JsonCreator
+ public ConstantHolder(
+ @JsonProperty("type") MinorType type,
+ @JsonProperty("value") Object value) {
+this.type = type;
+this.value = value;
+ }
+
+ public ConstantHolder convertTo(MinorType toType) {
+if (type == toType) {
+ return this;
+}
+switch (toType) {
+case INT:
+ return toInt();
+case BIGINT:
+ return toBigInt();
+case TIMESTAMP:
+ return toTimestamp(null);
+case VARCHAR:
+ return toVarChar();
+default:
+ throw conversionError(toType);
+}
+ }
+
+ public ConstantHolder normalize(MinorType toType) {
+try {
+ return convertTo(toType);
+} catch (Throwable e) {
+ return null;
Review comment:
Added support for all literal types. Added conversion rules where they make
sense. Added a unit test to verify that the values are correctly "harvested"
from the logical plan literal nodes.
In doing this, an obvious question is: "why not just use the literal nodes"?
Two reasons. First, those nodes are not associated with the `MinorType`:
plugins would have to just know that association. Second, the literal nodes
would make it harder to convert types as we'd have to create "orphan" literal
nodes when doing conversion. Certainly possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027159#comment-17027159
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373276107
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027158#comment-17027158
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373269059
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
+ public static final String DEFAULT_SCHEMA_NAME = "default";
+
+ public static class StoragePluginOptions {
+
+/**
+ * Identifies if the plugin supports read.
+ * true by default.
+ */
+public boolean supportsRead = true;
+
+/**
+ * Identifies if the plugin supports writes (as in
+ * CREATE TABLE AS. false Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
>
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027156#comment-17027156
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373268557
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
Review comment:
I looked around. We create multiple mapper versions.
* `PhysicalPlanReader` constructor creates the most complete version.
* `DrillStatsTable.getMapper()` adds many custom serdes, but not all of them.
* Many, many others.
DRILL-7555 was filed to track this issue as it is too big a change to make
as part of this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027164#comment-17027164
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373291307
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestProjectPushDown.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestProjectPushDown extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config1 =
+new DummyStoragePluginConfig(true, false, true);
+pluginRegistry.createOrUpdate("dummy", config1, true);
+ }
+
+ @Test
+ public void testPushDownEnabled() throws Exception {
+String plan = client.queryBuilder().sql("SELECT a, b, c from
dummy.myTable").explainJson();
+// DRILL-7451: should be 0
+assertEquals(1, StringUtils.countMatches(plan, "\"pop\" : \"project\""));
+ }
+
+ @Test
+ public void testDummyReader() throws Exception {
+RowSet results = client.queryBuilder().sql("SELECT a, b, c from
dummy.myTable").rowSet();
+assertEquals(3, results.rowCount());
Review comment:
Filed DRILL-7560 to track the request for a generic solution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027162#comment-17027162
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373284322
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownStrategy.java
##
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.planner.common.DrillRelOptUtil;
+import org.apache.drill.exec.planner.logical.DrillFilterRel;
+import org.apache.drill.exec.planner.logical.DrillOptiq;
+import org.apache.drill.exec.planner.logical.DrillParseContext;
+import org.apache.drill.exec.planner.logical.DrillProjectRel;
+import org.apache.drill.exec.planner.logical.DrillScanRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+import org.apache.drill.exec.planner.physical.FilterPrel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+
+/**
+ * Generalized filter push-down strategy which performs all the tree-walking
+ * and tree restructuring work, allowing a "listener" to do the work needed
+ * for a particular scan.
+ *
+ * General usage in a storage plugin:
+ * public Set getPhysicalOptimizerRules(
+ *OptimizerRulesContext optimizerRulesContext) {
+ * return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+ * new MyPushDownListener(...));
+ * }
+ *
+ */
+
+public class FilterPushDownStrategy {
+
+ private static final Collection BANNED_OPERATORS =
+ Lists.newArrayList("flatten");
+
+ /**
+ * Base rule that passes target information to the push-down strategy
+ */
+
+ private static abstract class AbstractFilterPushDownRule extends
StoragePluginOptimizerRule {
+
+protected final FilterPushDownStrategy strategy;
+
+public AbstractFilterPushDownRule(RelOptRuleOperand operand, String
description,
+FilterPushDownStrategy strategy) {
+ super(operand, description);
+ this.strategy = strategy;
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> PROJECT --> SCAN
+ */
+
+ private static class ProjectAndFilterRule extends AbstractFilterPushDownRule
{
+
+private ProjectAndFilterRule(FilterPushDownStrategy strategy) {
+ super(RelOptHelper.some(FilterPrel.class,
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
+strategy.namePrefix() + "PushDownFilter:Filter_On_Project",
+strategy);
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+ if (!super.matches(call)) {
+return false;
+ }
+ DrillScanRel scan = call.rel(2);
+ return strategy.isTargetScan(scan);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ DrillFilterRel filterRel = call.rel(0);
+ DrillProjectRel projectRel = call.rel(1);
+ DrillScanRel scanRel = call.rel(2);
+ strategy.onMatch(call, filterRel, projectRel, scanRel);
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> SCAN
+ */
+
+ private static class FilterWithoutProjectRule extends
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027163#comment-17027163
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373289294
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyStoragePlugin.java
##
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+import java.util.Set;
+
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ReaderFactory;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.physical.impl.scan.framework.SchemaNegotiator;
+import org.apache.drill.exec.planner.PlannerPhase;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.exec.store.base.filter.RelOp;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+
+/**
+ * "Test mule" for the base storage plugin and the filter push down
+ * framework.
+ */
+
+public class DummyStoragePlugin
+ extends BaseStoragePlugin {
+
+ private static class DummyScanFactory extends
+BaseScanFactory {
+
+@Override
+public DummyGroupScan newGroupScan(DummyStoragePlugin storagePlugin,
+String userName, DummyScanSpec scanSpec,
+SessionOptionManager sessionOptions,
+MetadataProviderManager metadataProviderManager) {
+
+ // Force user name to "dummy" so golden and actual test files are stable
+ return new DummyGroupScan(storagePlugin, "dummy", scanSpec);
+}
+
+@Override
+public DummyGroupScan groupWithColumns(DummyGroupScan group,
+List columns) {
+ return new DummyGroupScan(group, columns);
+}
+
+@Override
+public ScanFrameworkBuilder scanBuilder(DummyStoragePlugin storagePlugin,
+OptionManager options, DummySubScan subScan) {
+ ScanFrameworkBuilder builder = new ScanFrameworkBuilder();
Review comment:
Added an explanation. Also filed DRILL-7559 to request that we provide a
general, reusable solution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027157#comment-17027157
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r373279889
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021373#comment-17021373
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369729458
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownStrategy.java
##
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.planner.common.DrillRelOptUtil;
+import org.apache.drill.exec.planner.logical.DrillFilterRel;
+import org.apache.drill.exec.planner.logical.DrillOptiq;
+import org.apache.drill.exec.planner.logical.DrillParseContext;
+import org.apache.drill.exec.planner.logical.DrillProjectRel;
+import org.apache.drill.exec.planner.logical.DrillScanRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+import org.apache.drill.exec.planner.physical.FilterPrel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+
+/**
+ * Generalized filter push-down strategy which performs all the tree-walking
+ * and tree restructuring work, allowing a "listener" to do the work needed
+ * for a particular scan.
+ *
+ * General usage in a storage plugin:
+ * public Set getPhysicalOptimizerRules(
+ *OptimizerRulesContext optimizerRulesContext) {
+ * return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+ * new MyPushDownListener(...));
+ * }
+ *
+ */
+
+public class FilterPushDownStrategy {
+
+ private static final Collection BANNED_OPERATORS =
+ Lists.newArrayList("flatten");
+
+ /**
+ * Base rule that passes target information to the push-down strategy
+ */
+
+ private static abstract class AbstractFilterPushDownRule extends
StoragePluginOptimizerRule {
+
+protected final FilterPushDownStrategy strategy;
+
+public AbstractFilterPushDownRule(RelOptRuleOperand operand, String
description,
+FilterPushDownStrategy strategy) {
+ super(operand, description);
+ this.strategy = strategy;
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> PROJECT --> SCAN
+ */
+
+ private static class ProjectAndFilterRule extends AbstractFilterPushDownRule
{
+
+private ProjectAndFilterRule(FilterPushDownStrategy strategy) {
+ super(RelOptHelper.some(FilterPrel.class,
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
+strategy.namePrefix() + "PushDownFilter:Filter_On_Project",
+strategy);
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+ if (!super.matches(call)) {
+return false;
+ }
+ DrillScanRel scan = call.rel(2);
+ return strategy.isTargetScan(scan);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ DrillFilterRel filterRel = call.rel(0);
+ DrillProjectRel projectRel = call.rel(1);
+ DrillScanRel scanRel = call.rel(2);
+ strategy.onMatch(call, filterRel, projectRel, scanRel);
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> SCAN
+ */
+
+ private static class FilterWithoutProjectRule extends
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[ https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021357#comment-17021357 ] ASF GitHub Bot commented on DRILL-7458: --- vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for storage plugins URL: https://github.com/apache/drill/pull/1914#discussion_r369673001 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/package-info.java ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +/** + * Provides a standard, reusable framework for implementing filter push-down + * for storage plugins. Handles the work of parsing a Drill physical plan + * (using Calcite rules) to extract candidate predicates, converting those + * to a normalized form, and calling a listener to check if the predicates + * are eligible for push-down, then to implement the push-down. + * + * Some plugins may which to remove the pushed conditions. That way, Drill Review comment: Looks like a mistake: may which This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Base storage plugin framework > - > > Key: DRILL-7458 > URL: https://issues.apache.org/jira/browse/DRILL-7458 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Labels: doc-impacting > Fix For: 1.18.0 > > > The "Easy" framework allows third-parties to add format plugins to Drill with > moderate effort. (The process could be easier, but "Easy" makes it as simple > as possible given the current structure.) > At present, no such "starter" framework exists for storage plugins. Further, > multiple storage plugins have implemented filter push down, seemingly by > copying large blocks of code. > This ticket offers a "base" framework for storage plugins and for filter > push-downs. The framework builds on the EVF, allowing plugins to also support > project push down. > The framework has a "test mule" storage plugin to verify functionality, and > was used as the basis of an REST-like plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021355#comment-17021355
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369663654
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownStrategy.java
##
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.planner.common.DrillRelOptUtil;
+import org.apache.drill.exec.planner.logical.DrillFilterRel;
+import org.apache.drill.exec.planner.logical.DrillOptiq;
+import org.apache.drill.exec.planner.logical.DrillParseContext;
+import org.apache.drill.exec.planner.logical.DrillProjectRel;
+import org.apache.drill.exec.planner.logical.DrillScanRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+import org.apache.drill.exec.planner.physical.FilterPrel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+
+/**
+ * Generalized filter push-down strategy which performs all the tree-walking
+ * and tree restructuring work, allowing a "listener" to do the work needed
+ * for a particular scan.
+ *
+ * General usage in a storage plugin:
+ * public Set getPhysicalOptimizerRules(
+ *OptimizerRulesContext optimizerRulesContext) {
+ * return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+ * new MyPushDownListener(...));
+ * }
+ *
+ */
+public class FilterPushDownStrategy {
+
+ private static final Collection BANNED_OPERATORS =
+ Collections.singletonList("flatten");
+
+ /**
+ * Base rule that passes target information to the push-down strategy
+ */
+ private static abstract class AbstractFilterPushDownRule extends
StoragePluginOptimizerRule {
+
+protected final FilterPushDownStrategy strategy;
+
+public AbstractFilterPushDownRule(RelOptRuleOperand operand, String
description,
+FilterPushDownStrategy strategy) {
+ super(operand, description);
+ this.strategy = strategy;
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> PROJECT --> SCAN
Review comment:
Please remove Calcite here, replace it with a planner or something like that
to avoid confusion. We use Calcite rules, but these rules are provided by
Calcite.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021359#comment-17021359
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369665119
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownStrategy.java
##
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.planner.common.DrillRelOptUtil;
+import org.apache.drill.exec.planner.logical.DrillFilterRel;
+import org.apache.drill.exec.planner.logical.DrillOptiq;
+import org.apache.drill.exec.planner.logical.DrillParseContext;
+import org.apache.drill.exec.planner.logical.DrillProjectRel;
+import org.apache.drill.exec.planner.logical.DrillScanRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+import org.apache.drill.exec.planner.physical.FilterPrel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+
+/**
+ * Generalized filter push-down strategy which performs all the tree-walking
+ * and tree restructuring work, allowing a "listener" to do the work needed
+ * for a particular scan.
+ *
+ * General usage in a storage plugin:
+ * public Set getPhysicalOptimizerRules(
+ *OptimizerRulesContext optimizerRulesContext) {
+ * return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+ * new MyPushDownListener(...));
+ * }
+ *
+ */
+public class FilterPushDownStrategy {
+
+ private static final Collection BANNED_OPERATORS =
+ Collections.singletonList("flatten");
+
+ /**
+ * Base rule that passes target information to the push-down strategy
+ */
+ private static abstract class AbstractFilterPushDownRule extends
StoragePluginOptimizerRule {
+
+protected final FilterPushDownStrategy strategy;
+
+public AbstractFilterPushDownRule(RelOptRuleOperand operand, String
description,
+FilterPushDownStrategy strategy) {
+ super(operand, description);
+ this.strategy = strategy;
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> PROJECT --> SCAN
+ */
+ private static class ProjectAndFilterRule extends AbstractFilterPushDownRule
{
+
+private ProjectAndFilterRule(FilterPushDownStrategy strategy) {
+ super(RelOptHelper.some(FilterPrel.class,
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
+strategy.namePrefix() + "PushDownFilter:Filter_On_Project",
+strategy);
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+ if (!super.matches(call)) {
+return false;
+ }
+ DrillScanRel scan = call.rel(2);
+ return strategy.isTargetScan(scan);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ DrillFilterRel filterRel = call.rel(0);
+ DrillProjectRel projectRel = call.rel(1);
+ DrillScanRel scanRel = call.rel(2);
+ strategy.onMatch(call, filterRel, projectRel, scanRel);
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> SCAN
+ */
+ private static class FilterWithoutProjectRule extends
AbstractFilterPushDownRule {
+
+private
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021356#comment-17021356
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369664401
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownStrategy.java
##
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.planner.common.DrillRelOptUtil;
+import org.apache.drill.exec.planner.logical.DrillFilterRel;
+import org.apache.drill.exec.planner.logical.DrillOptiq;
+import org.apache.drill.exec.planner.logical.DrillParseContext;
+import org.apache.drill.exec.planner.logical.DrillProjectRel;
+import org.apache.drill.exec.planner.logical.DrillScanRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+import org.apache.drill.exec.planner.physical.FilterPrel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+
+/**
+ * Generalized filter push-down strategy which performs all the tree-walking
+ * and tree restructuring work, allowing a "listener" to do the work needed
+ * for a particular scan.
+ *
+ * General usage in a storage plugin:
+ * public Set getPhysicalOptimizerRules(
+ *OptimizerRulesContext optimizerRulesContext) {
+ * return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+ * new MyPushDownListener(...));
+ * }
+ *
+ */
+public class FilterPushDownStrategy {
+
+ private static final Collection BANNED_OPERATORS =
+ Collections.singletonList("flatten");
+
+ /**
+ * Base rule that passes target information to the push-down strategy
+ */
+ private static abstract class AbstractFilterPushDownRule extends
StoragePluginOptimizerRule {
+
+protected final FilterPushDownStrategy strategy;
+
+public AbstractFilterPushDownRule(RelOptRuleOperand operand, String
description,
+FilterPushDownStrategy strategy) {
+ super(operand, description);
+ this.strategy = strategy;
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> PROJECT --> SCAN
+ */
+ private static class ProjectAndFilterRule extends AbstractFilterPushDownRule
{
+
+private ProjectAndFilterRule(FilterPushDownStrategy strategy) {
+ super(RelOptHelper.some(FilterPrel.class,
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
+strategy.namePrefix() + "PushDownFilter:Filter_On_Project",
+strategy);
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+ if (!super.matches(call)) {
+return false;
+ }
+ DrillScanRel scan = call.rel(2);
+ return strategy.isTargetScan(scan);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ DrillFilterRel filterRel = call.rel(0);
+ DrillProjectRel projectRel = call.rel(1);
+ DrillScanRel scanRel = call.rel(2);
+ strategy.onMatch(call, filterRel, projectRel, scanRel);
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> SCAN
+ */
+ private static class FilterWithoutProjectRule extends
AbstractFilterPushDownRule {
+
+private
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021360#comment-17021360
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369720302
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestFilterPushDown.java
##
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * Tests for the Filter push-down helper classes as part of the
+ * "Base" storage plugin to be used for add-on plugins. Uses a
+ * "test mule" ("Dummy") plug-in which goes through the paces of
+ * planning push down, but then glosses over the details at run time.
+ * The focus here are plans: the tests plan a query then compare the
+ * actual plan against and expected ("golden") plan.
+ */
+public class TestFilterPushDown extends ClusterTest {
+
+ private static final String BASE_SQL = "SELECT a, b FROM dummy.myTable";
+ private static final String BASE_WHERE = BASE_SQL + " WHERE ";
+ private static final PlanVerifier verifier = new
PlanVerifier("/store/base/");
+
+ // Uncomment the next line to save failing plans to /tmp
+ // static { verifier.saveResults = true; }
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+DummyStoragePluginConfig config =
+new DummyStoragePluginConfig(true, true, false);
+cluster.defineStoragePlugin("dummy", config);
+ }
+
+ //-
+ // Unsupported filter push-down cases
+
+ // No predicates
+
+ @Test
+ public void testNoPushDown() throws Exception
+ {
Review comment:
Please move brace to the same line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[ https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021358#comment-17021358 ] ASF GitHub Bot commented on DRILL-7458: --- vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for storage plugins URL: https://github.com/apache/drill/pull/1914#discussion_r369689965 ## File path: exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyBatchReader.java ## @@ -0,0 +1,83 @@ +package org.apache.drill.exec.store.base; Review comment: Please move it below license. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Base storage plugin framework > - > > Key: DRILL-7458 > URL: https://issues.apache.org/jira/browse/DRILL-7458 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Labels: doc-impacting > Fix For: 1.18.0 > > > The "Easy" framework allows third-parties to add format plugins to Drill with > moderate effort. (The process could be easier, but "Easy" makes it as simple > as possible given the current structure.) > At present, no such "starter" framework exists for storage plugins. Further, > multiple storage plugins have implemented filter push down, seemingly by > copying large blocks of code. > This ticket offers a "base" framework for storage plugins and for filter > push-downs. The framework builds on the EVF, allowing plugins to also support > project push down. > The framework has a "test mule" storage plugin to verify functionality, and > was used as the basis of an REST-like plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021234#comment-17021234
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369654393
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021174#comment-17021174
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369617363
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL {@code IN} clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+public interface FilterPushDownListener {
+
+ /**
+ * @return a prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ * @return true if the given group scan is one this listener can
+ * handle, false otherwise
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param groupScan the scan node
+ * @return true if filter push-down should be applied
+ */
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * {@code }, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param groupScan the scan element. Use {@code scan.getGroupScan()}
+ * to get the group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, @{code null} if not and thus the relop should remain in
+ * the Drill plan
+ * @see {@link
ConstantHolder#normalize(org.apache.drill.common.types.TypeProtos.MinorType)}
+ */
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each {@code OR} term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020955#comment-17020955
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369492043
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020901#comment-17020901
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369434855
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyGroupScan.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.filter.FilterSpec;
+import org.apache.drill.exec.store.base.filter.RelOp;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+@JsonTypeName("dummy-scan")
+@JsonPropertyOrder({"userName", "scanSpec", "columns",
+"filters", "cost", "config"})
+@JsonInclude(value=Include.NON_EMPTY, content=Include.NON_NULL)
+public class DummyGroupScan extends BaseGroupScan {
+
+ private final DummyScanSpec scanSpec;
+ private final FilterSpec filters;
+
+ public DummyGroupScan(DummyStoragePlugin storagePlugin, String userName,
+ DummyScanSpec scanSpec) {
+super(storagePlugin, userName, null);
+this.scanSpec = scanSpec;
+filters = null;
+ }
+
+ public DummyGroupScan(DummyGroupScan from, List columns) {
+super(from.storagePlugin, from.getUserName(), columns);
+this.scanSpec = from.scanSpec;
+this.filters = from.filters;
+ }
+
+ @JsonCreator
+ public DummyGroupScan(
+ @JsonProperty("config") DummyStoragePluginConfig config,
+ @JsonProperty("userName") String userName,
+ @JsonProperty("scanSpec") DummyScanSpec scanSpec,
+ @JsonProperty("columns") List columns,
+ @JsonProperty("filters") FilterSpec filters,
+ @JacksonInject StoragePluginRegistry engineRegistry) {
+super(config, userName, columns, engineRegistry);
+this.scanSpec = scanSpec;
+this.filters = filters;
+ }
+
+ public DummyGroupScan(DummyGroupScan from, FilterSpec filters) {
+super(from);
+this.scanSpec = from.scanSpec;
+this.filters = filters;
+ }
+
+ @JsonProperty("scanSpec")
+ public DummyScanSpec scanSpec() { return scanSpec; }
+
+ @JsonProperty("filters")
+ public FilterSpec andFilters() { return filters; }
+
+ public boolean hasFilters() {
+return filters != null && ! filters.isEmpty();
+ }
+
+ private static final List FILTER_COLS = ImmutableList.of("a", "b",
"id");
Review comment:
Use `Arrays.asList`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020899#comment-17020899
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369426397
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/ConstantHolder.java
##
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.joda.time.DateTimeZone;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Description of a constant argument of an expression.
+ */
+
+@JsonPropertyOrder({"type", "value"})
+public class ConstantHolder {
+ @JsonProperty("type")
+ public final MinorType type;
+ @JsonProperty("value")
+ public final Object value;
+
+ @JsonCreator
+ public ConstantHolder(
+ @JsonProperty("type") MinorType type,
+ @JsonProperty("value") Object value) {
+this.type = type;
+this.value = value;
+ }
+
+ public ConstantHolder convertTo(MinorType toType) {
+if (type == toType) {
+ return this;
+}
+switch (toType) {
+case INT:
+ return toInt();
+case BIGINT:
+ return toBigInt();
+case TIMESTAMP:
+ return toTimestamp(null);
+case VARCHAR:
+ return toVarChar();
+default:
+ throw conversionError(toType);
+}
+ }
+
+ public ConstantHolder normalize(MinorType toType) {
+try {
+ return convertTo(toType);
+} catch (Throwable e) {
+ return null;
Review comment:
What is others need to push down more types? What they should do?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020898#comment-17020898
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369433958
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseScanFactory.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+
+/**
+ * Creates the multiple classes needed to plan and execute a scan. Centralizes
+ * most implementation-specific object creation in one location to avoid
needing
+ * to implement many methods just for type-specific creation.
+ *
+ * @param the storage plugin class
+ * @param the scan specification class
+ * @param the group scan class
+ * @param the sub scan class
+ */
+
+public abstract class BaseScanFactory<
+ PLUGIN extends BaseStoragePlugin,
+ SPEC,
+ GROUP extends BaseGroupScan,
+ SUB extends BaseSubScan> {
+
+ @SuppressWarnings("unchecked")
Review comment:
I see, I guess we don't have netter solution than casting...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020897#comment-17020897
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369433503
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestFilterPushDown.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * Tests for the Filter push-down helper classes as part of the
+ * "Base" storage plugin to be used for add-on plugins. Uses a
+ * "test mule" ("Dummy") plug-in which goes through the paces of
+ * planning push down, but then glosses over the details at run time.
+ * The focus here are plans: the tests plan a query then compare the
+ * actual plan against and expected ("golden") plan.
+ *
+ * If comparison fails, the tests print the actual plan to the console.
+ * Use this, when adding new tests, to create the initial "golden" file.
+ * Also, on failures, actual output is written to
+ * /tmp/drill/store/base. You can use your IDE to compare
+ * the actual and golden files to understand changes. If the changes
+ * are expected, use that same IDE to copy changes from the actual
+ * to the golden file.
+ *
+ * The JSON properties of the serialized classes are all controlled
+ * to have a fixed order to ensure that files compare across test
+ * runs.
+ */
+public class TestFilterPushDown extends ClusterTest {
+
+ private static final String BASE_SQL = "SELECT a, b FROM dummy.myTable";
+ private static final String BASE_WHERE = BASE_SQL + " WHERE ";
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config =
+new DummyStoragePluginConfig(true, true, false);
+pluginRegistry.createOrUpdate("dummy", config, true);
Review comment:
It is used in JDBC plugin tests, please leave only one verions than yours or
previous one, I don't think we don't need two.
https://github.com/apache/drill/blob/465fae6b9dcea9e02a4bfb71d9a3b9d4b99e169d/contrib/storage-jdbc/src/test/java/org/apache/drill/exec/store/jdbc/TestJdbcPluginWithH2IT.java#L68
https://github.com/apache/drill/blob/465fae6b9dcea9e02a4bfb71d9a3b9d4b99e169d/contrib/storage-jdbc/src/test/java/org/apache/drill/exec/store/jdbc/TestJdbcPluginWithMySQLIT.java#L85
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020892#comment-17020892
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369429278
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/PlanVerifier.java
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.test.ClientFixture;
+
+/**
+ * Verifier for execution plans. A handy tool to ensure that the
+ * planner produces the expected plan given some set of conditions.
+ *
+ * The test works by comparing the actual {@code EXPLAIN} output
+ * to a "golden" file with the expected ("golden") plan.
+ *
+ * To create a test, just write it and let it fail due to a missing file.
+ * The output will go to the console. Inspect it. If it looks good,
+ * copy the plan to the golden file and run again.
+ *
+ * If comparison fails, you can optionally ask the verifier to write the
+ * output to {@code /tmp} so you can compare the golden and actual
+ * outputs using your favorite diff tool to understand changes. If the changes
+ * are expected, use that same IDE to copy changes from the actual
+ * to the golden file.
+ *
+ * The JSON properties of the serialized classes are all controlled
+ * to have a fixed order to ensure that files compare across test
+ * runs. If you see spurious failures do to changed JSON order, consider
+ * adding a {@code @JsonPropertyOrder} tag to enforce a consistent order.
+ *
+ * A fancier version of this class would use a regex or other mechanism
+ * to say "ignore differences in this bit of the plan." For example, when
+ * using systems with metadata, the exact values might bounce around
+ * some.
+ */
+public class PlanVerifier {
+
+ private final String basePath;
+
+ /**
+ * Persist test results. Turn this on if you have failing tests.
+ ( Then, you can diff the actual and golden results in your favorite
+ * tool or IDE.
+ */
+ public boolean saveResults;
Review comment:
Maybe add setter and make it private.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020891#comment-17020891
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369428809
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/PlanVerifier.java
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.test.ClientFixture;
+
+/**
+ * Verifier for execution plans. A handy tool to ensure that the
+ * planner produces the expected plan given some set of conditions.
+ *
+ * The test works by comparing the actual {@code EXPLAIN} output
+ * to a "golden" file with the expected ("golden") plan.
+ *
+ * To create a test, just write it and let it fail due to a missing file.
+ * The output will go to the console. Inspect it. If it looks good,
+ * copy the plan to the golden file and run again.
+ *
+ * If comparison fails, you can optionally ask the verifier to write the
+ * output to {@code /tmp} so you can compare the golden and actual
Review comment:
```suggestion
* output to {@code /tmp/drill/test} so you can compare the golden and actual
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020890#comment-17020890
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369428346
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyGroupScan.java
##
@@ -97,36 +100,33 @@ public RelOp acceptFilter(RelOp relOp) {
// Pretend that "id" is a special integer column. Can handle
// equality only.
-
if (relOp.colName.contentEquals("id")) {
// To allow easier testing, require exact type match: no
// attempt at type conversion here.
-
if (relOp.op != RelOp.Op.EQ || relOp.value.type != MinorType.INT) {
return null;
}
return relOp;
}
-// All other columns apply only if projected
-
-if (!FILTER_COLS.contains(relOp.colName)) {
+// "allTypes" table filters everything. All other tables
+// only project a fixed set of columns. Simulates a plugin
+// which can project only some columns.
+if (!scanSpec.tableName.equals("allTypes") &&
!FILTER_COLS.contains(relOp.colName)) {
Review comment:
Nit: usually constant goes first to avoid NPE `"allTypes".equals(...`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020888#comment-17020888
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369426743
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyStoragePlugin.java
##
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+import java.util.Set;
+
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ReaderFactory;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.physical.impl.scan.framework.SchemaNegotiator;
+import org.apache.drill.exec.planner.PlannerPhase;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.exec.store.base.filter.RelOp;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+
+/**
+ * "Test mule" for the base storage plugin and the filter push down
+ * framework.
+ */
+
+public class DummyStoragePlugin
+ extends BaseStoragePlugin {
+
+ private static class DummyScanFactory extends
+BaseScanFactory {
+
+@Override
+public DummyGroupScan newGroupScan(DummyStoragePlugin storagePlugin,
+String userName, DummyScanSpec scanSpec,
+SessionOptionManager sessionOptions,
+MetadataProviderManager metadataProviderManager) {
+
+ // Force user name to "dummy" so golden and actual test files are stable
+ return new DummyGroupScan(storagePlugin, "dummy", scanSpec);
+}
+
+@Override
+public DummyGroupScan groupWithColumns(DummyGroupScan group,
+List columns) {
+ return new DummyGroupScan(group, columns);
+}
+
+@Override
+public ScanFrameworkBuilder scanBuilder(DummyStoragePlugin storagePlugin,
+OptionManager options, DummySubScan subScan) {
+ ScanFrameworkBuilder builder = new ScanFrameworkBuilder();
Review comment:
Maybe add comment in Java doc somewhere? I could not have guessed this
without your comment...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020886#comment-17020886
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369426397
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/ConstantHolder.java
##
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.joda.time.DateTimeZone;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Description of a constant argument of an expression.
+ */
+
+@JsonPropertyOrder({"type", "value"})
+public class ConstantHolder {
+ @JsonProperty("type")
+ public final MinorType type;
+ @JsonProperty("value")
+ public final Object value;
+
+ @JsonCreator
+ public ConstantHolder(
+ @JsonProperty("type") MinorType type,
+ @JsonProperty("value") Object value) {
+this.type = type;
+this.value = value;
+ }
+
+ public ConstantHolder convertTo(MinorType toType) {
+if (type == toType) {
+ return this;
+}
+switch (toType) {
+case INT:
+ return toInt();
+case BIGINT:
+ return toBigInt();
+case TIMESTAMP:
+ return toTimestamp(null);
+case VARCHAR:
+ return toVarChar();
+default:
+ throw conversionError(toType);
+}
+ }
+
+ public ConstantHolder normalize(MinorType toType) {
+try {
+ return convertTo(toType);
+} catch (Throwable e) {
+ return null;
Review comment:
What is others need to push down more? What they should do?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020885#comment-17020885
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369426136
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
Review comment:
Well, unless it is static would would have to inject it using constructor or
though some context (for example, DrillbitContext).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020884#comment-17020884
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369425418
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
+ public static final String DEFAULT_SCHEMA_NAME = "default";
+
+ public static class StoragePluginOptions {
+
+/**
+ * Identifies if the plugin supports read.
+ * true by default.
+ */
+public boolean supportsRead = true;
+
+/**
+ * Identifies if the plugin supports writes (as in
+ * CREATE TABLE AS. false Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality,
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020883#comment-17020883
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369425418
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
+ public static final String DEFAULT_SCHEMA_NAME = "default";
+
+ public static class StoragePluginOptions {
+
+/**
+ * Identifies if the plugin supports read.
+ * true by default.
+ */
+public boolean supportsRead = true;
+
+/**
+ * Identifies if the plugin supports writes (as in
+ * CREATE TABLE AS. false Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality,
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020782#comment-17020782
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369362626
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestFilterPushDown.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * Tests for the Filter push-down helper classes as part of the
+ * "Base" storage plugin to be used for add-on plugins. Uses a
+ * "test mule" ("Dummy") plug-in which goes through the paces of
+ * planning push down, but then glosses over the details at run time.
+ * The focus here are plans: the tests plan a query then compare the
+ * actual plan against and expected ("golden") plan.
+ *
+ * If comparison fails, the tests print the actual plan to the console.
+ * Use this, when adding new tests, to create the initial "golden" file.
+ * Also, on failures, actual output is written to
+ * /tmp/drill/store/base. You can use your IDE to compare
+ * the actual and golden files to understand changes. If the changes
+ * are expected, use that same IDE to copy changes from the actual
+ * to the golden file.
+ *
+ * The JSON properties of the serialized classes are all controlled
+ * to have a fixed order to ensure that files compare across test
+ * runs.
+ */
+public class TestFilterPushDown extends ClusterTest {
+
+ private static final String BASE_SQL = "SELECT a, b FROM dummy.myTable";
+ private static final String BASE_WHERE = BASE_SQL + " WHERE ";
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config =
+new DummyStoragePluginConfig(true, true, false);
+pluginRegistry.createOrUpdate("dummy", config, true);
+ }
+
+ //-
+ // Unsupported filter push-down cases
+
+ // No predicates
+
+ @Test
+ public void testNoPushDown() throws Exception
+ {
+verifyPlan(BASE_SQL, "noPushDown.json");
+ }
+
+ // Predicate mismatch on type (id should be INT, dummy does
+ // not try to convert)
+
+ @Test
+ public void testTypeMismatch() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id = 'foo'", "typeMismatch.json");
+ }
+
+ // Unsupported relop type (dummy supports limited set)
+
+ @Test
+ public void testUnsupportedOp() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a <> 'foo'", "unsupportedOp.json");
+ }
+
+ // Column reference rather than constant
+
+ @Test
+ public void testNonConst() throws Exception
+ {
+ verifyPlan(BASE_WHERE + "a = b", "nonConstPred.json");
+ }
+
+ // Unknown column (dummy only knows columns a and b)
+
+ @Test
+ public void testUnsupportedCol() throws Exception
+ {
+verifyPlan(BASE_WHERE + "c = 'foo'", "unsupportedColPred.json");
+ }
+
+ // Not simple col = const
+
+ @Test
+ public void testComplexPred() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id + 10 = 20", "complexPred.json");
+ }
+
+ // Complex schema paths
+
+ @Test
+ public void testComplexCols() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a[10] = 'foo' AND
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020772#comment-17020772
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369224588
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020770#comment-17020770
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369224013
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020774#comment-17020774
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369337169
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/ConstantHolder.java
##
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.joda.time.DateTimeZone;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Description of a constant argument of an expression.
+ */
+
+@JsonPropertyOrder({"type", "value"})
+public class ConstantHolder {
+ @JsonProperty("type")
+ public final MinorType type;
+ @JsonProperty("value")
+ public final Object value;
+
+ @JsonCreator
+ public ConstantHolder(
+ @JsonProperty("type") MinorType type,
+ @JsonProperty("value") Object value) {
+this.type = type;
+this.value = value;
+ }
+
+ public ConstantHolder convertTo(MinorType toType) {
+if (type == toType) {
+ return this;
+}
+switch (toType) {
+case INT:
+ return toInt();
+case BIGINT:
+ return toBigInt();
+case TIMESTAMP:
+ return toTimestamp(null);
+case VARCHAR:
+ return toVarChar();
+default:
+ throw conversionError(toType);
+}
+ }
+
+ public ConstantHolder normalize(MinorType toType) {
+try {
+ return convertTo(toType);
+} catch (Throwable e) {
+ return null;
Review comment:
Added a comment. The idea here is that my plugin can say something like "I
can push down a value if can be a `BIGINT` but I don't know how to handle
other types. The value itself could expressed in the query as:
```
c = 10 -- INT
c = 50 -- BIGINT
c = 10.0 -- VARDECIMAL
c = '10' - VARCHAR
```
The idea here is that the constant holder will convert if valid, return
`null` if not. If the value is `null`, we just ignore that filter and let Drill
handle it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020779#comment-17020779
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369342360
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownStrategy.java
##
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.planner.common.DrillRelOptUtil;
+import org.apache.drill.exec.planner.logical.DrillFilterRel;
+import org.apache.drill.exec.planner.logical.DrillOptiq;
+import org.apache.drill.exec.planner.logical.DrillParseContext;
+import org.apache.drill.exec.planner.logical.DrillProjectRel;
+import org.apache.drill.exec.planner.logical.DrillScanRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+import org.apache.drill.exec.planner.physical.FilterPrel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+
+/**
+ * Generalized filter push-down strategy which performs all the tree-walking
+ * and tree restructuring work, allowing a "listener" to do the work needed
+ * for a particular scan.
+ *
+ * General usage in a storage plugin:
+ * public Set getPhysicalOptimizerRules(
+ *OptimizerRulesContext optimizerRulesContext) {
+ * return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+ * new MyPushDownListener(...));
+ * }
+ *
+ */
+
+public class FilterPushDownStrategy {
+
+ private static final Collection BANNED_OPERATORS =
+ Lists.newArrayList("flatten");
+
+ /**
+ * Base rule that passes target information to the push-down strategy
+ */
+
+ private static abstract class AbstractFilterPushDownRule extends
StoragePluginOptimizerRule {
+
+protected final FilterPushDownStrategy strategy;
+
+public AbstractFilterPushDownRule(RelOptRuleOperand operand, String
description,
+FilterPushDownStrategy strategy) {
+ super(operand, description);
+ this.strategy = strategy;
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> PROJECT --> SCAN
+ */
+
+ private static class ProjectAndFilterRule extends AbstractFilterPushDownRule
{
+
+private ProjectAndFilterRule(FilterPushDownStrategy strategy) {
+ super(RelOptHelper.some(FilterPrel.class,
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
+strategy.namePrefix() + "PushDownFilter:Filter_On_Project",
+strategy);
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+ if (!super.matches(call)) {
+return false;
+ }
+ DrillScanRel scan = call.rel(2);
+ return strategy.isTargetScan(scan);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ DrillFilterRel filterRel = call.rel(0);
+ DrillProjectRel projectRel = call.rel(1);
+ DrillScanRel scanRel = call.rel(2);
+ strategy.onMatch(call, filterRel, projectRel, scanRel);
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> SCAN
+ */
+
+ private static class FilterWithoutProjectRule extends
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020787#comment-17020787
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369364526
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestProjectPushDown.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestProjectPushDown extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config1 =
+new DummyStoragePluginConfig(true, false, true);
+pluginRegistry.createOrUpdate("dummy", config1, true);
+ }
+
+ @Test
+ public void testPushDownEnabled() throws Exception {
+String plan = client.queryBuilder().sql("SELECT a, b, c from
dummy.myTable").explainJson();
+// DRILL-7451: should be 0
+assertEquals(1, StringUtils.countMatches(plan, "\"pop\" : \"project\""));
+ }
+
+ @Test
+ public void testDummyReader() throws Exception {
+RowSet results = client.queryBuilder().sql("SELECT a, b, c from
dummy.myTable").rowSet();
+assertEquals(3, results.rowCount());
Review comment:
No. But that is true in all tests. If a test fails, we leave vectors in
memory and we get messages about allocators that still have allocations.
Would be very messy to try to clean this up in every test. And, after all,
tests are not supposed to fail.
If we wanted to solve this, a better solution would be to come up with a way
to walk the allocator tree and free any remaining resources. Call this when we
shut down the test cluster.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020771#comment-17020771
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369227180
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseScanFactory.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+
+/**
+ * Creates the multiple classes needed to plan and execute a scan. Centralizes
+ * most implementation-specific object creation in one location to avoid
needing
+ * to implement many methods just for type-specific creation.
+ *
+ * @param the storage plugin class
+ * @param the scan specification class
+ * @param the group scan class
+ * @param the sub scan class
+ */
+
+public abstract class BaseScanFactory<
+ PLUGIN extends BaseStoragePlugin,
+ SPEC,
+ GROUP extends BaseGroupScan,
+ SUB extends BaseSubScan> {
+
+ @SuppressWarnings("unchecked")
Review comment:
So here is the thing. The storage plugin mechanism is awkwardly designed.
This class is trying to hide that fact. Base classes call into methods we
define, passing us generic classes, such as the plugin class. However, the
methods for any one plugin want to use plugin-specific classes.
In the past, we just cast wherever needed. Here, I'm trying to use
parameterized classes to create a "walled garden" in which methods use
plugin-specific classes. To do that, we do the casting behind the scenes: from
base classes to user-defined classes represented as parameters.
We *could* require that the user give us the `Class` objects for their
classes so that we can do type-safe casting. But, since all the casting is
static (the types don't change often), this old-school casting seems
sufficient. If something is wrong, it will blow up on the first test.
Does this make sense? Is there a better way to solve these same issues?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020781#comment-17020781
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369346139
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterSpec.java
##
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+
+@JsonInclude(value=Include.NON_EMPTY, content=Include.NON_NULL)
+@JsonPropertyOrder({"andTerms", "orTerms"})
+public class FilterSpec {
+
+ private final List andTerms;
+ private final DisjunctionFilterSpec orTerms;
+
+ @JsonCreator
+ public FilterSpec(
+ @JsonProperty("andTerms") List andFilters,
+ @JsonProperty("orTerms") DisjunctionFilterSpec orFilters) {
+this.andTerms = andFilters == null || andFilters.isEmpty() ? null :
andFilters;
+this.orTerms = orFilters == null || orFilters.values.length == 0 ? null :
orFilters;
+ }
+
+ public static FilterSpec build(List andFilters, DisjunctionFilterSpec
orFilters) {
+FilterSpec spec = new FilterSpec(andFilters, orFilters);
+return spec.isEmpty() ? null : spec;
+ }
+
+ @JsonProperty("andTerms")
+ public List andTerms() { return andTerms; }
+
+ @JsonProperty("orTerms")
+ public DisjunctionFilterSpec orTerms() { return orTerms; }
+
+ /**
+ * The number of partitions in the sense of scan "segments" given
+ * by an OR clause. (May not correspond to HDFS file partitions.)
+ *
+ * @return the number of logical partitions, which is the number
+ * of OR clause terms (or 1 if no OR terms exist)
+ */
+
+ public int partitionCount() {
+return orTerms == null ? 1 : orTerms.values.length;
+ }
+
+ public static int parititonCount(FilterSpec filters) {
+return filters == null ? 1 : filters.partitionCount();
+ }
+
+ public List distribute(int orTerm) {
+return DisjunctionFilterSpec.distribute(andTerms, orTerms, orTerm);
+ }
+
+ /**
+ * Compute selectivity of a CNF form of equality conditions. Without stats,
+ * Drill assumes a selectivity of 0.15 for each equality, then multiplies
+ * the selectivity of multiple columns. Place a lower limit of 0.001 on
+ * the result, assuming the user wants to return some rows.
+ *
+ * @param andTerms list of AND filters (in CNF form), or null if no
+ * filters apply to a scan
+ * @return selectivity of the filters
+ */
+
+ public double cnfSelectivity() {
+if (andTerms == null || andTerms.isEmpty()) {
+ return 1.0;
+}
+double selectivity = 1.0;
+for (RelOp relOp : andTerms) {
+ selectivity *= relOp.op.selectivity();
+}
+return Math.max(0.001, selectivity);
+ }
+
+ /**
+ * Compute the combined selectivity of a set of AND (CNF) and OR
+ * (DNF) filters. We assume the DNF returns a subset of rows, which
+ * the CNF terms further reduce. Selectivity will be at least
+ * 0.001, which assumes the user wants to return some rows.
+ *
+ * @param andTerms and filters for the scan, or null if none
+ * @param orTerms or filters for the scan, or null if none
+ * @return combined selectivity
+ */
+
+ public double selectivity() {
+double selectivity = cnfSelectivity();
+if (orTerms != null) {
+ selectivity = Math.max(0.001, selectivity * orTerms.selectivity());
+}
+return selectivity;
+ }
+
+ /**
+ * Drill scan stats want a row count, the selectivity is a means to
+ * that end. Apply the selectivity of a set of filters to the given
+ * row count to produce a reduced row
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020784#comment-17020784
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369354317
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyGroupScan.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.filter.FilterSpec;
+import org.apache.drill.exec.store.base.filter.RelOp;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+@JsonTypeName("dummy-scan")
+@JsonPropertyOrder({"userName", "scanSpec", "columns",
+"filters", "cost", "config"})
+@JsonInclude(value=Include.NON_EMPTY, content=Include.NON_NULL)
+public class DummyGroupScan extends BaseGroupScan {
+
+ private final DummyScanSpec scanSpec;
+ private final FilterSpec filters;
+
+ public DummyGroupScan(DummyStoragePlugin storagePlugin, String userName,
+ DummyScanSpec scanSpec) {
+super(storagePlugin, userName, null);
+this.scanSpec = scanSpec;
+filters = null;
+ }
+
+ public DummyGroupScan(DummyGroupScan from, List columns) {
+super(from.storagePlugin, from.getUserName(), columns);
+this.scanSpec = from.scanSpec;
+this.filters = from.filters;
+ }
+
+ @JsonCreator
+ public DummyGroupScan(
+ @JsonProperty("config") DummyStoragePluginConfig config,
+ @JsonProperty("userName") String userName,
+ @JsonProperty("scanSpec") DummyScanSpec scanSpec,
+ @JsonProperty("columns") List columns,
+ @JsonProperty("filters") FilterSpec filters,
+ @JacksonInject StoragePluginRegistry engineRegistry) {
+super(config, userName, columns, engineRegistry);
+this.scanSpec = scanSpec;
+this.filters = filters;
+ }
+
+ public DummyGroupScan(DummyGroupScan from, FilterSpec filters) {
+super(from);
+this.scanSpec = from.scanSpec;
+this.filters = filters;
+ }
+
+ @JsonProperty("scanSpec")
+ public DummyScanSpec scanSpec() { return scanSpec; }
+
+ @JsonProperty("filters")
+ public FilterSpec andFilters() { return filters; }
+
+ public boolean hasFilters() {
+return filters != null && ! filters.isEmpty();
+ }
+
+ private static final List FILTER_COLS = ImmutableList.of("a", "b",
"id");
Review comment:
I'm probably being dense, but I didn't see a `Collections` list builder that
takes multiple items as varargs, only `Collections.singletonList()`. Is there
one somewhere?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project:
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020783#comment-17020783
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369364072
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestProjectPushDown.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestProjectPushDown extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config1 =
+new DummyStoragePluginConfig(true, false, true);
+pluginRegistry.createOrUpdate("dummy", config1, true);
+ }
+
+ @Test
+ public void testPushDownEnabled() throws Exception {
+String plan = client.queryBuilder().sql("SELECT a, b, c from
dummy.myTable").explainJson();
+// DRILL-7451: should be 0
+assertEquals(1, StringUtils.countMatches(plan, "\"pop\" : \"project\""));
Review comment:
Nice! Added the JSON version. However, won't work here because I need a
count.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020775#comment-17020775
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369228790
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
+ public static final String DEFAULT_SCHEMA_NAME = "default";
+
+ public static class StoragePluginOptions {
+
+/**
+ * Identifies if the plugin supports read.
+ * true by default.
+ */
+public boolean supportsRead = true;
+
+/**
+ * Identifies if the plugin supports writes (as in
+ * CREATE TABLE AS. false Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
>
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020785#comment-17020785
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369355344
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyStoragePlugin.java
##
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+import java.util.Set;
+
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ReaderFactory;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.physical.impl.scan.framework.SchemaNegotiator;
+import org.apache.drill.exec.planner.PlannerPhase;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.exec.store.base.filter.RelOp;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+
+/**
+ * "Test mule" for the base storage plugin and the filter push down
+ * framework.
+ */
+
+public class DummyStoragePlugin
+ extends BaseStoragePlugin {
+
+ private static class DummyScanFactory extends
+BaseScanFactory {
+
+@Override
+public DummyGroupScan newGroupScan(DummyStoragePlugin storagePlugin,
+String userName, DummyScanSpec scanSpec,
+SessionOptionManager sessionOptions,
+MetadataProviderManager metadataProviderManager) {
+
+ // Force user name to "dummy" so golden and actual test files are stable
+ return new DummyGroupScan(storagePlugin, "dummy", scanSpec);
+}
+
+@Override
+public DummyGroupScan groupWithColumns(DummyGroupScan group,
+List columns) {
+ return new DummyGroupScan(group, columns);
+}
+
+@Override
+public ScanFrameworkBuilder scanBuilder(DummyStoragePlugin storagePlugin,
+OptionManager options, DummySubScan subScan) {
+ ScanFrameworkBuilder builder = new ScanFrameworkBuilder();
Review comment:
We can probably make this easier. Somehow the plugin has to obtain a schema;
we don't seem to provide a general-purpose registry. Maybe we need one that
works like the storage and format plugin registries?
Presumably, the schema factory would look up the schema at the point that it
looks up the table name. It would be passed (via the scan spec) to the group
scan, and from there to the sub scan.
Then, in this method, given a schema in the sub scan, we add one line:
```
builder.typeConverterBuilder().providedSchema(subScan.getSchema());
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter:
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020786#comment-17020786
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369362324
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestFilterPushDown.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * Tests for the Filter push-down helper classes as part of the
+ * "Base" storage plugin to be used for add-on plugins. Uses a
+ * "test mule" ("Dummy") plug-in which goes through the paces of
+ * planning push down, but then glosses over the details at run time.
+ * The focus here are plans: the tests plan a query then compare the
+ * actual plan against and expected ("golden") plan.
+ *
+ * If comparison fails, the tests print the actual plan to the console.
+ * Use this, when adding new tests, to create the initial "golden" file.
+ * Also, on failures, actual output is written to
+ * /tmp/drill/store/base. You can use your IDE to compare
+ * the actual and golden files to understand changes. If the changes
+ * are expected, use that same IDE to copy changes from the actual
+ * to the golden file.
+ *
+ * The JSON properties of the serialized classes are all controlled
+ * to have a fixed order to ensure that files compare across test
+ * runs.
+ */
+public class TestFilterPushDown extends ClusterTest {
+
+ private static final String BASE_SQL = "SELECT a, b FROM dummy.myTable";
+ private static final String BASE_WHERE = BASE_SQL + " WHERE ";
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config =
+new DummyStoragePluginConfig(true, true, false);
+pluginRegistry.createOrUpdate("dummy", config, true);
+ }
+
+ //-
+ // Unsupported filter push-down cases
+
+ // No predicates
+
+ @Test
+ public void testNoPushDown() throws Exception
+ {
+verifyPlan(BASE_SQL, "noPushDown.json");
+ }
+
+ // Predicate mismatch on type (id should be INT, dummy does
+ // not try to convert)
+
+ @Test
+ public void testTypeMismatch() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id = 'foo'", "typeMismatch.json");
+ }
+
+ // Unsupported relop type (dummy supports limited set)
+
+ @Test
+ public void testUnsupportedOp() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a <> 'foo'", "unsupportedOp.json");
+ }
+
+ // Column reference rather than constant
+
+ @Test
+ public void testNonConst() throws Exception
+ {
+ verifyPlan(BASE_WHERE + "a = b", "nonConstPred.json");
+ }
+
+ // Unknown column (dummy only knows columns a and b)
+
+ @Test
+ public void testUnsupportedCol() throws Exception
+ {
+verifyPlan(BASE_WHERE + "c = 'foo'", "unsupportedColPred.json");
+ }
+
+ // Not simple col = const
+
+ @Test
+ public void testComplexPred() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id + 10 = 20", "complexPred.json");
+ }
+
+ // Complex schema paths
+
+ @Test
+ public void testComplexCols() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a[10] = 'foo' AND
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020777#comment-17020777
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369224848
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020776#comment-17020776
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369358074
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestFilterPushDown.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * Tests for the Filter push-down helper classes as part of the
+ * "Base" storage plugin to be used for add-on plugins. Uses a
+ * "test mule" ("Dummy") plug-in which goes through the paces of
+ * planning push down, but then glosses over the details at run time.
+ * The focus here are plans: the tests plan a query then compare the
+ * actual plan against and expected ("golden") plan.
+ *
+ * If comparison fails, the tests print the actual plan to the console.
+ * Use this, when adding new tests, to create the initial "golden" file.
+ * Also, on failures, actual output is written to
+ * /tmp/drill/store/base. You can use your IDE to compare
+ * the actual and golden files to understand changes. If the changes
+ * are expected, use that same IDE to copy changes from the actual
+ * to the golden file.
+ *
+ * The JSON properties of the serialized classes are all controlled
+ * to have a fixed order to ensure that files compare across test
+ * runs.
+ */
+public class TestFilterPushDown extends ClusterTest {
+
+ private static final String BASE_SQL = "SELECT a, b FROM dummy.myTable";
+ private static final String BASE_WHERE = BASE_SQL + " WHERE ";
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config =
+new DummyStoragePluginConfig(true, true, false);
+pluginRegistry.createOrUpdate("dummy", config, true);
Review comment:
Thanks. That method needs a bit of work. Not currently called from anywhere.
It wants a storage plugin object. To create one, I need a bunch of extra cruft
that is provided by `StoragePluginRegistry.create()` which is private.
What we want is a function that will call `createOrUpdate` as is done here,
with a name and a config. So, added one.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter"
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020778#comment-17020778
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369346923
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/RelOp.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Semanticized form of a Calcite relational operator. Abstracts
+ * out the Drill implementation details to capture just the
+ * column name, operator and value. Supports only expressions
+ * of the form:
+ * column> relop> const>
+ * Where the column is a simple name (not an array or map reference),
+ * the relop is one of a defined set, and the constant is one
+ * of the defined Drill types.
+ *
+ * (The driver will convert expressions of the form:
+ * const> relop> column>
+ * into the normalized form represented here.
+ */
+
+@JsonInclude(Include.NON_NULL)
+@JsonPropertyOrder({"op", "colName", "value"})
+public class RelOp {
+
+ public enum Op {
+EQ, NE, LT, LE, GT, GE, IS_NULL, IS_NOT_NULL;
Review comment:
Glad you saw that. Originally, we use constant strings everywhere. And,
since people are not sure the correct strings, I see copy/pasted blocks of name
variations.
The idea here is to say, 1) there is a fixed set of RelOps, and 2) there is
a fixed set of names for those ops.
This is part of the analysis mechanism: take the general-purpose function
call, expression and constant semantics of the query plan and distill it down
to a sharply-defined set of constant expressions for use in filter push-down.
Would actually be handy to define a metadata class for operations that
encapsulates this stuff. At present, that "metadata" is spread across dozens of
classes. Example: default selectivity is embedded somewhere in Calcite. Seems
Calcite has something, (`SqlBinaryOperator`), but it is a bit too abstract.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020780#comment-17020780
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369351240
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/RelOp.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Semanticized form of a Calcite relational operator. Abstracts
+ * out the Drill implementation details to capture just the
+ * column name, operator and value. Supports only expressions
+ * of the form:
+ * column> relop> const>
+ * Where the column is a simple name (not an array or map reference),
+ * the relop is one of a defined set, and the constant is one
+ * of the defined Drill types.
+ *
+ * (The driver will convert expressions of the form:
+ * const> relop> column>
+ * into the normalized form represented here.
+ */
+
+@JsonInclude(Include.NON_NULL)
+@JsonPropertyOrder({"op", "colName", "value"})
+public class RelOp {
+
+ public enum Op {
+EQ, NE, LT, LE, GT, GE, IS_NULL, IS_NOT_NULL;
+
+/**
+ * Return the result of flipping the sides of an
+ * expression:
+ * a op b b op.invert() a
+ */
+public RelOp.Op invert() {
+ switch(this) {
+ case LT:
+return GT;
+ case LE:
+return GE;
+ case GT:
+return LT;
+ case GE:
+return LE;
+ default:
+return this;
+ }
+}
+
+public int argCount() {
+ switch (this) {
+ case IS_NULL:
+ case IS_NOT_NULL:
+return 1;
+ default:
+return 2;
+ }
+}
+
+/**
+ * Poor-man's guess at selectivity of each operator.
+ * Should match Calcite's built-in defaults (which are
+ * hard to find.)
+ *
+ * TODO: Double check against Drill defaults.
Review comment:
Done. Do you know if Calcite nodes carry a computed selectivity? Cloning the
Calcite defaults is a kludge. And, if Drill uses metadata to estimate a better
selectivity, we should reuse that here. I looked at the `FunctionCall` node,
the `RexNode` but there is nothing obvious. Calcite must track this vitally
important number. Would be horrible if it is recomputed on every plan change.
Do you know where to find it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020773#comment-17020773
]
ASF GitHub Bot commented on DRILL-7458:
---
paul-rogers commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369228513
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
Review comment:
Bingo! You hit on a nasty issue. Yes, Drill adds on all kinds of helpful
ser/des. But, as it turns out, there is no good way to get that mapper into
this class. I hit this very issue when trying to change the mock plugin: it
can't deserialize things like `ColumnMetadata` which require help.
Most plugins do exactly what is done here because the scan spec tends to be
simple.
A larger question is why the heck we serialize the scan spec: it never
leaves the planner. There is probably some hacky reason it was done this way.
Do you know of a way to get Drill's mapper into this class without changing
the constructor?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
>
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020422#comment-17020422
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369122660
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
+ public static final String DEFAULT_SCHEMA_NAME = "default";
+
+ public static class StoragePluginOptions {
+
+/**
+ * Identifies if the plugin supports read.
+ * true by default.
+ */
+public boolean supportsRead = true;
+
+/**
+ * Identifies if the plugin supports writes (as in
+ * CREATE TABLE AS. false scanSpecType;
+
+/**
+ * The Jackson object mapper to use to deserialize the
+ * scan spec. The default is fine for most cases.
+ */
+public ObjectMapper objectMapper = DEFAULT_MAPPER;
+
+/**
+ * The scan factory used to create the group scan and
+ * batch readers. Due to circular dependencies, it may be
+ * more convenient to set this field after calling the
+ * super class constructor.
+ */
+public BaseScanFactory scanFactory;
+ }
+
+ protected final C config;
+ protected final StoragePluginOptions options;
+ protected SchemaFactory schemaFactory;
+
+ protected BaseStoragePlugin(DrillbitContext context, C config, String name,
StoragePluginOptions options) {
+super(context, name);
+this.config = config;
+this.options = options;
+Preconditions.checkNotNull(options.scanSpecType);
+Preconditions.checkNotNull(options.scanFactory);
+ }
+
+ @Override
+ public StoragePluginConfig getConfig() { return config; }
+
+ public C config() { return config; }
+
+ public
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020426#comment-17020426
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369127232
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseSubScan.java
##
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractSubScan;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+/**
+ * Specification for a specific execution-time scan. Holds the information
+ * passed form the Planner to the Drillbit so that the Drillbit can execute
+ * the scan. Holds the config of the storage plugin associated with the
+ * scan. The config allows creating a storage plugin instance, as needed,
+ * to implement the scan.
+ *
+ * @see {@link BaseGroupScan} for additional details of the scan
+ * life-cycle.
+ */
+
+@JsonTypeName("base-sub-scan")
+public class BaseSubScan extends AbstractSubScan {
+
+ protected final BaseStoragePlugin storagePlugin;
+ protected final List columns;
+
+ public BaseSubScan(BaseGroupScan groupScan) {
+super(groupScan.getUserName());
+storagePlugin = groupScan.storagePlugin();
Review comment:
```suggestion
this.storagePlugin = groupScan.storagePlugin();
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020427#comment-17020427
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369146546
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownStrategy.java
##
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.planner.common.DrillRelOptUtil;
+import org.apache.drill.exec.planner.logical.DrillFilterRel;
+import org.apache.drill.exec.planner.logical.DrillOptiq;
+import org.apache.drill.exec.planner.logical.DrillParseContext;
+import org.apache.drill.exec.planner.logical.DrillProjectRel;
+import org.apache.drill.exec.planner.logical.DrillScanRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+import org.apache.drill.exec.planner.physical.FilterPrel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+
+/**
+ * Generalized filter push-down strategy which performs all the tree-walking
+ * and tree restructuring work, allowing a "listener" to do the work needed
+ * for a particular scan.
+ *
+ * General usage in a storage plugin:
+ * public Set getPhysicalOptimizerRules(
+ *OptimizerRulesContext optimizerRulesContext) {
+ * return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+ * new MyPushDownListener(...));
+ * }
+ *
+ */
+
+public class FilterPushDownStrategy {
+
+ private static final Collection BANNED_OPERATORS =
+ Lists.newArrayList("flatten");
+
+ /**
+ * Base rule that passes target information to the push-down strategy
+ */
+
+ private static abstract class AbstractFilterPushDownRule extends
StoragePluginOptimizerRule {
+
+protected final FilterPushDownStrategy strategy;
+
+public AbstractFilterPushDownRule(RelOptRuleOperand operand, String
description,
+FilterPushDownStrategy strategy) {
+ super(operand, description);
+ this.strategy = strategy;
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> PROJECT --> SCAN
+ */
+
+ private static class ProjectAndFilterRule extends AbstractFilterPushDownRule
{
+
+private ProjectAndFilterRule(FilterPushDownStrategy strategy) {
+ super(RelOptHelper.some(FilterPrel.class,
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
+strategy.namePrefix() + "PushDownFilter:Filter_On_Project",
+strategy);
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+ if (!super.matches(call)) {
+return false;
+ }
+ DrillScanRel scan = call.rel(2);
+ return strategy.isTargetScan(scan);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ DrillFilterRel filterRel = call.rel(0);
+ DrillProjectRel projectRel = call.rel(1);
+ DrillScanRel scanRel = call.rel(2);
+ strategy.onMatch(call, filterRel, projectRel, scanRel);
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> SCAN
+ */
+
+ private static class FilterWithoutProjectRule extends
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020425#comment-17020425
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369116226
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020428#comment-17020428
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369129074
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020420#comment-17020420
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369107575
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020423#comment-17020423
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369116545
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020418#comment-17020418
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369112612
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020424#comment-17020424
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369126051
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseSubScan.java
##
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractSubScan;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+/**
+ * Specification for a specific execution-time scan. Holds the information
+ * passed form the Planner to the Drillbit so that the Drillbit can execute
+ * the scan. Holds the config of the storage plugin associated with the
+ * scan. The config allows creating a storage plugin instance, as needed,
+ * to implement the scan.
+ *
+ * @see {@link BaseGroupScan} for additional details of the scan
Review comment:
A class name may be used without @link tag in @see:
```suggestion
* @see BaseGroupScan for additional details of the scan
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020421#comment-17020421
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369120596
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
+ public static final String DEFAULT_SCHEMA_NAME = "default";
+
+ public static class StoragePluginOptions {
+
+/**
+ * Identifies if the plugin supports read.
+ * true by default.
+ */
+public boolean supportsRead = true;
+
+/**
+ * Identifies if the plugin supports writes (as in
+ * CREATE TABLE AS. false scanSpecType;
+
+/**
+ * The Jackson object mapper to use to deserialize the
+ * scan spec. The default is fine for most cases.
+ */
+public ObjectMapper objectMapper = DEFAULT_MAPPER;
+
+/**
+ * The scan factory used to create the group scan and
+ * batch readers. Due to circular dependencies, it may be
+ * more convenient to set this field after calling the
+ * super class constructor.
+ */
+public BaseScanFactory scanFactory;
Review comment:
```suggestion
public BaseScanFactory scanFactory;
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
>
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020419#comment-17020419
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369114727
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseGroupScan.java
##
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.BaseStoragePlugin.StoragePluginOptions;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Base group scan for storage plugins. A group scan is a "logical" scan: it is
+ * the representation of the scan used during the logical and physical planning
+ * phases. The group scan is converted to a "sub scan" (an executable scan
+ * specification) for inclusion in the physical plan sent to Drillbits for
+ * execution. The group scan represents the entire scan of a table or data
+ * source. The sub scan divides that scan into files, storage blocks or other
+ * forms of parallelism.
+ *
+ * The group scan participates in both logical and physical planning. As
+ * noted below, logical plan information is JSON serialized, but physical
+ * plan information is not. The transition from logical
+ * to physical planning is not clear. The first call to
+ * {@link #getMinParallelizationWidth()} or
+ * {@link #getMaxParallelizationWidth()} is a good signal. The group scan
+ * is copied multiple times (each with more information) during logical
+ * planning, but is not copied during physical planning. This means that
+ * physical planning state can be thought of as transient: it should not
+ * be serialized or copied.
+ *
+ * Because the group scan is part of the Calcite planning process, it is
+ * very helpful to understand the basics of query planning and how
+ * Calcite implements that process.
+ *
+ * Serialization
+ *
+ * Drill provides the ability to serialize the logical plan. This is most
+ * easily seen by issuing the EXPLAIN PLAN FOR command for a
+ * query. The Jackson-serialized representation of the group scan appears
+ * in the JSON partition of the EXPLAIN output, while the
+ * toString() output appears in the text output of that
+ * command.
+ *
+ * Care must be taken when serializing: include only the logical
+ * plan information, omit the physical plan information.
+ *
+ * Any fields that are part of the logical plan must be Jackson
+ * serializable. The following information should be serialized in the
+ * logical plan:
+ *
+ * Information from the scan spec (from the table lookup in the
+ * schema) if relevant to the plugin.
+ * The list of columns projected in the scan.
+ * Any filters pushed into the query (in whatever form makes sense
+ * for the plugin.
+ * Any other plugin-specific logical plan information.
+ *
+ * This base class (and its superclasses) serialize the plugin config,
+ * user name, column list and scan stats. The derived class should handle
+ * other fields.
+ *
+ * On the other hand, the kinds of information should not be
+ * serialized:
+ *
+ * The set of drillbits on which queries will run.
+ * The
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020343#comment-17020343
]
ASF GitHub Bot commented on DRILL-7458:
---
vvysotskyi commented on pull request #1914: DRILL-7458: Base framework for
storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r369095160
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java
##
@@ -238,22 +251,41 @@ public RuleSet getRules(OptimizerRulesContext context,
Collection
this.description = description;
}
+ /**
+ * Filter push-down is best done during logical planning so that the result
can
+ * influence parallelization in the physical phase. The specific phase
differs
+ * depending on which planning mode is enabled. This check hides those
details
+ * from storage plugins that simply want to know "should I add my filter
+ * push-down rules in the given phase?"
+ *
+ * @return true if filter push-down rules should be applied in this phase
+ */
+ public boolean isFilterPushDownPhase() {
Review comment:
I disagree with limiting the possibility of pushdown to several phases. In
some cases, there may be additional rel nodes between filter and scan, so
pushdown cannot happen, but in other planning stages, these rel nodes may be
switched with filter, or filter may be extracted from join condition, etc. so
pushdown may happen only at a specific phase, but not at the phase we want.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019616#comment-17019616
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368567474
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/RelOp.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Semanticized form of a Calcite relational operator. Abstracts
+ * out the Drill implementation details to capture just the
+ * column name, operator and value. Supports only expressions
+ * of the form:
+ * column> relop> const>
+ * Where the column is a simple name (not an array or map reference),
+ * the relop is one of a defined set, and the constant is one
+ * of the defined Drill types.
+ *
+ * (The driver will convert expressions of the form:
+ * const> relop> column>
+ * into the normalized form represented here.
+ */
+
+@JsonInclude(Include.NON_NULL)
+@JsonPropertyOrder({"op", "colName", "value"})
+public class RelOp {
+
+ public enum Op {
+EQ, NE, LT, LE, GT, GE, IS_NULL, IS_NOT_NULL;
+
+/**
+ * Return the result of flipping the sides of an
+ * expression:
+ * a op b b op.invert() a
+ */
+public RelOp.Op invert() {
Review comment:
Please add `@return`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019605#comment-17019605
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368564847
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyGroupScan.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.filter.FilterSpec;
+import org.apache.drill.exec.store.base.filter.RelOp;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+@JsonTypeName("dummy-scan")
+@JsonPropertyOrder({"userName", "scanSpec", "columns",
+"filters", "cost", "config"})
+@JsonInclude(value=Include.NON_EMPTY, content=Include.NON_NULL)
+public class DummyGroupScan extends BaseGroupScan {
+
+ private final DummyScanSpec scanSpec;
+ private final FilterSpec filters;
+
+ public DummyGroupScan(DummyStoragePlugin storagePlugin, String userName,
+ DummyScanSpec scanSpec) {
+super(storagePlugin, userName, null);
+this.scanSpec = scanSpec;
+filters = null;
+ }
+
+ public DummyGroupScan(DummyGroupScan from, List columns) {
+super(from.storagePlugin, from.getUserName(), columns);
+this.scanSpec = from.scanSpec;
+this.filters = from.filters;
+ }
+
+ @JsonCreator
+ public DummyGroupScan(
+ @JsonProperty("config") DummyStoragePluginConfig config,
+ @JsonProperty("userName") String userName,
+ @JsonProperty("scanSpec") DummyScanSpec scanSpec,
+ @JsonProperty("columns") List columns,
+ @JsonProperty("filters") FilterSpec filters,
+ @JacksonInject StoragePluginRegistry engineRegistry) {
+super(config, userName, columns, engineRegistry);
+this.scanSpec = scanSpec;
+this.filters = filters;
+ }
+
+ public DummyGroupScan(DummyGroupScan from, FilterSpec filters) {
+super(from);
+this.scanSpec = from.scanSpec;
+this.filters = filters;
+ }
+
+ @JsonProperty("scanSpec")
+ public DummyScanSpec scanSpec() { return scanSpec; }
+
+ @JsonProperty("filters")
+ public FilterSpec andFilters() { return filters; }
+
+ public boolean hasFilters() {
+return filters != null && ! filters.isEmpty();
+ }
+
+ private static final List FILTER_COLS = ImmutableList.of("a", "b",
"id");
Review comment:
Use Java built in methods instead of Guava
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019602#comment-17019602
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368554017
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
+ public static final String DEFAULT_SCHEMA_NAME = "default";
+
+ public static class StoragePluginOptions {
+
+/**
+ * Identifies if the plugin supports read.
+ * true by default.
+ */
+public boolean supportsRead = true;
+
+/**
+ * Identifies if the plugin supports writes (as in
+ * CREATE TABLE AS. false scanSpecType;
+
+/**
+ * The Jackson object mapper to use to deserialize the
+ * scan spec. The default is fine for most cases.
+ */
+public ObjectMapper objectMapper = DEFAULT_MAPPER;
+
+/**
+ * The scan factory used to create the group scan and
+ * batch readers. Due to circular dependencies, it may be
+ * more convenient to set this field after calling the
+ * super class constructor.
+ */
+public BaseScanFactory scanFactory;
+ }
+
+ protected final C config;
+ protected final StoragePluginOptions options;
+ protected SchemaFactory schemaFactory;
+
+ protected BaseStoragePlugin(DrillbitContext context, C config, String name,
StoragePluginOptions options) {
+super(context, name);
+this.config = config;
+this.options = options;
+Preconditions.checkNotNull(options.scanSpecType);
+Preconditions.checkNotNull(options.scanFactory);
+ }
+
+ @Override
+ public StoragePluginConfig getConfig() { return config; }
+
+ public C config() { return config; }
+
+ public
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019621#comment-17019621
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368564310
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyFilterPushDownListener.java
##
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.exec.store.base.filter.DisjunctionFilterSpec;
+import org.apache.drill.exec.store.base.filter.FilterPushDownListener;
+import org.apache.drill.exec.store.base.filter.FilterPushDownStrategy;
+import org.apache.drill.exec.store.base.filter.FilterSpec;
+import org.apache.drill.exec.store.base.filter.RelOp;
+
+public class DummyFilterPushDownListener implements FilterPushDownListener {
+
+ private final DummyStoragePluginConfig config;
+
+ public DummyFilterPushDownListener(DummyStoragePluginConfig config) {
+this.config = config;
+ }
+
+ public static Set rulesFor(
+ OptimizerRulesContext optimizerRulesContext, DummyStoragePluginConfig
config) {
+return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+new DummyFilterPushDownListener(config));
+ }
+
+ @Override
+ public String prefix() { return "Dummy"; }
+
+ @Override
+ public boolean isTargetScan(GroupScan groupScan) {
+return groupScan instanceof DummyGroupScan;
+ }
+
+ @Override
+ public boolean needsApplication(GroupScan groupScan) {
+DummyGroupScan dummyScan = (DummyGroupScan) groupScan;
+return !dummyScan.hasFilters();
+ }
+
+ @Override
+ public RelOp accept(GroupScan groupScan, RelOp relOp) {
+
+// Determine if filter applies to this scan
+
+DummyGroupScan dummyScan = (DummyGroupScan) groupScan;
+return dummyScan.acceptFilter(relOp);
+ }
+
+ @Override
+ public Pair> transform(GroupScan groupScan,
+ List> andTerms, Pair orTerm) {
+List andExprs;
+if (andTerms == null || andTerms.isEmpty()) {
+ andExprs = null;
+} else {
+ andExprs = andTerms.stream().map(t ->
t.right).collect(Collectors.toList());
+}
+DisjunctionFilterSpec orExprs;
+if (orTerm == null) {
+ orExprs = null;
+} else {
+ orExprs = orTerm.right;
+}
+FilterSpec filters = FilterSpec.build(andExprs, orExprs);
+DummyGroupScan dummyScan = (DummyGroupScan) groupScan;
+GroupScan newScan = new DummyGroupScan(dummyScan, filters);
+
+List exprs;
+if (config.keepFilters()) {
+ exprs = new ArrayList<>();
+ if (andTerms != null) {
+exprs.addAll(andTerms.stream().map(t ->
t.left).collect(Collectors.toList()));
Review comment:
```suggestion
exprs.addAll(andTerms.stream()
.map(t -> t.left)
.collect(Collectors.toList()));
```
Chaining is usually written from the new line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy"
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019599#comment-17019599
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368556478
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/ConstantHolder.java
##
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.joda.time.DateTimeZone;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Description of a constant argument of an expression.
+ */
+
+@JsonPropertyOrder({"type", "value"})
+public class ConstantHolder {
+ @JsonProperty("type")
+ public final MinorType type;
+ @JsonProperty("value")
+ public final Object value;
+
+ @JsonCreator
+ public ConstantHolder(
+ @JsonProperty("type") MinorType type,
+ @JsonProperty("value") Object value) {
+this.type = type;
+this.value = value;
+ }
+
+ public ConstantHolder convertTo(MinorType toType) {
+if (type == toType) {
+ return this;
+}
+switch (toType) {
+case INT:
+ return toInt();
+case BIGINT:
+ return toBigInt();
+case TIMESTAMP:
+ return toTimestamp(null);
+case VARCHAR:
+ return toVarChar();
+default:
+ throw conversionError(toType);
+}
+ }
+
+ public ConstantHolder normalize(MinorType toType) {
Review comment:
Please explain this logic.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019623#comment-17019623
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368566269
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyStoragePluginConfig.java
##
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.shaded.guava.com.google.common.base.Objects;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+/**
+ * Options for the "test mule" storage plugin. The options
+ * control how the plugin behaves for testing. A real plugin
+ * would simply implement project push down, and, if filter push-down
+ * is needed, would implement one of the strategies described
+ * here.
+ */
+
+@JsonTypeName(DummyStoragePluginConfig.NAME)
+@JsonPropertyOrder({"enableProjectPushDown", "enableFilterPushDown",
+ "keepFilters", "enable"})
+public class DummyStoragePluginConfig extends StoragePluginConfig {
+
+ public static final String NAME = "dummy";
+
+ /**
+ * Whether to enable or disable project push down.
+ */
+ private final boolean enableProjectPushDown;
+
+ /**
+ * Whether to enable or disable filter push down.
+ */
+ private final boolean enableFilterPushDown;
+
+ /**
+ * When doing filter push-down, whether to keep the filters in
+ * the plan, or remove them (because they are done (simulated)
+ * in the reader.
+ */
+
+ private final boolean keepFilters;
+
+ public DummyStoragePluginConfig(
+ @JsonProperty("enableProjectPushDown") boolean enableProjectPushDown,
+ @JsonProperty("enableFilterPushDown") boolean enableFilterPushDown,
+ @JsonProperty("keepFilters") boolean keepFilters) {
+this.enableProjectPushDown = enableProjectPushDown;
+this.enableFilterPushDown = enableFilterPushDown;
+this.keepFilters = keepFilters;
+setEnabled(true);
+ }
+
+ @JsonProperty("enableProjectPushDown")
+ public boolean enableProjectPushDown() { return enableProjectPushDown; }
+
+ @JsonProperty("enableFilterPushDown")
+ public boolean enableFilterPushDown() { return enableFilterPushDown; }
+
+ @JsonProperty("keepFilters")
+ public boolean keepFilters() { return keepFilters; }
+
+ @Override
+ public boolean equals(Object o) {
+if (o == this) {
+ return true;
+}
+if (o == null || !(o instanceof DummyStoragePluginConfig)) {
+ return false;
+}
+DummyStoragePluginConfig other = (DummyStoragePluginConfig) o;
+return enableProjectPushDown == other.enableProjectPushDown &&
+ enableFilterPushDown == other.enableFilterPushDown &&
+ keepFilters == other.keepFilters;
+ }
+
+ @Override
+ public int hashCode() {
Review comment:
Though IDE generates otherwise, it's more common to have hashCode before
equals.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019600#comment-17019600
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368543408
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestFilterPushDown.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * Tests for the Filter push-down helper classes as part of the
+ * "Base" storage plugin to be used for add-on plugins. Uses a
+ * "test mule" ("Dummy") plug-in which goes through the paces of
+ * planning push down, but then glosses over the details at run time.
+ * The focus here are plans: the tests plan a query then compare the
+ * actual plan against and expected ("golden") plan.
+ *
+ * If comparison fails, the tests print the actual plan to the console.
+ * Use this, when adding new tests, to create the initial "golden" file.
+ * Also, on failures, actual output is written to
+ * /tmp/drill/store/base. You can use your IDE to compare
+ * the actual and golden files to understand changes. If the changes
+ * are expected, use that same IDE to copy changes from the actual
+ * to the golden file.
+ *
+ * The JSON properties of the serialized classes are all controlled
+ * to have a fixed order to ensure that files compare across test
+ * runs.
+ */
+public class TestFilterPushDown extends ClusterTest {
+
+ private static final String BASE_SQL = "SELECT a, b FROM dummy.myTable";
+ private static final String BASE_WHERE = BASE_SQL + " WHERE ";
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config =
+new DummyStoragePluginConfig(true, true, false);
+pluginRegistry.createOrUpdate("dummy", config, true);
+ }
+
+ //-
+ // Unsupported filter push-down cases
+
+ // No predicates
+
+ @Test
+ public void testNoPushDown() throws Exception
+ {
+verifyPlan(BASE_SQL, "noPushDown.json");
+ }
+
+ // Predicate mismatch on type (id should be INT, dummy does
+ // not try to convert)
+
+ @Test
+ public void testTypeMismatch() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id = 'foo'", "typeMismatch.json");
+ }
+
+ // Unsupported relop type (dummy supports limited set)
+
+ @Test
+ public void testUnsupportedOp() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a <> 'foo'", "unsupportedOp.json");
+ }
+
+ // Column reference rather than constant
+
+ @Test
+ public void testNonConst() throws Exception
+ {
+ verifyPlan(BASE_WHERE + "a = b", "nonConstPred.json");
+ }
+
+ // Unknown column (dummy only knows columns a and b)
+
+ @Test
+ public void testUnsupportedCol() throws Exception
+ {
+verifyPlan(BASE_WHERE + "c = 'foo'", "unsupportedColPred.json");
+ }
+
+ // Not simple col = const
+
+ @Test
+ public void testComplexPred() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id + 10 = 20", "complexPred.json");
+ }
+
+ // Complex schema paths
+
+ @Test
+ public void testComplexCols() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a[10] = 'foo' AND
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019615#comment-17019615
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368561344
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterSpec.java
##
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+
+@JsonInclude(value=Include.NON_EMPTY, content=Include.NON_NULL)
+@JsonPropertyOrder({"andTerms", "orTerms"})
+public class FilterSpec {
+
+ private final List andTerms;
+ private final DisjunctionFilterSpec orTerms;
+
+ @JsonCreator
+ public FilterSpec(
+ @JsonProperty("andTerms") List andFilters,
+ @JsonProperty("orTerms") DisjunctionFilterSpec orFilters) {
+this.andTerms = andFilters == null || andFilters.isEmpty() ? null :
andFilters;
+this.orTerms = orFilters == null || orFilters.values.length == 0 ? null :
orFilters;
+ }
+
+ public static FilterSpec build(List andFilters, DisjunctionFilterSpec
orFilters) {
+FilterSpec spec = new FilterSpec(andFilters, orFilters);
+return spec.isEmpty() ? null : spec;
+ }
+
+ @JsonProperty("andTerms")
+ public List andTerms() { return andTerms; }
+
+ @JsonProperty("orTerms")
+ public DisjunctionFilterSpec orTerms() { return orTerms; }
+
+ /**
+ * The number of partitions in the sense of scan "segments" given
+ * by an OR clause. (May not correspond to HDFS file partitions.)
+ *
+ * @return the number of logical partitions, which is the number
+ * of OR clause terms (or 1 if no OR terms exist)
+ */
+
+ public int partitionCount() {
+return orTerms == null ? 1 : orTerms.values.length;
+ }
+
+ public static int parititonCount(FilterSpec filters) {
+return filters == null ? 1 : filters.partitionCount();
+ }
+
+ public List distribute(int orTerm) {
+return DisjunctionFilterSpec.distribute(andTerms, orTerms, orTerm);
+ }
+
+ /**
+ * Compute selectivity of a CNF form of equality conditions. Without stats,
+ * Drill assumes a selectivity of 0.15 for each equality, then multiplies
+ * the selectivity of multiple columns. Place a lower limit of 0.001 on
+ * the result, assuming the user wants to return some rows.
+ *
+ * @param andTerms list of AND filters (in CNF form), or null if no
+ * filters apply to a scan
+ * @return selectivity of the filters
+ */
+
+ public double cnfSelectivity() {
+if (andTerms == null || andTerms.isEmpty()) {
+ return 1.0;
+}
+double selectivity = 1.0;
+for (RelOp relOp : andTerms) {
+ selectivity *= relOp.op.selectivity();
+}
+return Math.max(0.001, selectivity);
+ }
+
+ /**
+ * Compute the combined selectivity of a set of AND (CNF) and OR
+ * (DNF) filters. We assume the DNF returns a subset of rows, which
+ * the CNF terms further reduce. Selectivity will be at least
+ * 0.001, which assumes the user wants to return some rows.
+ *
+ * @param andTerms and filters for the scan, or null if none
+ * @param orTerms or filters for the scan, or null if none
+ * @return combined selectivity
+ */
+
+ public double selectivity() {
+double selectivity = cnfSelectivity();
+if (orTerms != null) {
+ selectivity = Math.max(0.001, selectivity * orTerms.selectivity());
+}
+return selectivity;
+ }
+
+ /**
+ * Drill scan stats want a row count, the selectivity is a means to
+ * that end. Apply the selectivity of a set of filters to the given
+ * row count to produce a reduced
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019589#comment-17019589
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368552400
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
+ public static final String DEFAULT_SCHEMA_NAME = "default";
+
+ public static class StoragePluginOptions {
+
+/**
+ * Identifies if the plugin supports read.
+ * true by default.
+ */
+public boolean supportsRead = true;
+
+/**
+ * Identifies if the plugin supports writes (as in
+ * CREATE TABLE AS. false Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality,
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019598#comment-17019598
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368556896
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/ConstantHolder.java
##
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.joda.time.DateTimeZone;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Description of a constant argument of an expression.
+ */
+
+@JsonPropertyOrder({"type", "value"})
+public class ConstantHolder {
+ @JsonProperty("type")
+ public final MinorType type;
+ @JsonProperty("value")
+ public final Object value;
+
+ @JsonCreator
+ public ConstantHolder(
+ @JsonProperty("type") MinorType type,
+ @JsonProperty("value") Object value) {
+this.type = type;
+this.value = value;
+ }
+
+ public ConstantHolder convertTo(MinorType toType) {
+if (type == toType) {
+ return this;
+}
+switch (toType) {
+case INT:
Review comment:
Why only these types?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019619#comment-17019619
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368557624
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL IN clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+
+public interface FilterPushDownListener {
+
+ /**
+ * Prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ */
+ boolean isTargetScan(GroupScan groupScan);
Review comment:
Add `@return` and `@param` to java doc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019588#comment-17019588
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368550936
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseScanFactory.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+
+/**
+ * Creates the multiple classes needed to plan and execute a scan. Centralizes
+ * most implementation-specific object creation in one location to avoid
needing
+ * to implement many methods just for type-specific creation.
+ *
+ * @param the storage plugin class
+ * @param the scan specification class
+ * @param the group scan class
+ * @param the sub scan class
+ */
+
+public abstract class BaseScanFactory<
+ PLUGIN extends BaseStoragePlugin,
+ SPEC,
+ GROUP extends BaseGroupScan,
+ SUB extends BaseSubScan> {
+
+ @SuppressWarnings("unchecked")
Review comment:
I see many unchecked suppressions in newly added code? Is there a way we can
avoid them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019620#comment-17019620
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368562233
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/RelOp.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+
+/**
+ * Semanticized form of a Calcite relational operator. Abstracts
+ * out the Drill implementation details to capture just the
+ * column name, operator and value. Supports only expressions
+ * of the form:
+ * column> relop> const>
+ * Where the column is a simple name (not an array or map reference),
+ * the relop is one of a defined set, and the constant is one
+ * of the defined Drill types.
+ *
+ * (The driver will convert expressions of the form:
+ * const> relop> column>
+ * into the normalized form represented here.
+ */
+
+@JsonInclude(Include.NON_NULL)
+@JsonPropertyOrder({"op", "colName", "value"})
+public class RelOp {
+
+ public enum Op {
+EQ, NE, LT, LE, GT, GE, IS_NULL, IS_NOT_NULL;
+
+/**
+ * Return the result of flipping the sides of an
+ * expression:
+ * a op b b op.invert() a
+ */
+public RelOp.Op invert() {
+ switch(this) {
Review comment:
I believe `case`-s after switch should go with indention:
```
switch(this) {
case LT:
return GT;
case LE:
return GE;
case GT:
return LT;
case GE:
return LE;
default:
return this;
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019586#comment-17019586
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368538149
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestProjectPushDown.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestProjectPushDown extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config1 =
+new DummyStoragePluginConfig(true, false, true);
+pluginRegistry.createOrUpdate("dummy", config1, true);
+ }
+
+ @Test
+ public void testPushDownEnabled() throws Exception {
+String plan = client.queryBuilder().sql("SELECT a, b, c from
dummy.myTable").explainJson();
+// DRILL-7451: should be 0
+assertEquals(1, StringUtils.countMatches(plan, "\"pop\" : \"project\""));
Review comment:
There is plan matcher in query builder which matches text plan. We can add
similar method to match son plan to avoid using custom methods. Example of plan
matcher:
```
queryBuilder()
.sql(query)
.planMatcher()
.exclude("Filter")
.match();
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019611#comment-17019611
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368558626
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL IN clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+
+public interface FilterPushDownListener {
+
+ /**
+ * Prefix to display in filter rules
+ */
+ String prefix();
+
+ /**
+ * Broad check to see if the scan is of the correct type for this
+ * listener. Generally implemented as:
+ * public boolean isTargetScan(ScanPrel scan) {
+ * return scan.getGroupScan() instanceof MyGroupScan;
+ * }
+ *
+ */
+ boolean isTargetScan(GroupScan groupScan);
+
+ /**
+ * Check if the filter rule should be applied to the target group scan.
+ * Calcite will run this rule multiple times during planning, but the
+ * transform only needs to occur once.
+ * Allows the group scan to mark in its own way whether the rule has
+ * been applied.
+ *
+ * @param scan the scan node
+ * @return true if filter push-down should be applied
+ */
+
+ boolean needsApplication(GroupScan groupScan);
+
+ /**
+ * Determine if the given relational operator (which is already in the form
+ * col name> relop> const>, qualifies for push down for
+ * this scan.
+ *
+ * If so, return an equivalent RelOp with the value normalized to what
+ * the plugin needs. The returned value may be the same as the original
+ * one if the value is already normalized.
+ *
+ * @param scan the scan element. Use scan.getGroupScan() to get the
+ * group scan
+ * @param relOp the description of the relational operator expression
+ * @return a normalized RelOp if this relop can be transformed into a filter
+ * push-down, null if not and thus the relop should remain in
+ * the Drill plan
+ */
+
+ RelOp accept(GroupScan groupScan, RelOp relOp);
+
+ /**
+ * Transform a normalized DNF term into a new scan. Normalized form is:
+ *
+ * (a AND b AND (x OR y))
+ * In which each OR term represents a scan partition. It
+ * is up to the code here to determine if the scan partition can be handled,
+ * corresponds to a storage partition, or can be done as a separate
+ * scan (as for a JDBC or REST plugin, say.)
+ *
+ * Each term is accompanied by the Calcite expression from which it was
+ * derived. The caller is responsible for determining which expressions,
+ * if
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019622#comment-17019622
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368564680
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyGroupScan.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.filter.FilterSpec;
+import org.apache.drill.exec.store.base.filter.RelOp;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+@JsonTypeName("dummy-scan")
+@JsonPropertyOrder({"userName", "scanSpec", "columns",
Review comment:
Why order matters?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019587#comment-17019587
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368543045
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestFilterPushDown.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * Tests for the Filter push-down helper classes as part of the
+ * "Base" storage plugin to be used for add-on plugins. Uses a
+ * "test mule" ("Dummy") plug-in which goes through the paces of
+ * planning push down, but then glosses over the details at run time.
+ * The focus here are plans: the tests plan a query then compare the
+ * actual plan against and expected ("golden") plan.
+ *
+ * If comparison fails, the tests print the actual plan to the console.
+ * Use this, when adding new tests, to create the initial "golden" file.
+ * Also, on failures, actual output is written to
+ * /tmp/drill/store/base. You can use your IDE to compare
+ * the actual and golden files to understand changes. If the changes
+ * are expected, use that same IDE to copy changes from the actual
+ * to the golden file.
+ *
+ * The JSON properties of the serialized classes are all controlled
+ * to have a fixed order to ensure that files compare across test
+ * runs.
+ */
+public class TestFilterPushDown extends ClusterTest {
+
+ private static final String BASE_SQL = "SELECT a, b FROM dummy.myTable";
+ private static final String BASE_WHERE = BASE_SQL + " WHERE ";
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config =
+new DummyStoragePluginConfig(true, true, false);
+pluginRegistry.createOrUpdate("dummy", config, true);
+ }
+
+ //-
+ // Unsupported filter push-down cases
+
+ // No predicates
+
+ @Test
+ public void testNoPushDown() throws Exception
+ {
+verifyPlan(BASE_SQL, "noPushDown.json");
+ }
+
+ // Predicate mismatch on type (id should be INT, dummy does
+ // not try to convert)
+
+ @Test
+ public void testTypeMismatch() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id = 'foo'", "typeMismatch.json");
+ }
+
+ // Unsupported relop type (dummy supports limited set)
+
+ @Test
+ public void testUnsupportedOp() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a <> 'foo'", "unsupportedOp.json");
+ }
+
+ // Column reference rather than constant
+
+ @Test
+ public void testNonConst() throws Exception
+ {
+ verifyPlan(BASE_WHERE + "a = b", "nonConstPred.json");
+ }
+
+ // Unknown column (dummy only knows columns a and b)
+
+ @Test
+ public void testUnsupportedCol() throws Exception
+ {
+verifyPlan(BASE_WHERE + "c = 'foo'", "unsupportedColPred.json");
+ }
+
+ // Not simple col = const
+
+ @Test
+ public void testComplexPred() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id + 10 = 20", "complexPred.json");
+ }
+
+ // Complex schema paths
+
+ @Test
+ public void testComplexCols() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a[10] = 'foo' AND
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019582#comment-17019582
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368540780
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestFilterPushDown.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * Tests for the Filter push-down helper classes as part of the
+ * "Base" storage plugin to be used for add-on plugins. Uses a
+ * "test mule" ("Dummy") plug-in which goes through the paces of
+ * planning push down, but then glosses over the details at run time.
+ * The focus here are plans: the tests plan a query then compare the
+ * actual plan against and expected ("golden") plan.
+ *
+ * If comparison fails, the tests print the actual plan to the console.
+ * Use this, when adding new tests, to create the initial "golden" file.
+ * Also, on failures, actual output is written to
+ * /tmp/drill/store/base. You can use your IDE to compare
+ * the actual and golden files to understand changes. If the changes
+ * are expected, use that same IDE to copy changes from the actual
+ * to the golden file.
+ *
+ * The JSON properties of the serialized classes are all controlled
+ * to have a fixed order to ensure that files compare across test
+ * runs.
+ */
+public class TestFilterPushDown extends ClusterTest {
+
+ private static final String BASE_SQL = "SELECT a, b FROM dummy.myTable";
+ private static final String BASE_WHERE = BASE_SQL + " WHERE ";
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config =
+new DummyStoragePluginConfig(true, true, false);
+pluginRegistry.createOrUpdate("dummy", config, true);
+ }
+
+ //-
+ // Unsupported filter push-down cases
+
+ // No predicates
+
+ @Test
+ public void testNoPushDown() throws Exception
+ {
+verifyPlan(BASE_SQL, "noPushDown.json");
+ }
+
+ // Predicate mismatch on type (id should be INT, dummy does
+ // not try to convert)
+
+ @Test
+ public void testTypeMismatch() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id = 'foo'", "typeMismatch.json");
+ }
+
+ // Unsupported relop type (dummy supports limited set)
+
+ @Test
+ public void testUnsupportedOp() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a <> 'foo'", "unsupportedOp.json");
+ }
+
+ // Column reference rather than constant
+
+ @Test
+ public void testNonConst() throws Exception
+ {
+ verifyPlan(BASE_WHERE + "a = b", "nonConstPred.json");
+ }
+
+ // Unknown column (dummy only knows columns a and b)
+
+ @Test
+ public void testUnsupportedCol() throws Exception
+ {
+verifyPlan(BASE_WHERE + "c = 'foo'", "unsupportedColPred.json");
+ }
+
+ // Not simple col = const
+
+ @Test
+ public void testComplexPred() throws Exception
+ {
+verifyPlan(BASE_WHERE + "id + 10 = 20", "complexPred.json");
+ }
+
+ // Complex schema paths
+
+ @Test
+ public void testComplexCols() throws Exception
+ {
+verifyPlan(BASE_WHERE + "a[10] = 'foo' AND
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019597#comment-17019597
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368557664
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownListener.java
##
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.exec.physical.base.GroupScan;
+
+/**
+ * Call-back (listener) implementation for a push-down filter.
+ * Abstracts away the common work; plugins implement this class
+ * to do work specific to the plugin.
+ *
+ * Supports two kinds of filter push down:
+ *
+ * Conjunctive Normal Form (CNF)
+ * A series of expressions joined by an AND: the scan should
+ * produce only rows that satisfy all the conditions.
+ * Disjunctive Normal Form (DNF)
+ * A series of alternative values for a single column, essentially
+ * a set of expressions joined by OR. The scan spits into multiple
+ * scans, each scanning one of the partitions (or regions or
+ * queries) identified by the case. This is an implementation of the
+ * SQL IN clause.
+ *
+ *
+ * In both cases, the conditions are in the form of a
+ * {@link RelOp} in which one side refers to a column in the scan
+ * and the other is a constant expression. The "driver" will ensure
+ * the rel op is of the correct form; this class ensures that the
+ * column is valid for the scan and the type of the value matches the
+ * column type (or can be converted.)
+ *
+ * The DNF form further ensures that all rel ops refer to the same
+ * column, and that only the equality operator appears in the
+ * terms.
+ */
+
+public interface FilterPushDownListener {
+
+ /**
+ * Prefix to display in filter rules
+ */
+ String prefix();
Review comment:
Add `@return`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019617#comment-17019617
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368564963
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/DummyGroupScan.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.base.filter.FilterSpec;
+import org.apache.drill.exec.store.base.filter.RelOp;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+@JsonTypeName("dummy-scan")
+@JsonPropertyOrder({"userName", "scanSpec", "columns",
+"filters", "cost", "config"})
+@JsonInclude(value=Include.NON_EMPTY, content=Include.NON_NULL)
+public class DummyGroupScan extends BaseGroupScan {
+
+ private final DummyScanSpec scanSpec;
+ private final FilterSpec filters;
+
+ public DummyGroupScan(DummyStoragePlugin storagePlugin, String userName,
+ DummyScanSpec scanSpec) {
+super(storagePlugin, userName, null);
+this.scanSpec = scanSpec;
+filters = null;
+ }
+
+ public DummyGroupScan(DummyGroupScan from, List columns) {
+super(from.storagePlugin, from.getUserName(), columns);
+this.scanSpec = from.scanSpec;
+this.filters = from.filters;
+ }
+
+ @JsonCreator
+ public DummyGroupScan(
+ @JsonProperty("config") DummyStoragePluginConfig config,
+ @JsonProperty("userName") String userName,
+ @JsonProperty("scanSpec") DummyScanSpec scanSpec,
+ @JsonProperty("columns") List columns,
+ @JsonProperty("filters") FilterSpec filters,
+ @JacksonInject StoragePluginRegistry engineRegistry) {
+super(config, userName, columns, engineRegistry);
+this.scanSpec = scanSpec;
+this.filters = filters;
+ }
+
+ public DummyGroupScan(DummyGroupScan from, FilterSpec filters) {
+super(from);
+this.scanSpec = from.scanSpec;
+this.filters = filters;
+ }
+
+ @JsonProperty("scanSpec")
+ public DummyScanSpec scanSpec() { return scanSpec; }
+
+ @JsonProperty("filters")
+ public FilterSpec andFilters() { return filters; }
+
+ public boolean hasFilters() {
+return filters != null && ! filters.isEmpty();
+ }
+
+ private static final List FILTER_COLS = ImmutableList.of("a", "b",
"id");
+
+ public RelOp acceptFilter(RelOp relOp) {
+
+// Pretend that "id" is a special integer column. Can handle
+// equality only.
+
+if (relOp.colName.contentEquals("id")) {
+
+ // To allow easier testing, require exact type match: no
+ // attempt at type conversion here.
+
+ if (relOp.op != RelOp.Op.EQ || relOp.value.type != MinorType.INT) {
+return null;
+ }
+ return relOp;
+}
+
+// All other columns apply only if projected
+
+if (!FILTER_COLS.contains(relOp.colName)) {
+ return null;
+}
+
+// Only supports a few operators so we can verify that the
+// others are left in the WHERE clause.
+// Neither are really implemented.
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019601#comment-17019601
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368561238
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterSpec.java
##
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+
+@JsonInclude(value=Include.NON_EMPTY, content=Include.NON_NULL)
+@JsonPropertyOrder({"andTerms", "orTerms"})
+public class FilterSpec {
+
+ private final List andTerms;
+ private final DisjunctionFilterSpec orTerms;
+
+ @JsonCreator
+ public FilterSpec(
+ @JsonProperty("andTerms") List andFilters,
+ @JsonProperty("orTerms") DisjunctionFilterSpec orFilters) {
+this.andTerms = andFilters == null || andFilters.isEmpty() ? null :
andFilters;
+this.orTerms = orFilters == null || orFilters.values.length == 0 ? null :
orFilters;
+ }
+
+ public static FilterSpec build(List andFilters, DisjunctionFilterSpec
orFilters) {
+FilterSpec spec = new FilterSpec(andFilters, orFilters);
+return spec.isEmpty() ? null : spec;
+ }
+
+ @JsonProperty("andTerms")
+ public List andTerms() { return andTerms; }
+
+ @JsonProperty("orTerms")
+ public DisjunctionFilterSpec orTerms() { return orTerms; }
+
+ /**
+ * The number of partitions in the sense of scan "segments" given
+ * by an OR clause. (May not correspond to HDFS file partitions.)
+ *
+ * @return the number of logical partitions, which is the number
+ * of OR clause terms (or 1 if no OR terms exist)
+ */
+
+ public int partitionCount() {
+return orTerms == null ? 1 : orTerms.values.length;
+ }
+
+ public static int parititonCount(FilterSpec filters) {
+return filters == null ? 1 : filters.partitionCount();
+ }
+
+ public List distribute(int orTerm) {
+return DisjunctionFilterSpec.distribute(andTerms, orTerms, orTerm);
+ }
+
+ /**
+ * Compute selectivity of a CNF form of equality conditions. Without stats,
+ * Drill assumes a selectivity of 0.15 for each equality, then multiplies
+ * the selectivity of multiple columns. Place a lower limit of 0.001 on
+ * the result, assuming the user wants to return some rows.
+ *
+ * @param andTerms list of AND filters (in CNF form), or null if no
Review comment:
Please revisit java doc, it is not uo-to-date
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019613#comment-17019613
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368560147
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterPushDownStrategy.java
##
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.util.Pair;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.ops.OptimizerRulesContext;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.planner.common.DrillRelOptUtil;
+import org.apache.drill.exec.planner.logical.DrillFilterRel;
+import org.apache.drill.exec.planner.logical.DrillOptiq;
+import org.apache.drill.exec.planner.logical.DrillParseContext;
+import org.apache.drill.exec.planner.logical.DrillProjectRel;
+import org.apache.drill.exec.planner.logical.DrillScanRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+import org.apache.drill.exec.planner.physical.FilterPrel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+
+/**
+ * Generalized filter push-down strategy which performs all the tree-walking
+ * and tree restructuring work, allowing a "listener" to do the work needed
+ * for a particular scan.
+ *
+ * General usage in a storage plugin:
+ * public Set getPhysicalOptimizerRules(
+ *OptimizerRulesContext optimizerRulesContext) {
+ * return FilterPushDownStrategy.rulesFor(optimizerRulesContext,
+ * new MyPushDownListener(...));
+ * }
+ *
+ */
+
+public class FilterPushDownStrategy {
+
+ private static final Collection BANNED_OPERATORS =
+ Lists.newArrayList("flatten");
+
+ /**
+ * Base rule that passes target information to the push-down strategy
+ */
+
+ private static abstract class AbstractFilterPushDownRule extends
StoragePluginOptimizerRule {
+
+protected final FilterPushDownStrategy strategy;
+
+public AbstractFilterPushDownRule(RelOptRuleOperand operand, String
description,
+FilterPushDownStrategy strategy) {
+ super(operand, description);
+ this.strategy = strategy;
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> PROJECT --> SCAN
+ */
+
+ private static class ProjectAndFilterRule extends AbstractFilterPushDownRule
{
+
+private ProjectAndFilterRule(FilterPushDownStrategy strategy) {
+ super(RelOptHelper.some(FilterPrel.class,
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
+strategy.namePrefix() + "PushDownFilter:Filter_On_Project",
+strategy);
+}
+
+@Override
+public boolean matches(RelOptRuleCall call) {
+ if (!super.matches(call)) {
+return false;
+ }
+ DrillScanRel scan = call.rel(2);
+ return strategy.isTargetScan(scan);
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ DrillFilterRel filterRel = call.rel(0);
+ DrillProjectRel projectRel = call.rel(1);
+ DrillScanRel scanRel = call.rel(2);
+ strategy.onMatch(call, filterRel, projectRel, scanRel);
+}
+ }
+
+ /**
+ * Calcite rule for FILTER --> SCAN
+ */
+
+ private static class FilterWithoutProjectRule extends
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019604#comment-17019604
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368554595
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/BaseStoragePlugin.java
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import java.io.IOException;
+
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.ChildErrorContext;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import
org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.ScanFrameworkBuilder;
+import org.apache.drill.exec.proto.UserBitShared.CoreOperatorType;
+import org.apache.drill.exec.record.CloseableRecordBatch;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SessionOptionManager;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.StoragePlugin;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.shaded.guava.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Base class for non-DFS storage plugins. Provides a number of convinces to
+ * abstract away some of the complexity around such plugins. The
+ * {@link StoragePluginOptions} class captures the many options that would
+ * otherwise be specified by overriding methods. The
+ * {@link BaseScanFactory} class provides methods to create various
+ * objects during planning and execution.
+ *
+ * @param the storage plugin configuration class
+ */
+
+public abstract class BaseStoragePlugin
+extends AbstractStoragePlugin {
+
+ private static final Logger logger =
LoggerFactory.getLogger(BaseStoragePlugin.class);
+ protected static final ObjectMapper DEFAULT_MAPPER = new ObjectMapper();
+ public static final String DEFAULT_SCHEMA_NAME = "default";
+
+ public static class StoragePluginOptions {
+
+/**
+ * Identifies if the plugin supports read.
+ * true by default.
+ */
+public boolean supportsRead = true;
+
+/**
+ * Identifies if the plugin supports writes (as in
+ * CREATE TABLE AS. false scanSpecType;
+
+/**
+ * The Jackson object mapper to use to deserialize the
+ * scan spec. The default is fine for most cases.
+ */
+public ObjectMapper objectMapper = DEFAULT_MAPPER;
+
+/**
+ * The scan factory used to create the group scan and
+ * batch readers. Due to circular dependencies, it may be
+ * more convenient to set this field after calling the
+ * super class constructor.
+ */
+public BaseScanFactory scanFactory;
+ }
+
+ protected final C config;
+ protected final StoragePluginOptions options;
+ protected SchemaFactory schemaFactory;
+
+ protected BaseStoragePlugin(DrillbitContext context, C config, String name,
StoragePluginOptions options) {
+super(context, name);
+this.config = config;
+this.options = options;
+Preconditions.checkNotNull(options.scanSpecType);
+Preconditions.checkNotNull(options.scanFactory);
+ }
+
+ @Override
+ public StoragePluginConfig getConfig() { return config; }
+
+ public C config() { return config; }
+
+ public
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019585#comment-17019585
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368538674
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestFilterPushDown.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.Reader;
+import java.io.StringReader;
+import java.net.URL;
+
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * Tests for the Filter push-down helper classes as part of the
+ * "Base" storage plugin to be used for add-on plugins. Uses a
+ * "test mule" ("Dummy") plug-in which goes through the paces of
+ * planning push down, but then glosses over the details at run time.
+ * The focus here are plans: the tests plan a query then compare the
+ * actual plan against and expected ("golden") plan.
+ *
+ * If comparison fails, the tests print the actual plan to the console.
+ * Use this, when adding new tests, to create the initial "golden" file.
+ * Also, on failures, actual output is written to
+ * /tmp/drill/store/base. You can use your IDE to compare
+ * the actual and golden files to understand changes. If the changes
+ * are expected, use that same IDE to copy changes from the actual
+ * to the golden file.
+ *
+ * The JSON properties of the serialized classes are all controlled
+ * to have a fixed order to ensure that files compare across test
+ * runs.
+ */
+public class TestFilterPushDown extends ClusterTest {
+
+ private static final String BASE_SQL = "SELECT a, b FROM dummy.myTable";
+ private static final String BASE_WHERE = BASE_SQL + " WHERE ";
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config =
+new DummyStoragePluginConfig(true, true, false);
+pluginRegistry.createOrUpdate("dummy", config, true);
Review comment:
Use handy cluster#defineStoragePlugin method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019612#comment-17019612
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368561277
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/base/filter/FilterSpec.java
##
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base.filter;
+
+import java.util.List;
+
+import org.apache.drill.exec.store.base.PlanStringBuilder;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import com.fasterxml.jackson.annotation.JsonInclude.Include;
+
+@JsonInclude(value=Include.NON_EMPTY, content=Include.NON_NULL)
+@JsonPropertyOrder({"andTerms", "orTerms"})
+public class FilterSpec {
+
+ private final List andTerms;
+ private final DisjunctionFilterSpec orTerms;
+
+ @JsonCreator
+ public FilterSpec(
+ @JsonProperty("andTerms") List andFilters,
+ @JsonProperty("orTerms") DisjunctionFilterSpec orFilters) {
+this.andTerms = andFilters == null || andFilters.isEmpty() ? null :
andFilters;
+this.orTerms = orFilters == null || orFilters.values.length == 0 ? null :
orFilters;
+ }
+
+ public static FilterSpec build(List andFilters, DisjunctionFilterSpec
orFilters) {
+FilterSpec spec = new FilterSpec(andFilters, orFilters);
+return spec.isEmpty() ? null : spec;
+ }
+
+ @JsonProperty("andTerms")
+ public List andTerms() { return andTerms; }
+
+ @JsonProperty("orTerms")
+ public DisjunctionFilterSpec orTerms() { return orTerms; }
+
+ /**
+ * The number of partitions in the sense of scan "segments" given
+ * by an OR clause. (May not correspond to HDFS file partitions.)
+ *
+ * @return the number of logical partitions, which is the number
+ * of OR clause terms (or 1 if no OR terms exist)
+ */
+
+ public int partitionCount() {
+return orTerms == null ? 1 : orTerms.values.length;
+ }
+
+ public static int parititonCount(FilterSpec filters) {
+return filters == null ? 1 : filters.partitionCount();
+ }
+
+ public List distribute(int orTerm) {
+return DisjunctionFilterSpec.distribute(andTerms, orTerms, orTerm);
+ }
+
+ /**
+ * Compute selectivity of a CNF form of equality conditions. Without stats,
+ * Drill assumes a selectivity of 0.15 for each equality, then multiplies
+ * the selectivity of multiple columns. Place a lower limit of 0.001 on
+ * the result, assuming the user wants to return some rows.
+ *
+ * @param andTerms list of AND filters (in CNF form), or null if no
+ * filters apply to a scan
+ * @return selectivity of the filters
+ */
+
+ public double cnfSelectivity() {
+if (andTerms == null || andTerms.isEmpty()) {
+ return 1.0;
+}
+double selectivity = 1.0;
+for (RelOp relOp : andTerms) {
+ selectivity *= relOp.op.selectivity();
+}
+return Math.max(0.001, selectivity);
+ }
+
+ /**
+ * Compute the combined selectivity of a set of AND (CNF) and OR
+ * (DNF) filters. We assume the DNF returns a subset of rows, which
+ * the CNF terms further reduce. Selectivity will be at least
+ * 0.001, which assumes the user wants to return some rows.
+ *
+ * @param andTerms and filters for the scan, or null if none
Review comment:
Same here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL:
[jira] [Commented] (DRILL-7458) Base storage plugin framework
[
https://issues.apache.org/jira/browse/DRILL-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019584#comment-17019584
]
ASF GitHub Bot commented on DRILL-7458:
---
arina-ielchiieva commented on pull request #1914: DRILL-7458: Base framework
for storage plugins
URL: https://github.com/apache/drill/pull/1914#discussion_r368537140
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/base/TestProjectPushDown.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.base;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.exec.physical.rowSet.RowSet;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestProjectPushDown extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
+startCluster(builder);
+
+StoragePluginRegistry pluginRegistry =
cluster.drillbit().getContext().getStorage();
+DummyStoragePluginConfig config1 =
+new DummyStoragePluginConfig(true, false, true);
+pluginRegistry.createOrUpdate("dummy", config1, true);
+ }
+
+ @Test
+ public void testPushDownEnabled() throws Exception {
+String plan = client.queryBuilder().sql("SELECT a, b, c from
dummy.myTable").explainJson();
+// DRILL-7451: should be 0
+assertEquals(1, StringUtils.countMatches(plan, "\"pop\" : \"project\""));
+ }
+
+ @Test
+ public void testDummyReader() throws Exception {
+RowSet results = client.queryBuilder().sql("SELECT a, b, c from
dummy.myTable").rowSet();
+assertEquals(3, results.rowCount());
Review comment:
If assert fails results will never be cleared?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Base storage plugin framework
> -
>
> Key: DRILL-7458
> URL: https://issues.apache.org/jira/browse/DRILL-7458
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.18.0
>
>
> The "Easy" framework allows third-parties to add format plugins to Drill with
> moderate effort. (The process could be easier, but "Easy" makes it as simple
> as possible given the current structure.)
> At present, no such "starter" framework exists for storage plugins. Further,
> multiple storage plugins have implemented filter push down, seemingly by
> copying large blocks of code.
> This ticket offers a "base" framework for storage plugins and for filter
> push-downs. The framework builds on the EVF, allowing plugins to also support
> project push down.
> The framework has a "test mule" storage plugin to verify functionality, and
> was used as the basis of an REST-like plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
