This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cb2323a6e1 Publish built docs triggered by
6ed2e84da8154931c393a9885b80c4f8a9d47ebb
cb2323a6e1 is described below
commit cb2323a6e11a651d25e83f31c2cd614054c754b7
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Sep 29 14:07:20 2023 +0000
Publish built docs triggered by 6ed2e84da8154931c393a9885b80c4f8a9d47ebb
---
.../custom-table-providers.md.txt | 16 +++++++++
library-user-guide/custom-table-providers.html | 39 ++++++++++++++++++++++
searchindex.js | 2 +-
3 files changed, 56 insertions(+), 1 deletion(-)
diff --git a/_sources/library-user-guide/custom-table-providers.md.txt
b/_sources/library-user-guide/custom-table-providers.md.txt
index 30721d6a5b..2de4b1ba05 100644
--- a/_sources/library-user-guide/custom-table-providers.md.txt
+++ b/_sources/library-user-guide/custom-table-providers.md.txt
@@ -121,6 +121,22 @@ impl TableProvider for CustomDataSource {
With this, and the implementation of the omitted methods, we can now use the
`CustomDataSource` as a `TableProvider` in DataFusion.
+##### Additional `TableProvider` Methods
+
+`scan` has no default implementation, so it needed to be written. There are
other methods on the `TableProvider` that have default implementations, but can
be overridden if needed to provide additional functionality.
+
+###### `supports_filters_pushdown`
+
+The `supports_filters_pushdown` method can be overridden to indicate which
filter expressions support being pushed down to the data source and within that
the specificity of the pushdown.
+
+This returns a `Vec` of `TableProviderFilterPushDown` enums where each enum
represents a filter that can be pushed down. The `TableProviderFilterPushDown`
enum has three variants:
+
+- `TableProviderFilterPushDown::Unsupported` - the filter cannot be pushed down
+- `TableProviderFilterPushDown::Exact` - the filter can be pushed down and the
data source can guarantee that the filter will be applied completely to all
rows. This is the highest performance option.
+- `TableProviderFilterPushDown::Inexact` - the filter can be pushed down, but
the data source cannot guarantee that the filter will be applied to all rows.
DataFusion will apply `Inexact` filters again after the scan to ensure
correctness.
+
+For filters that can be pushed down, they'll be passed to the `scan` method as
the `filters` parameter and they can be made use of there.
+
## Using the Custom Table Provider
In order to use the custom table provider, we need to register it with
DataFusion. This is done by creating a `TableProvider` and registering it with
the `ExecutionContext`.
diff --git a/library-user-guide/custom-table-providers.html
b/library-user-guide/custom-table-providers.html
index 98eaa85d91..d944b6a7f1 100644
--- a/library-user-guide/custom-table-providers.html
+++ b/library-user-guide/custom-table-providers.html
@@ -365,6 +365,30 @@
<a class="reference internal nav-link" href="#scan-revisited">
Scan Revisited
</a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h5 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#additional-tableprovider-methods">
+ Additional
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ TableProvider
+ </span>
+ </code>
+ Methods
+ </a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h6 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#supports-filters-pushdown">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ supports_filters_pushdown
+ </span>
+ </code>
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
</li>
</ul>
</li>
@@ -523,6 +547,21 @@
</pre></div>
</div>
<p>With this, and the implementation of the omitted methods, we can now use
the <code class="docutils literal notranslate"><span
class="pre">CustomDataSource</span></code> as a <code class="docutils literal
notranslate"><span class="pre">TableProvider</span></code> in DataFusion.</p>
+<section id="additional-tableprovider-methods">
+<h5>Additional <code class="docutils literal notranslate"><span
class="pre">TableProvider</span></code> Methods<a class="headerlink"
href="#additional-tableprovider-methods" title="Link to this heading">¶</a></h5>
+<p><code class="docutils literal notranslate"><span
class="pre">scan</span></code> has no default implementation, so it needed to
be written. There are other methods on the <code class="docutils literal
notranslate"><span class="pre">TableProvider</span></code> that have default
implementations, but can be overridden if needed to provide additional
functionality.</p>
+<section id="supports-filters-pushdown">
+<h6><code class="docutils literal notranslate"><span
class="pre">supports_filters_pushdown</span></code><a class="headerlink"
href="#supports-filters-pushdown" title="Link to this heading">¶</a></h6>
+<p>The <code class="docutils literal notranslate"><span
class="pre">supports_filters_pushdown</span></code> method can be overridden to
indicate which filter expressions support being pushed down to the data source
and within that the specificity of the pushdown.</p>
+<p>This returns a <code class="docutils literal notranslate"><span
class="pre">Vec</span></code> of <code class="docutils literal
notranslate"><span class="pre">TableProviderFilterPushDown</span></code> enums
where each enum represents a filter that can be pushed down. The <code
class="docutils literal notranslate"><span
class="pre">TableProviderFilterPushDown</span></code> enum has three
variants:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span
class="pre">TableProviderFilterPushDown::Unsupported</span></code> - the filter
cannot be pushed down</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">TableProviderFilterPushDown::Exact</span></code> - the filter can
be pushed down and the data source can guarantee that the filter will be
applied completely to all rows. This is the highest performance option.</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">TableProviderFilterPushDown::Inexact</span></code> - the filter can
be pushed down, but the data source cannot guarantee that the filter will be
applied to all rows. DataFusion will apply <code class="docutils literal
notranslate"><span class="pre">Inexact</span></code> filters again after the
scan to ensure correctness.</p></li>
+</ul>
+<p>For filters that can be pushed down, they’ll be passed to the <code
class="docutils literal notranslate"><span class="pre">scan</span></code>
method as the <code class="docutils literal notranslate"><span
class="pre">filters</span></code> parameter and they can be made use of
there.</p>
+</section>
+</section>
</section>
</section>
</section>
diff --git a/searchindex.js b/searchindex.js
index 07df441b04..16edd8a80f 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"docnames": ["contributor-guide/architecture",
"contributor-guide/communication", "contributor-guide/index",
"contributor-guide/quarterly_roadmap", "contributor-guide/roadmap",
"contributor-guide/specification/index",
"contributor-guide/specification/invariants",
"contributor-guide/specification/output-field-name-semantic", "index",
"library-user-guide/adding-udfs", "library-user-guide/building-logical-plans",
"library-user-guide/catalogs", "library-user-guide/custom-tab [...]
\ No newline at end of file
+Search.setIndex({"docnames": ["contributor-guide/architecture",
"contributor-guide/communication", "contributor-guide/index",
"contributor-guide/quarterly_roadmap", "contributor-guide/roadmap",
"contributor-guide/specification/index",
"contributor-guide/specification/invariants",
"contributor-guide/specification/output-field-name-semantic", "index",
"library-user-guide/adding-udfs", "library-user-guide/building-logical-plans",
"library-user-guide/catalogs", "library-user-guide/custom-tab [...]
\ No newline at end of file
