This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1a29cc0 Commit build products
1a29cc0 is described below
commit 1a29cc0a3fea1fd0b7fc2df6cc881d3e201b94f7
Author: Build Pelican (action) <[email protected]>
AuthorDate: Mon Jan 12 21:28:04 2026 +0000
Commit build products
---
output/2026/01/12/extending-sql/index.html | 416 ++++++++++++++++++++++++++
output/author/geoffrey-claude-datadog.html | 62 ++++
output/category/blog.html | 30 ++
output/feed.xml | 22 +-
output/feeds/all-en.atom.xml | 286 +++++++++++++++++-
output/feeds/blog.atom.xml | 286 +++++++++++++++++-
output/feeds/geoffrey-claude-datadog.atom.xml | 286 ++++++++++++++++++
output/feeds/geoffrey-claude-datadog.rss.xml | 22 ++
output/images/extending-sql/architecture.svg | 151 ++++++++++
output/index.html | 39 +++
10 files changed, 1597 insertions(+), 3 deletions(-)
diff --git a/output/2026/01/12/extending-sql/index.html
b/output/2026/01/12/extending-sql/index.html
new file mode 100644
index 0000000..01007ff
--- /dev/null
+++ b/output/2026/01/12/extending-sql/index.html
@@ -0,0 +1,416 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <title>Extending SQL in DataFusion: from ->> to TABLESAMPLE - Apache
DataFusion Blog</title>
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script> </head>
+ <body class="d-flex flex-column h-100">
+ <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth
navbar example">
+ <div class="container-fluid">
+ <a class="navbar-brand" href="/blog"><img
src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache
DataFusion Blog</a>
+ <button class="navbar-toggler" type="button" data-bs-toggle="collapse"
data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false"
aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <div class="collapse navbar-collapse" id="navbarADP">
+ <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/about.html">About</a>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/blog/feed.xml">RSS</a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</nav>
+<!-- article contents -->
+<div id="contents">
+ <div class="bg-white p-4 p-md-5 rounded">
+ <div class="row justify-content-center">
+ <div class="col-12 col-md-8 main-content">
+ <h1>
+ Extending SQL in DataFusion: from ->> to TABLESAMPLE
+ </h1>
+ <p>Posted on: Mon 12 January 2026 by Geoffrey Claude (Datadog)</p>
+
+ <aside class="toc-container d-md-none mb-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#parse-plan-execute">Parse → Plan → Execute</a></li>
+<li><a href="#1-extending-parsing-wrapping-dfparser-for-custom-statements">1)
Extending parsing: wrapping DFParser for custom statements</a></li>
+<li><a href="#2-extending-expression-semantics-exprplanner">2) Extending
expression semantics: ExprPlanner</a><ul>
+<li><a href="#example-postgres-json-operators-">Example: Postgres JSON
operators (->, ->>)</a></li>
+</ul>
+</li>
+<li><a href="#3-extending-type-support-typeplanner">3) Extending type support:
TypePlanner</a></li>
+<li><a href="#4-extending-the-from-clause-relationplanner">4) Extending the
FROM clause: RelationPlanner</a><ul>
+<li><a href="#strategy-a-rewrite-to-existing-operators-pivot-unpivot">Strategy
A: rewrite to existing operators (PIVOT / UNPIVOT)</a></li>
+<li><a href="#strategy-b-custom-logical-physical-tablesample">Strategy B:
custom logical + physical (TABLESAMPLE)</a></li>
+<li><a href="#background-origin-of-the-api">Background: Origin of the
API</a></li>
+</ul>
+</li>
+<li><a href="#summary-the-extensibility-workflow">Summary: The Extensibility
Workflow</a></li>
+<li><a href="#debugging-tips">Debugging tips</a><ul>
+<li><a href="#print-the-logical-plan">Print the logical plan</a></li>
+<li><a href="#use-explain">Use EXPLAIN</a></li>
+</ul>
+</li>
+<li><a href="#when-hooks-arent-enough">When hooks aren't enough</a></li>
+<li><a href="#ideas-to-try">Ideas to try</a></li>
+<li><a href="#see-also">See also</a></li>
+<li><a href="#acknowledgements">Acknowledgements</a></li>
+<li><a href="#get-involved">Get Involved</a></li>
+</ul>
+</div>
+ </aside>
+
+ <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a href="https://datafusion.apache.org/";>DataFusion</a> in
your product, your users will eventually run SQL that DataFusion does not
recognize. Not because the query is unreasonable, but because SQL in practice
includes many dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to attach an
external catalog to query them. DataFusion has <code>CREATE EXTERNAL
TABLE</code> for individual tables, but no built-in equivalent for catalogs.
DuckDB has <code>ATTACH</code>, SQLite has its own variant, and maybe you
really want something even more flexible:</p>
+<pre><code class="language-sql">CREATE EXTERNAL CATALOG my_lake
+STORED AS iceberg
+LOCATION 's3://my-bucket/warehouse'
+OPTIONS ('region' 'eu-west-1');
+</code></pre>
+<p>This syntax does not exist in DataFusion today, but you can add it.</p>
+<hr/>
+<p>At the same time, many dialect gaps are smaller and show up in everyday
queries:</p>
+<pre><code class="language-sql">-- Postgres-style JSON operators
+SELECT payload->'user'->>'id' FROM logs;
+
+-- MySQL-specific types
+SELECT DATETIME '2001-01-01 18:00:00';
+
+-- Statistical sampling
+SELECT * FROM sensor_data TABLESAMPLE BERNOULLI(10 PERCENT);
+</code></pre>
+<p>You can implement all of these <em>without forking</em> DataFusion:</p>
+<ol>
+<li><strong>Parse</strong> new syntax (custom statements / dialect quirks)</li>
+<li><strong>Plan</strong> new semantics (expressions, types, FROM-clause
constructs)</li>
+<li><strong>Execute</strong> new operators when rewrites are not
sufficient</li>
+</ol>
+<p>This post explains where and how to hook into each stage. For complete,
working code, see the linked <code>datafusion-examples</code>.</p>
+<hr/>
+<h2 id="parse-plan-execute">Parse → Plan → Execute<a class="headerlink"
href="#parse-plan-execute" title="Permanent link">¶</a></h2>
+<p>DataFusion turns SQL into executable work in stages:</p>
+<ol>
+<li><strong>Parse</strong>: SQL text is parsed into an AST (<a
href="https://docs.rs/sqlparser/latest/sqlparser/ast/enum.Statement.html";>Statement</a>
from <a
href="https://github.com/sqlparser-rs/sqlparser-rs";>sqlparser-rs</a>)</li>
+<li><strong>Logical planning</strong>: <a
href="https://docs.rs/datafusion/latest/datafusion/sql/planner/struct.SqlToRel.html";>SqlToRel</a>
converts the AST into a <a
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html";>LogicalPlan</a></li>
+<li><strong>Physical planning</strong>: The <a
href="https://docs.rs/datafusion/latest/datafusion/physical_planner/trait.PhysicalPlanner.html";>PhysicalPlanner</a>
turns the logical plan into an <a
href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html";>ExecutionPlan</a></li>
+</ol>
+<p>Each stage has extension points.</p>
+<figure>
+<img alt="DataFusion SQL processing pipeline: SQL String flows through Parser
to AST, then SqlToRel (with Extension Planners) to LogicalPlan, then
PhysicalPlanner to ExecutionPlan" class="img-responsive"
src="/blog/images/extending-sql/architecture.svg" width="100%"/>
+<figcaption>
+<b>Figure 1:</b> SQL flows through three stages: parsing, logical planning
(via <code>SqlToRel</code>, where the Extension Planners hook in), and physical
planning. Each stage has extension points: wrap the parser, implement planner
traits, or add physical operators.
+ </figcaption>
+</figure>
+<p>To choose the right extension point, look at where the query fails.</p>
+<table class="table">
+<thead>
+<tr>
+<th>What fails?</th>
+<th>What it looks like</th>
+<th>Where to hook in</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Parsing</td>
+<td><code>Expected: TABLE, found: CATALOG</code></td>
+<td>configure dialect or wrap <code>DFParser</code></td>
+</tr>
+<tr>
+<td>Planning</td>
+<td><code>This feature is not implemented: DATETIME</code></td>
+<td><code>ExprPlanner</code>, <code>TypePlanner</code>,
<code>RelationPlanner</code></td>
+</tr>
+<tr>
+<td>Execution</td>
+<td><code>No physical plan for TableSample</code></td>
+<td><code>ExtensionPlanner</code> (+ physical operator)</td>
+</tr>
+</tbody>
+</table>
+<p>We will follow that pipeline order.</p>
+<hr/>
+<h2 id="1-extending-parsing-wrapping-dfparser-for-custom-statements">1)
Extending parsing: wrapping <code>DFParser</code> for custom statements<a
class="headerlink"
href="#1-extending-parsing-wrapping-dfparser-for-custom-statements"
title="Permanent link">¶</a></h2>
+<p>The <code>CREATE EXTERNAL CATALOG</code> syntax from the introduction fails
at the parser because DataFusion only recognizes <code>CREATE EXTERNAL
TABLE</code>. To support new statement-level syntax, you can <strong>wrap
<code>DFParser</code></strong>. Peek ahead <strong>in the token stream</strong>
to detect your custom syntax, handle it yourself, and delegate everything else
to DataFusion.</p>
+<p>The <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs";><code>custom_sql_parser.rs</code></a>
example demonstrates this pattern:</p>
+<pre><code class="language-rust">struct CustomParser<'a> { df_parser:
DFParser<'a> }
+
+impl<'a> CustomParser<'a> {
+ pub fn parse_statement(&mut self) -> Result<CustomStatement> {
+ // Peek tokens to detect CREATE EXTERNAL CATALOG
+ if self.is_create_external_catalog() {
+ return self.parse_create_external_catalog();
+ }
+ // Delegate everything else to DataFusion
+ Ok(CustomStatement::DFStatement(Box::new(
+ self.df_parser.parse_statement()?,
+ )))
+ }
+}
+</code></pre>
+<p>You do not need to implement a full SQL parser. Reuse DataFusion's
tokenizer and parser helpers to consume tokens, parse identifiers, and handle
options—the example shows how.</p>
+<p>Once parsed, the simplest integration is to treat custom statements as
<strong>application commands</strong>:</p>
+<pre><code class="language-rust">match parser.parse_statement()? {
+ CustomStatement::DFStatement(stmt) =>
ctx.sql(&stmt.to_string()).await?,
+ CustomStatement::CreateExternalCatalog(stmt) => {
+ handle_create_external_catalog(&ctx, stmt).await?
+ }
+}
+</code></pre>
+<p>This keeps the extension logic in your embedding application. The example
includes a complete <code>handle_create_external_catalog</code> that registers
tables from a location into a catalog, making them queryable immediately.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs";><code>custom_sql_parser.rs</code></a></p>
+<hr/>
+<h2 id="2-extending-expression-semantics-exprplanner">2) Extending expression
semantics: <code>ExprPlanner</code><a class="headerlink"
href="#2-extending-expression-semantics-exprplanner" title="Permanent
link">¶</a></h2>
+<p>Once SQL <em>parses</em>, the next failure is often that DataFusion does
not know what a particular expression means.</p>
+<p>This is where dialect differences show up in day-to-day queries: operators
like Postgres JSON arrows, vendor-specific functions, or small syntactic sugar
that users expect to keep working when you switch engines.</p>
+<p><code>ExprPlanner</code> lets you define how specific SQL expressions
become DataFusion <code>Expr</code>. Common examples:</p>
+<ul>
+<li>Non-standard operators (JSON / geometry / regex operators)</li>
+<li>Custom function syntaxes</li>
+<li>Special identifier behavior</li>
+</ul>
+<h3 id="example-postgres-json-operators-">Example: Postgres JSON operators
(<code>-></code>, <code>->></code>)<a class="headerlink"
href="#example-postgres-json-operators-" title="Permanent link">¶</a></h3>
+<p>The Postgres <code>-></code> operator is a good illustration because it
is widely used and parses only under the PostgreSQL dialect.</p>
+<p>Configure the dialect:</p>
+<pre><code class="language-rust">let config = SessionConfig::new()
+ .set_str("datafusion.sql_parser.dialect", "postgres");
+let ctx = SessionContext::new_with_config(config);
+</code></pre>
+<p>Then implement <code>ExprPlanner</code> to map the parsed operator
(<code>BinaryOperator::Arrow</code>) to DataFusion semantics:</p>
+<pre><code class="language-rust">fn plan_binary_op(&self, expr:
RawBinaryExpr, _schema: &DFSchema)
+ -> Result<PlannerResult<RawBinaryExpr>> {
+ match expr.op {
+ BinaryOperator::Arrow => Ok(Planned(/* your Expr */)),
+ _ => Ok(Original(expr)),
+ }
+}
+</code></pre>
+<p>Return <code>Planned(...)</code> when you handled the expression; return
<code>Original(...)</code> to pass it to the next planner.</p>
+<p>For a complete JSON implementation, see <a
href="https://github.com/datafusion-contrib/datafusion-functions-json";>datafusion-functions-json</a>.
For a minimal end-to-end example in the DataFusion repo, see <a
href="https://github.com/apache/datafusion/blob/main/datafusion/core/tests/user_defined/expr_planner.rs";><code>expr_planner_tests</code></a>.</p>
+<hr/>
+<h2 id="3-extending-type-support-typeplanner">3) Extending type support:
<code>TypePlanner</code><a class="headerlink"
href="#3-extending-type-support-typeplanner" title="Permanent link">¶</a></h2>
+<p>After expressions, types are often the next thing to break. Schemas and DDL
may reference types that DataFusion does not support out of the box, like
MySQL's <code>DATETIME</code>.</p>
+<p>Type planning tends to come up when interoperating with other systems. You
want to accept DDL or infer schemas from external catalogs without forcing
users to rewrite types.</p>
+<p><code>TypePlanner</code> maps SQL types to Arrow/DataFusion types:</p>
+<pre><code class="language-rust">impl TypePlanner for MyTypePlanner {
+ fn plan_type(&self, sql_type: &ast::DataType) ->
Result<Option<DataType>> {
+ match sql_type {
+ ast::DataType::Datetime(Some(3)) =>
Ok(Some(DataType::Timestamp(TimeUnit::Millisecond, None))),
+ _ => Ok(None), // let the default planner handle it
+ }
+ }
+}
+</code></pre>
+<p>It is installed when building session state:</p>
+<pre><code class="language-rust">let state = SessionStateBuilder::new()
+ .with_default_features()
+ .with_type_planner(Arc::new(MyTypePlanner))
+ .build();
+</code></pre>
+<p>Once installed, if your <code>CREATE EXTERNAL CATALOG</code> statement
exposes tables with MySQL types, DataFusion can interpret them correctly.</p>
+<hr/>
+<h2 id="4-extending-the-from-clause-relationplanner">4) Extending the FROM
clause: <code>RelationPlanner</code><a class="headerlink"
href="#4-extending-the-from-clause-relationplanner" title="Permanent
link">¶</a></h2>
+<p>Some extensions change what a <em>relation</em> means, not just expressions
or types. <code>RelationPlanner</code> (available starting in DataFusion 52)
intercepts FROM-clause constructs while SQL is being converted into a
<code>LogicalPlan</code>.</p>
+<p>Once you have <code>RelationPlanner</code>, there are two main approaches
to implementing your extension.</p>
+<h3 id="strategy-a-rewrite-to-existing-operators-pivot-unpivot">Strategy A:
rewrite to existing operators (PIVOT / UNPIVOT)<a class="headerlink"
href="#strategy-a-rewrite-to-existing-operators-pivot-unpivot" title="Permanent
link">¶</a></h3>
+<p>If you can translate your syntax into relational algebra that DataFusion
already supports, you can implement the feature with <strong>no custom physical
operator</strong>.</p>
+<p><code>PIVOT</code> rotates rows into columns, and <code>UNPIVOT</code> does
the reverse. Neither requires new execution logic: <code>PIVOT</code> is just
<code>GROUP BY</code> with <code>CASE</code> expressions, and
<code>UNPIVOT</code> is a <code>UNION ALL</code> of each column. The planner
rewrites them accordingly:</p>
+<pre><code class="language-rust">match relation {
+ TableFactor::Pivot { .. } => /* rewrite to GROUP BY + CASE */,
+ TableFactor::Unpivot { .. } => /* rewrite to UNION ALL */,
+ other => Original(other),
+}
+</code></pre>
+<p>Because the output is a standard <code>LogicalPlan</code>, DataFusion's
usual optimization and physical planning apply automatically.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/pivot_unpivot.rs";><code>pivot_unpivot.rs</code></a></p>
+<h3 id="strategy-b-custom-logical-physical-tablesample">Strategy B: custom
logical + physical (TABLESAMPLE)<a class="headerlink"
href="#strategy-b-custom-logical-physical-tablesample" title="Permanent
link">¶</a></h3>
+<p>Sometimes rewriting is not sufficient. <code>TABLESAMPLE</code> returns a
random subset of rows from a table and is useful for approximations or
debugging on large datasets. Because it requires runtime randomness, you cannot
express it as a rewrite to existing operators. Instead, you need a custom
logical node and physical operator to execute it.</p>
+<p>The approach (shown in <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs";><code>table_sample.rs</code></a>):</p>
+<ol>
+<li><code>RelationPlanner</code> recognizes <code>TABLESAMPLE</code> and
produces a custom logical node</li>
+<li>That node gets wrapped in <code>LogicalPlan::Extension</code></li>
+<li><code>ExtensionPlanner</code> converts it to a custom
<code>ExecutionPlan</code></li>
+</ol>
+<p>In code:</p>
+<pre><code class="language-rust">// Logical planning: FROM t TABLESAMPLE (...)
-> LogicalPlan::Extension(...)
+let plan = LogicalPlan::Extension(Extension { node:
Arc::new(TableSamplePlanNode { /* ... */ }) });
+</code></pre>
+<pre><code class="language-rust">// Physical planning: TableSamplePlanNode
-> SampleExec
+if let Some(sample_node) =
node.as_any().downcast_ref::<TableSamplePlanNode>() {
+ return Ok(Some(Arc::new(SampleExec::try_new(input, /* bounds, seed */)?)));
+}
+</code></pre>
+<p>This is the general pattern for custom FROM constructs that need runtime
behavior.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs";><code>table_sample.rs</code></a></p>
+<h3 id="background-origin-of-the-api">Background: Origin of the API<a
class="headerlink" href="#background-origin-of-the-api" title="Permanent
link">¶</a></h3>
+<p><code>RelationPlanner</code> originally came out of trying to build
<code>MATCH_RECOGNIZE</code> support in DataFusion as a Datadog hackathon
project. <code>MATCH_RECOGNIZE</code> is a complex SQL feature for detecting
patterns in sequences of rows, and it made sense to prototype as an extension
first. At the time, DataFusion had no extension point at the right stage of
SQL-to-rel planning to intercept and reinterpret relations.</p>
+<p><a href="https://github.com/theirix";>@theirix</a>'s
<code>TABLESAMPLE</code> work (<a
href="https://github.com/apache/datafusion/issues/13563";>#13563</a>, <a
href="https://github.com/apache/datafusion/pull/17633";>#17633</a>) demonstrated
exactly where the gap was: their extension only worked when
<code>TABLESAMPLE</code> appeared at the query root and any
<code>TABLESAMPLE</code> inside a CTE or JOIN would error. That limitation
motivated <a href="https://github.com/apache/datafusion/ [...]
+<p>This is how Datadog approaches compatibility work: build features in real
systems first, then upstream the building blocks. A full
<code>MATCH_RECOGNIZE</code> extension is now in progress, built on top of
<code>RelationPlanner</code>, with the <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/match_recognize.rs";><code>match_recognize.rs</code></a>
example as a starting point.</p>
+<hr/>
+<h2 id="summary-the-extensibility-workflow">Summary: The Extensibility
Workflow<a class="headerlink" href="#summary-the-extensibility-workflow"
title="Permanent link">¶</a></h2>
+<p>DataFusion's SQL extensibility follows its processing pipeline. When
building your own dialect extension, work incrementally:</p>
+<ol>
+<li><strong>Parse</strong>: Use a parser wrapper to intercept custom syntax in
the token stream. Produce either a standard <code>Statement</code> or your own
application-specific command.</li>
+<li><strong>Plan</strong>: Implement the planning traits
(<code>ExprPlanner</code>, <code>TypePlanner</code>,
<code>RelationPlanner</code>) to give your syntax meaning.</li>
+<li><strong>Execute</strong>: Prefer rewrites to existing operators (like
<code>PIVOT</code> to <code>CASE</code>). Only add custom physical operators
via <code>ExtensionPlanner</code> when you need specific runtime behavior like
randomness or specialized I/O.</li>
+</ol>
+<hr/>
+<h2 id="debugging-tips">Debugging tips<a class="headerlink"
href="#debugging-tips" title="Permanent link">¶</a></h2>
+<h3 id="print-the-logical-plan">Print the logical plan<a class="headerlink"
href="#print-the-logical-plan" title="Permanent link">¶</a></h3>
+<pre><code class="language-rust">let df = ctx.sql("SELECT * FROM t TABLESAMPLE
(10 PERCENT)").await?;
+println!("{}", df.logical_plan().display_indent());
+</code></pre>
+<h3 id="use-explain">Use <a
href="https://datafusion.apache.org/user-guide/sql/explain.html";><code>EXPLAIN</code></a><a
class="headerlink" href="#use-explain" title="Permanent link">¶</a></h3>
+<pre><code class="language-sql">EXPLAIN SELECT * FROM t TABLESAMPLE (10
PERCENT);
+</code></pre>
+<p>If your extension is not being invoked, it is usually visible in the
logical plan first.</p>
+<hr/>
+<h2 id="when-hooks-arent-enough">When hooks aren't enough<a class="headerlink"
href="#when-hooks-arent-enough" title="Permanent link">¶</a></h2>
+<p>While these extension points cover the majority of dialect needs, some deep
architectural areas still have limited or no hooks. If you are working in these
parts of the SQL surface area, you may need to contribute upstream:</p>
+<ul>
+<li>Statement-level planning: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/statement.rs";><code>statement.rs</code></a></li>
+<li>JOIN planning: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/relation/join.rs";><code>relation/join.rs</code></a></li>
+<li>TOP / FETCH clauses: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/select.rs";><code>select.rs</code></a>,
<a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/query.rs";><code>query.rs</code></a></li>
+</ul>
+<hr/>
+<h2 id="ideas-to-try">Ideas to try<a class="headerlink" href="#ideas-to-try"
title="Permanent link">¶</a></h2>
+<p>If you want to experiment with these extension points, here are a few
suggestions:</p>
+<ul>
+<li>Geometry operators (for example <code>@></code>, <code><@</code>)
via <code>ExprPlanner</code></li>
+<li>Oracle <code>NUMBER</code> or SQL Server <code>MONEY</code> via
<code>TypePlanner</code></li>
+<li><code>JSON_TABLE</code> or semantic-layer style relations via
<code>RelationPlanner</code></li>
+</ul>
+<hr/>
+<h2 id="see-also">See also<a class="headerlink" href="#see-also"
title="Permanent link">¶</a></h2>
+<ul>
+<li>Extending SQL Guide: <a
href="https://datafusion.apache.org/library-user-guide/extending-sql.html";>Extending
SQL Guide</a></li>
+<li>Parser wrapping example: <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs";><code>custom_sql_parser.rs</code></a></li>
+<li>
+<p>RelationPlanner examples:</p>
+</li>
+<li>
+<p><code>PIVOT</code> / <code>UNPIVOT</code>: <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/pivot_unpivot.rs";><code>pivot_unpivot.rs</code></a></p>
+</li>
+<li>
+<p><code>TABLESAMPLE</code>: <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs";><code>table_sample.rs</code></a></p>
+</li>
+<li>
+<p>ExprPlanner test examples: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/core/tests/user_defined/expr_planner.rs";><code>expr_planner_tests</code></a></p>
+</li>
+</ul>
+<h2 id="acknowledgements">Acknowledgements<a class="headerlink"
href="#acknowledgements" title="Permanent link">¶</a></h2>
+<p>Thank you to <a href="https://github.com/jayzhan211";>@jayzhan211</a> for
designing and implementing the original <code>ExprPlanner</code> API (<a
href="https://github.com/apache/datafusion/pull/11180";>#11180</a>), to <a
href="https://github.com/goldmedal";>@goldmedal</a> for adding
<code>TypePlanner</code> (<a
href="https://github.com/apache/datafusion/pull/13294";>#13294</a>), and to <a
href="https://github.com/theirix";>@theirix</a> for the <code>TABLESAMPLE</code>
work (<a href="https [...]
+<h2 id="get-involved">Get Involved<a class="headerlink" href="#get-involved"
title="Permanent link">¶</a></h2>
+<ul>
+<li><strong>Try it out</strong>: Implement one of the extension points and
share your experience</li>
+<li><strong>File issues or join the conversation</strong>: <a
href="https://github.com/apache/datafusion/";>GitHub</a> for bugs and feature
requests, <a
href="https://datafusion.apache.org/contributor-guide/communication.html";>Slack
or Discord</a> for discussion</li>
+</ul>
+<!-- Reference links -->
+
+<!--
+ Comments Section
+ Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div id="comments">
+ <hr>
+ <h3>Comments</h3>
+
+ <!-- Local loader script -->
+ <script src="/content/js/giscus-consent.js" defer></script>
+
+ <!-- Consent UI -->
+ <div id="giscus-consent">
+ <p>
+ We use <a href="https://giscus.app/";>Giscus</a> for comments, powered
by GitHub Discussions.
+ To respect your privacy, Giscus and comments will load only if you
click "Show Comments"
+ </p>
+
+ <div class="consent-actions">
+ <button id="giscus-load" type="button">Show Comments</button>
+ <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+ </div>
+
+ <noscript>JavaScript is required to load comments from Giscus.</noscript>
+ </div>
+
+ <!-- Container where Giscus will render -->
+ <div id="comment-thread"></div>
+</div> </div>
+ <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+ <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#parse-plan-execute">Parse → Plan → Execute</a></li>
+<li><a href="#1-extending-parsing-wrapping-dfparser-for-custom-statements">1)
Extending parsing: wrapping DFParser for custom statements</a></li>
+<li><a href="#2-extending-expression-semantics-exprplanner">2) Extending
expression semantics: ExprPlanner</a><ul>
+<li><a href="#example-postgres-json-operators-">Example: Postgres JSON
operators (->, ->>)</a></li>
+</ul>
+</li>
+<li><a href="#3-extending-type-support-typeplanner">3) Extending type support:
TypePlanner</a></li>
+<li><a href="#4-extending-the-from-clause-relationplanner">4) Extending the
FROM clause: RelationPlanner</a><ul>
+<li><a href="#strategy-a-rewrite-to-existing-operators-pivot-unpivot">Strategy
A: rewrite to existing operators (PIVOT / UNPIVOT)</a></li>
+<li><a href="#strategy-b-custom-logical-physical-tablesample">Strategy B:
custom logical + physical (TABLESAMPLE)</a></li>
+<li><a href="#background-origin-of-the-api">Background: Origin of the
API</a></li>
+</ul>
+</li>
+<li><a href="#summary-the-extensibility-workflow">Summary: The Extensibility
Workflow</a></li>
+<li><a href="#debugging-tips">Debugging tips</a><ul>
+<li><a href="#print-the-logical-plan">Print the logical plan</a></li>
+<li><a href="#use-explain">Use EXPLAIN</a></li>
+</ul>
+</li>
+<li><a href="#when-hooks-arent-enough">When hooks aren't enough</a></li>
+<li><a href="#ideas-to-try">Ideas to try</a></li>
+<li><a href="#see-also">See also</a></li>
+<li><a href="#acknowledgements">Acknowledgements</a></li>
+<li><a href="#get-involved">Get Involved</a></li>
+</ul>
+</div>
+ </aside>
+ </div>
+ </div>
+</div>
+ <!-- footer -->
+ <div class="row g-0">
+ <div class="col-12">
+ <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+ Copyright 2026, <a href="https://www.apache.org/";>The Apache
Software Foundation</a>, Licensed under the <a
href="https://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version
2.0</a>.<br/>
+ Apache® and the Apache feather logo are trademarks of The Apache
Software Foundation.
+ </p>
+ </div>
+ </div>
+ <script src="/blog/js/bootstrap.bundle.min.js"></script> </main>
+ </body>
+</html>
diff --git a/output/author/geoffrey-claude-datadog.html
b/output/author/geoffrey-claude-datadog.html
new file mode 100644
index 0000000..f6df445
--- /dev/null
+++ b/output/author/geoffrey-claude-datadog.html
@@ -0,0 +1,62 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <title>Apache DataFusion Blog - Articles by Geoffrey Claude
(Datadog)</title>
+ <meta charset="utf-8" />
+ <meta name="generator" content="Pelican" />
+ <link href="https://datafusion.apache.org/blog/feed.xml";
type="application/rss+xml" rel="alternate" title="Apache DataFusion Blog RSS
Feed" />
+</head>
+
+<body id="index" class="home">
+ <header id="banner" class="body">
+ <h1><a href="https://datafusion.apache.org/blog/";>Apache
DataFusion Blog</a></h1>
+ </header><!-- /#banner -->
+ <nav id="menu"><ul>
+ <li><a
href="https://datafusion.apache.org/blog/pages/about.html";>About</a></li>
+ <li><a
href="https://datafusion.apache.org/blog/pages/index.html";>index</a></li>
+ <li><a
href="https://datafusion.apache.org/blog/category/blog.html";>blog</a></li>
+ </ul></nav><!-- /#menu -->
+<section id="content">
+<h2>Articles by Geoffrey Claude (Datadog)</h2>
+
+<ol id="post-list">
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a
href="https://datafusion.apache.org/blog/2026/01/12/extending-sql";
rel="bookmark" title="Permalink to Extending SQL in DataFusion: from ->> to
TABLESAMPLE">Extending SQL in DataFusion: from ->> to TABLESAMPLE</a></h2>
</header>
+ <footer class="post-info">
+ <time class="published"
datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
+ <address class="vcard author">By
+ <a class="url fn"
href="https://datafusion.apache.org/blog/author/geoffrey-claude-datadog.html";>Geoffrey
Claude (Datadog)</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a href="https://datafusion.apache.org/";>DataFusion</a> in
your product, your users will eventually run SQL that DataFusion does not
recognize. Not because the query is unreasonable, but because SQL in practice
includes many dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to attach an
…</p> </div><!-- /.entry-content -->
+ </article></li>
+</ol><!-- /#posts-list -->
+</section><!-- /#content -->
+ <footer id="contentinfo" class="body">
+ <address id="about" class="vcard body">
+ Proudly powered by <a
href="https://getpelican.com/";>Pelican</a>,
+ which takes great advantage of <a
href="https://www.python.org/";>Python</a>.
+ </address><!-- /#about -->
+ </footer><!-- /#contentinfo -->
+</body>
+</html>
\ No newline at end of file
diff --git a/output/category/blog.html b/output/category/blog.html
index c027263..538cd20 100644
--- a/output/category/blog.html
+++ b/output/category/blog.html
@@ -21,6 +21,36 @@
<h2>Articles in the blog category</h2>
<ol id="post-list">
+ <li><article class="hentry">
+ <header> <h2 class="entry-title"><a
href="https://datafusion.apache.org/blog/2026/01/12/extending-sql";
rel="bookmark" title="Permalink to Extending SQL in DataFusion: from ->> to
TABLESAMPLE">Extending SQL in DataFusion: from ->> to TABLESAMPLE</a></h2>
</header>
+ <footer class="post-info">
+ <time class="published"
datetime="2026-01-12T00:00:00+00:00"> Mon 12 January 2026 </time>
+ <address class="vcard author">By
+ <a class="url fn"
href="https://datafusion.apache.org/blog/author/geoffrey-claude-datadog.html";>Geoffrey
Claude (Datadog)</a>
+ </address>
+ </footer><!-- /.post-info -->
+ <div class="entry-content"> <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a href="https://datafusion.apache.org/";>DataFusion</a> in
your product, your users will eventually run SQL that DataFusion does not
recognize. Not because the query is unreasonable, but because SQL in practice
includes many dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to attach an
…</p> </div><!-- /.entry-content -->
+ </article></li>
<li><article class="hentry">
<header> <h2 class="entry-title"><a
href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions";
rel="bookmark" title="Permalink to Optimizing Repartitions in DataFusion: How
I Went From Database Noob to Core Contribution">Optimizing Repartitions in
DataFusion: How I Went From Database Noob to Core Contribution</a></h2>
</header>
<footer class="post-info">
diff --git a/output/feed.xml b/output/feed.xml
index d95ba53..35ce56b 100644
--- a/output/feed.xml
+++ b/output/feed.xml
@@ -1,5 +1,25 @@
<?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0"><channel><title>Apache DataFusion
Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon,
15 Dec 2025 00:00:00 +0000</lastBuildDate><item><title>Optimizing Repartitions
in DataFusion: How I Went From Database Noob to Core
Contribution</title><link>https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions</link><description><!--
+<rss version="2.0"><channel><title>Apache DataFusion
Blog</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon,
12 Jan 2026 00:00:00 +0000</lastBuildDate><item><title>Extending SQL in
DataFusion: from ->> to
TABLESAMPLE</title><link>https://datafusion.apache.org/blog/2026/01/12/extending-sql</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a
href="https://datafusion.apache.org/">DataFusion</a> in your product,
your users will eventually run SQL that DataFusion does not recognize. Not
because the query is unreasonable, but because SQL in practice includes many
dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to
attach an …</p></description><dc:creator
xmlns:dc="http://purl.org/dc/elements/1.1/";>Geoffrey Claude
(Datadog)</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid
isPermaLink="false">tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/extending-sql</guid><category>blog</category></item><item><title>Optimizing
Repartitions in DataFusion: How I Went From Database Noob to Core
Contribution</titl [...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/output/feeds/all-en.atom.xml b/output/feeds/all-en.atom.xml
index 1182ba7..490c1f9 100644
--- a/output/feeds/all-en.atom.xml
+++ b/output/feeds/all-en.atom.xml
@@ -1,5 +1,289 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion
Blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-12-15T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Optimizing
Repartitions in DataFusion: How I Went From Database Noob to Core
Contribution</title><link href="https://datafusion.ap [...]
+<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion
Blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-01-12T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Extending
SQL in DataFusion: from ->> to TABLESAMPLE</title><link
href="https://datafusion.apache.org/blog/2026/01/12/extend [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a
href="https://datafusion.apache.org/">DataFusion</a> in your product,
your users will eventually run SQL that DataFusion does not recognize. Not
because the query is unreasonable, but because SQL in practice includes many
dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to
attach an …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a
href="https://datafusion.apache.org/">DataFusion</a> in your product,
your users will eventually run SQL that DataFusion does not recognize. Not
because the query is unreasonable, but because SQL in practice includes many
dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to
attach an external catalog to query them. DataFusion has <code>CREATE
EXTERNAL TABLE</code> for individual tables, but no built-in equivalent
for catalogs. DuckDB has <code>ATTACH</code>, SQLite has its own
variant, and maybe you really want something even more flexible:</p>
+<pre><code class="language-sql">CREATE EXTERNAL CATALOG my_lake
+STORED AS iceberg
+LOCATION 's3://my-bucket/warehouse'
+OPTIONS ('region' 'eu-west-1');
+</code></pre>
+<p>This syntax does not exist in DataFusion today, but you can add
it.</p>
+<hr/>
+<p>At the same time, many dialect gaps are smaller and show up in
everyday queries:</p>
+<pre><code class="language-sql">-- Postgres-style JSON operators
+SELECT payload-&gt;'user'-&gt;&gt;'id' FROM logs;
+
+-- MySQL-specific types
+SELECT DATETIME '2001-01-01 18:00:00';
+
+-- Statistical sampling
+SELECT * FROM sensor_data TABLESAMPLE BERNOULLI(10 PERCENT);
+</code></pre>
+<p>You can implement all of these <em>without forking</em>
DataFusion:</p>
+<ol>
+<li><strong>Parse</strong> new syntax (custom statements /
dialect quirks)</li>
+<li><strong>Plan</strong> new semantics (expressions, types,
FROM-clause constructs)</li>
+<li><strong>Execute</strong> new operators when rewrites are
not sufficient</li>
+</ol>
+<p>This post explains where and how to hook into each stage. For
complete, working code, see the linked
<code>datafusion-examples</code>.</p>
+<hr/>
+<h2 id="parse-plan-execute">Parse → Plan → Execute<a
class="headerlink" href="#parse-plan-execute" title="Permanent
link">¶</a></h2>
+<p>DataFusion turns SQL into executable work in stages:</p>
+<ol>
+<li><strong>Parse</strong>: SQL text is parsed into an AST
(<a
href="https://docs.rs/sqlparser/latest/sqlparser/ast/enum.Statement.html">Statement</a>
from <a
href="https://github.com/sqlparser-rs/sqlparser-rs">sqlparser-rs</a>)</li>
+<li><strong>Logical planning</strong>: <a
href="https://docs.rs/datafusion/latest/datafusion/sql/planner/struct.SqlToRel.html">SqlToRel</a>
converts the AST into a <a
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html">LogicalPlan</a></li>
+<li><strong>Physical planning</strong>: The <a
href="https://docs.rs/datafusion/latest/datafusion/physical_planner/trait.PhysicalPlanner.html">PhysicalPlanner</a>
turns the logical plan into an <a
href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html">ExecutionPlan</a></li>
+</ol>
+<p>Each stage has extension points.</p>
+<figure>
+<img alt="DataFusion SQL processing pipeline: SQL String flows through
Parser to AST, then SqlToRel (with Extension Planners) to LogicalPlan, then
PhysicalPlanner to ExecutionPlan" class="img-responsive"
src="/blog/images/extending-sql/architecture.svg" width="100%"/>
+<figcaption>
+<b>Figure 1:</b> SQL flows through three stages: parsing, logical
planning (via <code>SqlToRel</code>, where the Extension Planners
hook in), and physical planning. Each stage has extension points: wrap the
parser, implement planner traits, or add physical operators.
+ </figcaption>
+</figure>
+<p>To choose the right extension point, look at where the query
fails.</p>
+<table class="table">
+<thead>
+<tr>
+<th>What fails?</th>
+<th>What it looks like</th>
+<th>Where to hook in</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Parsing</td>
+<td><code>Expected: TABLE, found: CATALOG</code></td>
+<td>configure dialect or wrap
<code>DFParser</code></td>
+</tr>
+<tr>
+<td>Planning</td>
+<td><code>This feature is not implemented:
DATETIME</code></td>
+<td><code>ExprPlanner</code>,
<code>TypePlanner</code>,
<code>RelationPlanner</code></td>
+</tr>
+<tr>
+<td>Execution</td>
+<td><code>No physical plan for TableSample</code></td>
+<td><code>ExtensionPlanner</code> (+ physical
operator)</td>
+</tr>
+</tbody>
+</table>
+<p>We will follow that pipeline order.</p>
+<hr/>
+<h2 id="1-extending-parsing-wrapping-dfparser-for-custom-statements">1)
Extending parsing: wrapping <code>DFParser</code> for custom
statements<a class="headerlink"
href="#1-extending-parsing-wrapping-dfparser-for-custom-statements"
title="Permanent link">¶</a></h2>
+<p>The <code>CREATE EXTERNAL CATALOG</code> syntax from the
introduction fails at the parser because DataFusion only recognizes
<code>CREATE EXTERNAL TABLE</code>. To support new statement-level
syntax, you can <strong>wrap
<code>DFParser</code></strong>. Peek ahead <strong>in
the token stream</strong> to detect your custom syntax, handle it
yourself, and delegate everything else to DataFusion.</p>
+<p>The <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs"><code>custom_sql_parser.rs</code></a>
example demonstrates this pattern:</p>
+<pre><code class="language-rust">struct
CustomParser&lt;'a&gt; { df_parser: DFParser&lt;'a&gt; }
+
+impl&lt;'a&gt; CustomParser&lt;'a&gt; {
+ pub fn parse_statement(&amp;mut self) -&gt;
Result&lt;CustomStatement&gt; {
+ // Peek tokens to detect CREATE EXTERNAL CATALOG
+ if self.is_create_external_catalog() {
+ return self.parse_create_external_catalog();
+ }
+ // Delegate everything else to DataFusion
+ Ok(CustomStatement::DFStatement(Box::new(
+ self.df_parser.parse_statement()?,
+ )))
+ }
+}
+</code></pre>
+<p>You do not need to implement a full SQL parser. Reuse DataFusion's
tokenizer and parser helpers to consume tokens, parse identifiers, and handle
options—the example shows how.</p>
+<p>Once parsed, the simplest integration is to treat custom statements
as <strong>application commands</strong>:</p>
+<pre><code class="language-rust">match parser.parse_statement()? {
+ CustomStatement::DFStatement(stmt) =&gt;
ctx.sql(&amp;stmt.to_string()).await?,
+ CustomStatement::CreateExternalCatalog(stmt) =&gt; {
+ handle_create_external_catalog(&amp;ctx, stmt).await?
+ }
+}
+</code></pre>
+<p>This keeps the extension logic in your embedding application. The
example includes a complete
<code>handle_create_external_catalog</code> that registers tables
from a location into a catalog, making them queryable immediately.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs"><code>custom_sql_parser.rs</code></a></p>
+<hr/>
+<h2 id="2-extending-expression-semantics-exprplanner">2) Extending
expression semantics: <code>ExprPlanner</code><a
class="headerlink" href="#2-extending-expression-semantics-exprplanner"
title="Permanent link">¶</a></h2>
+<p>Once SQL <em>parses</em>, the next failure is often that
DataFusion does not know what a particular expression means.</p>
+<p>This is where dialect differences show up in day-to-day queries:
operators like Postgres JSON arrows, vendor-specific functions, or small
syntactic sugar that users expect to keep working when you switch
engines.</p>
+<p><code>ExprPlanner</code> lets you define how specific SQL
expressions become DataFusion <code>Expr</code>. Common
examples:</p>
+<ul>
+<li>Non-standard operators (JSON / geometry / regex operators)</li>
+<li>Custom function syntaxes</li>
+<li>Special identifier behavior</li>
+</ul>
+<h3 id="example-postgres-json-operators-">Example: Postgres JSON
operators (<code>-&gt;</code>,
<code>-&gt;&gt;</code>)<a class="headerlink"
href="#example-postgres-json-operators-" title="Permanent
link">¶</a></h3>
+<p>The Postgres <code>-&gt;</code> operator is a good
illustration because it is widely used and parses only under the PostgreSQL
dialect.</p>
+<p>Configure the dialect:</p>
+<pre><code class="language-rust">let config = SessionConfig::new()
+ .set_str("datafusion.sql_parser.dialect", "postgres");
+let ctx = SessionContext::new_with_config(config);
+</code></pre>
+<p>Then implement <code>ExprPlanner</code> to map the parsed
operator (<code>BinaryOperator::Arrow</code>) to DataFusion
semantics:</p>
+<pre><code class="language-rust">fn plan_binary_op(&amp;self,
expr: RawBinaryExpr, _schema: &amp;DFSchema)
+ -&gt; Result&lt;PlannerResult&lt;RawBinaryExpr&gt;&gt; {
+ match expr.op {
+ BinaryOperator::Arrow =&gt; Ok(Planned(/* your Expr */)),
+ _ =&gt; Ok(Original(expr)),
+ }
+}
+</code></pre>
+<p>Return <code>Planned(...)</code> when you handled the
expression; return <code>Original(...)</code> to pass it to the
next planner.</p>
+<p>For a complete JSON implementation, see <a
href="https://github.com/datafusion-contrib/datafusion-functions-json">datafusion-functions-json</a>.
For a minimal end-to-end example in the DataFusion repo, see <a
href="https://github.com/apache/datafusion/blob/main/datafusion/core/tests/user_defined/expr_planner.rs"><code>expr_planner_tests</code></a>.</p>
+<hr/>
+<h2 id="3-extending-type-support-typeplanner">3) Extending type support:
<code>TypePlanner</code><a class="headerlink"
href="#3-extending-type-support-typeplanner" title="Permanent
link">¶</a></h2>
+<p>After expressions, types are often the next thing to break. Schemas
and DDL may reference types that DataFusion does not support out of the box,
like MySQL's <code>DATETIME</code>.</p>
+<p>Type planning tends to come up when interoperating with other
systems. You want to accept DDL or infer schemas from external catalogs without
forcing users to rewrite types.</p>
+<p><code>TypePlanner</code> maps SQL types to
Arrow/DataFusion types:</p>
+<pre><code class="language-rust">impl TypePlanner for
MyTypePlanner {
+ fn plan_type(&amp;self, sql_type: &amp;ast::DataType) -&gt;
Result&lt;Option&lt;DataType&gt;&gt; {
+ match sql_type {
+ ast::DataType::Datetime(Some(3)) =&gt;
Ok(Some(DataType::Timestamp(TimeUnit::Millisecond, None))),
+ _ =&gt; Ok(None), // let the default planner handle it
+ }
+ }
+}
+</code></pre>
+<p>It is installed when building session state:</p>
+<pre><code class="language-rust">let state =
SessionStateBuilder::new()
+ .with_default_features()
+ .with_type_planner(Arc::new(MyTypePlanner))
+ .build();
+</code></pre>
+<p>Once installed, if your <code>CREATE EXTERNAL
CATALOG</code> statement exposes tables with MySQL types, DataFusion can
interpret them correctly.</p>
+<hr/>
+<h2 id="4-extending-the-from-clause-relationplanner">4) Extending the
FROM clause: <code>RelationPlanner</code><a class="headerlink"
href="#4-extending-the-from-clause-relationplanner" title="Permanent
link">¶</a></h2>
+<p>Some extensions change what a <em>relation</em> means,
not just expressions or types. <code>RelationPlanner</code>
(available starting in DataFusion 52) intercepts FROM-clause constructs while
SQL is being converted into a <code>LogicalPlan</code>.</p>
+<p>Once you have <code>RelationPlanner</code>, there are two
main approaches to implementing your extension.</p>
+<h3 id="strategy-a-rewrite-to-existing-operators-pivot-unpivot">Strategy
A: rewrite to existing operators (PIVOT / UNPIVOT)<a class="headerlink"
href="#strategy-a-rewrite-to-existing-operators-pivot-unpivot" title="Permanent
link">¶</a></h3>
+<p>If you can translate your syntax into relational algebra that
DataFusion already supports, you can implement the feature with
<strong>no custom physical operator</strong>.</p>
+<p><code>PIVOT</code> rotates rows into columns, and
<code>UNPIVOT</code> does the reverse. Neither requires new
execution logic: <code>PIVOT</code> is just <code>GROUP
BY</code> with <code>CASE</code> expressions, and
<code>UNPIVOT</code> is a <code>UNION ALL</code> of
each column. The planner rewrites them accordingly:</p>
+<pre><code class="language-rust">match relation {
+ TableFactor::Pivot { .. } =&gt; /* rewrite to GROUP BY + CASE */,
+ TableFactor::Unpivot { .. } =&gt; /* rewrite to UNION ALL */,
+ other =&gt; Original(other),
+}
+</code></pre>
+<p>Because the output is a standard
<code>LogicalPlan</code>, DataFusion's usual optimization and
physical planning apply automatically.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/pivot_unpivot.rs"><code>pivot_unpivot.rs</code></a></p>
+<h3 id="strategy-b-custom-logical-physical-tablesample">Strategy B:
custom logical + physical (TABLESAMPLE)<a class="headerlink"
href="#strategy-b-custom-logical-physical-tablesample" title="Permanent
link">¶</a></h3>
+<p>Sometimes rewriting is not sufficient.
<code>TABLESAMPLE</code> returns a random subset of rows from a
table and is useful for approximations or debugging on large datasets. Because
it requires runtime randomness, you cannot express it as a rewrite to existing
operators. Instead, you need a custom logical node and physical operator to
execute it.</p>
+<p>The approach (shown in <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs"><code>table_sample.rs</code></a>):</p>
+<ol>
+<li><code>RelationPlanner</code> recognizes
<code>TABLESAMPLE</code> and produces a custom logical
node</li>
+<li>That node gets wrapped in
<code>LogicalPlan::Extension</code></li>
+<li><code>ExtensionPlanner</code> converts it to a custom
<code>ExecutionPlan</code></li>
+</ol>
+<p>In code:</p>
+<pre><code class="language-rust">// Logical planning: FROM t
TABLESAMPLE (...) -&gt; LogicalPlan::Extension(...)
+let plan = LogicalPlan::Extension(Extension { node:
Arc::new(TableSamplePlanNode { /* ... */ }) });
+</code></pre>
+<pre><code class="language-rust">// Physical planning:
TableSamplePlanNode -&gt; SampleExec
+if let Some(sample_node) =
node.as_any().downcast_ref::&lt;TableSamplePlanNode&gt;() {
+ return Ok(Some(Arc::new(SampleExec::try_new(input, /* bounds, seed */)?)));
+}
+</code></pre>
+<p>This is the general pattern for custom FROM constructs that need
runtime behavior.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs"><code>table_sample.rs</code></a></p>
+<h3 id="background-origin-of-the-api">Background: Origin of the API<a
class="headerlink" href="#background-origin-of-the-api" title="Permanent
link">¶</a></h3>
+<p><code>RelationPlanner</code> originally came out of
trying to build <code>MATCH_RECOGNIZE</code> support in DataFusion
as a Datadog hackathon project. <code>MATCH_RECOGNIZE</code> is a
complex SQL feature for detecting patterns in sequences of rows, and it made
sense to prototype as an extension first. At the time, DataFusion had no
extension point at the right stage of SQL-to-rel planning to intercept and
reinterpret relations.</p>
+<p><a href="https://github.com/theirix">@theirix</a>'s
<code>TABLESAMPLE</code> work (<a
href="https://github.com/apache/datafusion/issues/13563">#13563</a>,
<a
href="https://github.com/apache/datafusion/pull/17633">#17633</a>)
demonstrated exactly where the gap was: their extension only worked when
<code>TABLESAMPLE</code> appeared at the query root and any
<code>TABLESAMPLE</code> inside a CTE or JOIN would e [...]
+<p>This is how Datadog approaches compatibility work: build features in
real systems first, then upstream the building blocks. A full
<code>MATCH_RECOGNIZE</code> extension is now in progress, built on
top of <code>RelationPlanner</code>, with the <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/match_recognize.rs"><code>match_recognize.rs</code></a>
example as a starting point.</p>
+<hr/>
+<h2 id="summary-the-extensibility-workflow">Summary: The Extensibility
Workflow<a class="headerlink" href="#summary-the-extensibility-workflow"
title="Permanent link">¶</a></h2>
+<p>DataFusion's SQL extensibility follows its processing pipeline. When
building your own dialect extension, work incrementally:</p>
+<ol>
+<li><strong>Parse</strong>: Use a parser wrapper to
intercept custom syntax in the token stream. Produce either a standard
<code>Statement</code> or your own application-specific
command.</li>
+<li><strong>Plan</strong>: Implement the planning traits
(<code>ExprPlanner</code>, <code>TypePlanner</code>,
<code>RelationPlanner</code>) to give your syntax
meaning.</li>
+<li><strong>Execute</strong>: Prefer rewrites to existing
operators (like <code>PIVOT</code> to
<code>CASE</code>). Only add custom physical operators via
<code>ExtensionPlanner</code> when you need specific runtime
behavior like randomness or specialized I/O.</li>
+</ol>
+<hr/>
+<h2 id="debugging-tips">Debugging tips<a class="headerlink"
href="#debugging-tips" title="Permanent link">¶</a></h2>
+<h3 id="print-the-logical-plan">Print the logical plan<a
class="headerlink" href="#print-the-logical-plan" title="Permanent
link">¶</a></h3>
+<pre><code class="language-rust">let df = ctx.sql("SELECT * FROM t
TABLESAMPLE (10 PERCENT)").await?;
+println!("{}", df.logical_plan().display_indent());
+</code></pre>
+<h3 id="use-explain">Use <a
href="https://datafusion.apache.org/user-guide/sql/explain.html"><code>EXPLAIN</code></a><a
class="headerlink" href="#use-explain" title="Permanent
link">¶</a></h3>
+<pre><code class="language-sql">EXPLAIN SELECT * FROM t
TABLESAMPLE (10 PERCENT);
+</code></pre>
+<p>If your extension is not being invoked, it is usually visible in the
logical plan first.</p>
+<hr/>
+<h2 id="when-hooks-arent-enough">When hooks aren't enough<a
class="headerlink" href="#when-hooks-arent-enough" title="Permanent
link">¶</a></h2>
+<p>While these extension points cover the majority of dialect needs,
some deep architectural areas still have limited or no hooks. If you are
working in these parts of the SQL surface area, you may need to contribute
upstream:</p>
+<ul>
+<li>Statement-level planning: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/statement.rs"><code>statement.rs</code></a></li>
+<li>JOIN planning: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/relation/join.rs"><code>relation/join.rs</code></a></li>
+<li>TOP / FETCH clauses: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/select.rs"><code>select.rs</code></a>,
<a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/query.rs"><code>query.rs</code></a></li>
+</ul>
+<hr/>
+<h2 id="ideas-to-try">Ideas to try<a class="headerlink"
href="#ideas-to-try" title="Permanent link">¶</a></h2>
+<p>If you want to experiment with these extension points, here are a few
suggestions:</p>
+<ul>
+<li>Geometry operators (for example <code>@&gt;</code>,
<code>&lt;@</code>) via
<code>ExprPlanner</code></li>
+<li>Oracle <code>NUMBER</code> or SQL Server
<code>MONEY</code> via
<code>TypePlanner</code></li>
+<li><code>JSON_TABLE</code> or semantic-layer style
relations via <code>RelationPlanner</code></li>
+</ul>
+<hr/>
+<h2 id="see-also">See also<a class="headerlink" href="#see-also"
title="Permanent link">¶</a></h2>
+<ul>
+<li>Extending SQL Guide: <a
href="https://datafusion.apache.org/library-user-guide/extending-sql.html">Extending
SQL Guide</a></li>
+<li>Parser wrapping example: <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs"><code>custom_sql_parser.rs</code></a></li>
+<li>
+<p>RelationPlanner examples:</p>
+</li>
+<li>
+<p><code>PIVOT</code> / <code>UNPIVOT</code>:
<a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/pivot_unpivot.rs"><code>pivot_unpivot.rs</code></a></p>
+</li>
+<li>
+<p><code>TABLESAMPLE</code>: <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs"><code>table_sample.rs</code></a></p>
+</li>
+<li>
+<p>ExprPlanner test examples: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/core/tests/user_defined/expr_planner.rs"><code>expr_planner_tests</code></a></p>
+</li>
+</ul>
+<h2 id="acknowledgements">Acknowledgements<a class="headerlink"
href="#acknowledgements" title="Permanent link">¶</a></h2>
+<p>Thank you to <a
href="https://github.com/jayzhan211">@jayzhan211</a> for designing and
implementing the original <code>ExprPlanner</code> API (<a
href="https://github.com/apache/datafusion/pull/11180">#11180</a>), to
<a href="https://github.com/goldmedal">@goldmedal</a> for adding
<code>TypePlanner</code> (<a
href="https://github.com/apache/datafusion/pull/13294">#13294</a>),
and to <a href="https://githu [...]
+<h2 id="get-involved">Get Involved<a class="headerlink"
href="#get-involved" title="Permanent link">¶</a></h2>
+<ul>
+<li><strong>Try it out</strong>: Implement one of the
extension points and share your experience</li>
+<li><strong>File issues or join the conversation</strong>:
<a href="https://github.com/apache/datafusion/">GitHub</a> for bugs
and feature requests, <a
href="https://datafusion.apache.org/contributor-guide/communication.html">Slack
or Discord</a> for discussion</li>
+</ul>
+<!-- Reference links --></content><category
term="blog"></category></entry><entry><title>Optimizing Repartitions in
DataFusion: How I Went From Database Noob to Core Contribution</title><link
href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions";
rel="alternate"></link><published>2025-12-15T00:00:00+00:00</published><updated>2025-12-15T00:00:00+00:00</updated><author><name>Gene
Bordegaray</name></author><id>tag:datafusion.apache.org,2025-12-15:/blog/202
[...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/output/feeds/blog.atom.xml b/output/feeds/blog.atom.xml
index 6154174..2645236 100644
--- a/output/feeds/blog.atom.xml
+++ b/output/feeds/blog.atom.xml
@@ -1,5 +1,289 @@
<?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/blog.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2025-12-15T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Optimizing
Repartitions in DataFusion: How I Went From Database Noob to Core
Contribution</title><link href="https://datafusi [...]
+<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
blog</title><link href="https://datafusion.apache.org/blog/";
rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/blog.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-01-12T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Extending
SQL in DataFusion: from ->> to TABLESAMPLE</title><link
href="https://datafusion.apache.org/blog/2026/01/12/e [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a
href="https://datafusion.apache.org/">DataFusion</a> in your product,
your users will eventually run SQL that DataFusion does not recognize. Not
because the query is unreasonable, but because SQL in practice includes many
dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to
attach an …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a
href="https://datafusion.apache.org/">DataFusion</a> in your product,
your users will eventually run SQL that DataFusion does not recognize. Not
because the query is unreasonable, but because SQL in practice includes many
dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to
attach an external catalog to query them. DataFusion has <code>CREATE
EXTERNAL TABLE</code> for individual tables, but no built-in equivalent
for catalogs. DuckDB has <code>ATTACH</code>, SQLite has its own
variant, and maybe you really want something even more flexible:</p>
+<pre><code class="language-sql">CREATE EXTERNAL CATALOG my_lake
+STORED AS iceberg
+LOCATION 's3://my-bucket/warehouse'
+OPTIONS ('region' 'eu-west-1');
+</code></pre>
+<p>This syntax does not exist in DataFusion today, but you can add
it.</p>
+<hr/>
+<p>At the same time, many dialect gaps are smaller and show up in
everyday queries:</p>
+<pre><code class="language-sql">-- Postgres-style JSON operators
+SELECT payload-&gt;'user'-&gt;&gt;'id' FROM logs;
+
+-- MySQL-specific types
+SELECT DATETIME '2001-01-01 18:00:00';
+
+-- Statistical sampling
+SELECT * FROM sensor_data TABLESAMPLE BERNOULLI(10 PERCENT);
+</code></pre>
+<p>You can implement all of these <em>without forking</em>
DataFusion:</p>
+<ol>
+<li><strong>Parse</strong> new syntax (custom statements /
dialect quirks)</li>
+<li><strong>Plan</strong> new semantics (expressions, types,
FROM-clause constructs)</li>
+<li><strong>Execute</strong> new operators when rewrites are
not sufficient</li>
+</ol>
+<p>This post explains where and how to hook into each stage. For
complete, working code, see the linked
<code>datafusion-examples</code>.</p>
+<hr/>
+<h2 id="parse-plan-execute">Parse → Plan → Execute<a
class="headerlink" href="#parse-plan-execute" title="Permanent
link">¶</a></h2>
+<p>DataFusion turns SQL into executable work in stages:</p>
+<ol>
+<li><strong>Parse</strong>: SQL text is parsed into an AST
(<a
href="https://docs.rs/sqlparser/latest/sqlparser/ast/enum.Statement.html">Statement</a>
from <a
href="https://github.com/sqlparser-rs/sqlparser-rs">sqlparser-rs</a>)</li>
+<li><strong>Logical planning</strong>: <a
href="https://docs.rs/datafusion/latest/datafusion/sql/planner/struct.SqlToRel.html">SqlToRel</a>
converts the AST into a <a
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html">LogicalPlan</a></li>
+<li><strong>Physical planning</strong>: The <a
href="https://docs.rs/datafusion/latest/datafusion/physical_planner/trait.PhysicalPlanner.html">PhysicalPlanner</a>
turns the logical plan into an <a
href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html">ExecutionPlan</a></li>
+</ol>
+<p>Each stage has extension points.</p>
+<figure>
+<img alt="DataFusion SQL processing pipeline: SQL String flows through
Parser to AST, then SqlToRel (with Extension Planners) to LogicalPlan, then
PhysicalPlanner to ExecutionPlan" class="img-responsive"
src="/blog/images/extending-sql/architecture.svg" width="100%"/>
+<figcaption>
+<b>Figure 1:</b> SQL flows through three stages: parsing, logical
planning (via <code>SqlToRel</code>, where the Extension Planners
hook in), and physical planning. Each stage has extension points: wrap the
parser, implement planner traits, or add physical operators.
+ </figcaption>
+</figure>
+<p>To choose the right extension point, look at where the query
fails.</p>
+<table class="table">
+<thead>
+<tr>
+<th>What fails?</th>
+<th>What it looks like</th>
+<th>Where to hook in</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Parsing</td>
+<td><code>Expected: TABLE, found: CATALOG</code></td>
+<td>configure dialect or wrap
<code>DFParser</code></td>
+</tr>
+<tr>
+<td>Planning</td>
+<td><code>This feature is not implemented:
DATETIME</code></td>
+<td><code>ExprPlanner</code>,
<code>TypePlanner</code>,
<code>RelationPlanner</code></td>
+</tr>
+<tr>
+<td>Execution</td>
+<td><code>No physical plan for TableSample</code></td>
+<td><code>ExtensionPlanner</code> (+ physical
operator)</td>
+</tr>
+</tbody>
+</table>
+<p>We will follow that pipeline order.</p>
+<hr/>
+<h2 id="1-extending-parsing-wrapping-dfparser-for-custom-statements">1)
Extending parsing: wrapping <code>DFParser</code> for custom
statements<a class="headerlink"
href="#1-extending-parsing-wrapping-dfparser-for-custom-statements"
title="Permanent link">¶</a></h2>
+<p>The <code>CREATE EXTERNAL CATALOG</code> syntax from the
introduction fails at the parser because DataFusion only recognizes
<code>CREATE EXTERNAL TABLE</code>. To support new statement-level
syntax, you can <strong>wrap
<code>DFParser</code></strong>. Peek ahead <strong>in
the token stream</strong> to detect your custom syntax, handle it
yourself, and delegate everything else to DataFusion.</p>
+<p>The <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs"><code>custom_sql_parser.rs</code></a>
example demonstrates this pattern:</p>
+<pre><code class="language-rust">struct
CustomParser&lt;'a&gt; { df_parser: DFParser&lt;'a&gt; }
+
+impl&lt;'a&gt; CustomParser&lt;'a&gt; {
+ pub fn parse_statement(&amp;mut self) -&gt;
Result&lt;CustomStatement&gt; {
+ // Peek tokens to detect CREATE EXTERNAL CATALOG
+ if self.is_create_external_catalog() {
+ return self.parse_create_external_catalog();
+ }
+ // Delegate everything else to DataFusion
+ Ok(CustomStatement::DFStatement(Box::new(
+ self.df_parser.parse_statement()?,
+ )))
+ }
+}
+</code></pre>
+<p>You do not need to implement a full SQL parser. Reuse DataFusion's
tokenizer and parser helpers to consume tokens, parse identifiers, and handle
options—the example shows how.</p>
+<p>Once parsed, the simplest integration is to treat custom statements
as <strong>application commands</strong>:</p>
+<pre><code class="language-rust">match parser.parse_statement()? {
+ CustomStatement::DFStatement(stmt) =&gt;
ctx.sql(&amp;stmt.to_string()).await?,
+ CustomStatement::CreateExternalCatalog(stmt) =&gt; {
+ handle_create_external_catalog(&amp;ctx, stmt).await?
+ }
+}
+</code></pre>
+<p>This keeps the extension logic in your embedding application. The
example includes a complete
<code>handle_create_external_catalog</code> that registers tables
from a location into a catalog, making them queryable immediately.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs"><code>custom_sql_parser.rs</code></a></p>
+<hr/>
+<h2 id="2-extending-expression-semantics-exprplanner">2) Extending
expression semantics: <code>ExprPlanner</code><a
class="headerlink" href="#2-extending-expression-semantics-exprplanner"
title="Permanent link">¶</a></h2>
+<p>Once SQL <em>parses</em>, the next failure is often that
DataFusion does not know what a particular expression means.</p>
+<p>This is where dialect differences show up in day-to-day queries:
operators like Postgres JSON arrows, vendor-specific functions, or small
syntactic sugar that users expect to keep working when you switch
engines.</p>
+<p><code>ExprPlanner</code> lets you define how specific SQL
expressions become DataFusion <code>Expr</code>. Common
examples:</p>
+<ul>
+<li>Non-standard operators (JSON / geometry / regex operators)</li>
+<li>Custom function syntaxes</li>
+<li>Special identifier behavior</li>
+</ul>
+<h3 id="example-postgres-json-operators-">Example: Postgres JSON
operators (<code>-&gt;</code>,
<code>-&gt;&gt;</code>)<a class="headerlink"
href="#example-postgres-json-operators-" title="Permanent
link">¶</a></h3>
+<p>The Postgres <code>-&gt;</code> operator is a good
illustration because it is widely used and parses only under the PostgreSQL
dialect.</p>
+<p>Configure the dialect:</p>
+<pre><code class="language-rust">let config = SessionConfig::new()
+ .set_str("datafusion.sql_parser.dialect", "postgres");
+let ctx = SessionContext::new_with_config(config);
+</code></pre>
+<p>Then implement <code>ExprPlanner</code> to map the parsed
operator (<code>BinaryOperator::Arrow</code>) to DataFusion
semantics:</p>
+<pre><code class="language-rust">fn plan_binary_op(&amp;self,
expr: RawBinaryExpr, _schema: &amp;DFSchema)
+ -&gt; Result&lt;PlannerResult&lt;RawBinaryExpr&gt;&gt; {
+ match expr.op {
+ BinaryOperator::Arrow =&gt; Ok(Planned(/* your Expr */)),
+ _ =&gt; Ok(Original(expr)),
+ }
+}
+</code></pre>
+<p>Return <code>Planned(...)</code> when you handled the
expression; return <code>Original(...)</code> to pass it to the
next planner.</p>
+<p>For a complete JSON implementation, see <a
href="https://github.com/datafusion-contrib/datafusion-functions-json">datafusion-functions-json</a>.
For a minimal end-to-end example in the DataFusion repo, see <a
href="https://github.com/apache/datafusion/blob/main/datafusion/core/tests/user_defined/expr_planner.rs"><code>expr_planner_tests</code></a>.</p>
+<hr/>
+<h2 id="3-extending-type-support-typeplanner">3) Extending type support:
<code>TypePlanner</code><a class="headerlink"
href="#3-extending-type-support-typeplanner" title="Permanent
link">¶</a></h2>
+<p>After expressions, types are often the next thing to break. Schemas
and DDL may reference types that DataFusion does not support out of the box,
like MySQL's <code>DATETIME</code>.</p>
+<p>Type planning tends to come up when interoperating with other
systems. You want to accept DDL or infer schemas from external catalogs without
forcing users to rewrite types.</p>
+<p><code>TypePlanner</code> maps SQL types to
Arrow/DataFusion types:</p>
+<pre><code class="language-rust">impl TypePlanner for
MyTypePlanner {
+ fn plan_type(&amp;self, sql_type: &amp;ast::DataType) -&gt;
Result&lt;Option&lt;DataType&gt;&gt; {
+ match sql_type {
+ ast::DataType::Datetime(Some(3)) =&gt;
Ok(Some(DataType::Timestamp(TimeUnit::Millisecond, None))),
+ _ =&gt; Ok(None), // let the default planner handle it
+ }
+ }
+}
+</code></pre>
+<p>It is installed when building session state:</p>
+<pre><code class="language-rust">let state =
SessionStateBuilder::new()
+ .with_default_features()
+ .with_type_planner(Arc::new(MyTypePlanner))
+ .build();
+</code></pre>
+<p>Once installed, if your <code>CREATE EXTERNAL
CATALOG</code> statement exposes tables with MySQL types, DataFusion can
interpret them correctly.</p>
+<hr/>
+<h2 id="4-extending-the-from-clause-relationplanner">4) Extending the
FROM clause: <code>RelationPlanner</code><a class="headerlink"
href="#4-extending-the-from-clause-relationplanner" title="Permanent
link">¶</a></h2>
+<p>Some extensions change what a <em>relation</em> means,
not just expressions or types. <code>RelationPlanner</code>
(available starting in DataFusion 52) intercepts FROM-clause constructs while
SQL is being converted into a <code>LogicalPlan</code>.</p>
+<p>Once you have <code>RelationPlanner</code>, there are two
main approaches to implementing your extension.</p>
+<h3 id="strategy-a-rewrite-to-existing-operators-pivot-unpivot">Strategy
A: rewrite to existing operators (PIVOT / UNPIVOT)<a class="headerlink"
href="#strategy-a-rewrite-to-existing-operators-pivot-unpivot" title="Permanent
link">¶</a></h3>
+<p>If you can translate your syntax into relational algebra that
DataFusion already supports, you can implement the feature with
<strong>no custom physical operator</strong>.</p>
+<p><code>PIVOT</code> rotates rows into columns, and
<code>UNPIVOT</code> does the reverse. Neither requires new
execution logic: <code>PIVOT</code> is just <code>GROUP
BY</code> with <code>CASE</code> expressions, and
<code>UNPIVOT</code> is a <code>UNION ALL</code> of
each column. The planner rewrites them accordingly:</p>
+<pre><code class="language-rust">match relation {
+ TableFactor::Pivot { .. } =&gt; /* rewrite to GROUP BY + CASE */,
+ TableFactor::Unpivot { .. } =&gt; /* rewrite to UNION ALL */,
+ other =&gt; Original(other),
+}
+</code></pre>
+<p>Because the output is a standard
<code>LogicalPlan</code>, DataFusion's usual optimization and
physical planning apply automatically.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/pivot_unpivot.rs"><code>pivot_unpivot.rs</code></a></p>
+<h3 id="strategy-b-custom-logical-physical-tablesample">Strategy B:
custom logical + physical (TABLESAMPLE)<a class="headerlink"
href="#strategy-b-custom-logical-physical-tablesample" title="Permanent
link">¶</a></h3>
+<p>Sometimes rewriting is not sufficient.
<code>TABLESAMPLE</code> returns a random subset of rows from a
table and is useful for approximations or debugging on large datasets. Because
it requires runtime randomness, you cannot express it as a rewrite to existing
operators. Instead, you need a custom logical node and physical operator to
execute it.</p>
+<p>The approach (shown in <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs"><code>table_sample.rs</code></a>):</p>
+<ol>
+<li><code>RelationPlanner</code> recognizes
<code>TABLESAMPLE</code> and produces a custom logical
node</li>
+<li>That node gets wrapped in
<code>LogicalPlan::Extension</code></li>
+<li><code>ExtensionPlanner</code> converts it to a custom
<code>ExecutionPlan</code></li>
+</ol>
+<p>In code:</p>
+<pre><code class="language-rust">// Logical planning: FROM t
TABLESAMPLE (...) -&gt; LogicalPlan::Extension(...)
+let plan = LogicalPlan::Extension(Extension { node:
Arc::new(TableSamplePlanNode { /* ... */ }) });
+</code></pre>
+<pre><code class="language-rust">// Physical planning:
TableSamplePlanNode -&gt; SampleExec
+if let Some(sample_node) =
node.as_any().downcast_ref::&lt;TableSamplePlanNode&gt;() {
+ return Ok(Some(Arc::new(SampleExec::try_new(input, /* bounds, seed */)?)));
+}
+</code></pre>
+<p>This is the general pattern for custom FROM constructs that need
runtime behavior.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs"><code>table_sample.rs</code></a></p>
+<h3 id="background-origin-of-the-api">Background: Origin of the API<a
class="headerlink" href="#background-origin-of-the-api" title="Permanent
link">¶</a></h3>
+<p><code>RelationPlanner</code> originally came out of
trying to build <code>MATCH_RECOGNIZE</code> support in DataFusion
as a Datadog hackathon project. <code>MATCH_RECOGNIZE</code> is a
complex SQL feature for detecting patterns in sequences of rows, and it made
sense to prototype as an extension first. At the time, DataFusion had no
extension point at the right stage of SQL-to-rel planning to intercept and
reinterpret relations.</p>
+<p><a href="https://github.com/theirix">@theirix</a>'s
<code>TABLESAMPLE</code> work (<a
href="https://github.com/apache/datafusion/issues/13563">#13563</a>,
<a
href="https://github.com/apache/datafusion/pull/17633">#17633</a>)
demonstrated exactly where the gap was: their extension only worked when
<code>TABLESAMPLE</code> appeared at the query root and any
<code>TABLESAMPLE</code> inside a CTE or JOIN would e [...]
+<p>This is how Datadog approaches compatibility work: build features in
real systems first, then upstream the building blocks. A full
<code>MATCH_RECOGNIZE</code> extension is now in progress, built on
top of <code>RelationPlanner</code>, with the <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/match_recognize.rs"><code>match_recognize.rs</code></a>
example as a starting point.</p>
+<hr/>
+<h2 id="summary-the-extensibility-workflow">Summary: The Extensibility
Workflow<a class="headerlink" href="#summary-the-extensibility-workflow"
title="Permanent link">¶</a></h2>
+<p>DataFusion's SQL extensibility follows its processing pipeline. When
building your own dialect extension, work incrementally:</p>
+<ol>
+<li><strong>Parse</strong>: Use a parser wrapper to
intercept custom syntax in the token stream. Produce either a standard
<code>Statement</code> or your own application-specific
command.</li>
+<li><strong>Plan</strong>: Implement the planning traits
(<code>ExprPlanner</code>, <code>TypePlanner</code>,
<code>RelationPlanner</code>) to give your syntax
meaning.</li>
+<li><strong>Execute</strong>: Prefer rewrites to existing
operators (like <code>PIVOT</code> to
<code>CASE</code>). Only add custom physical operators via
<code>ExtensionPlanner</code> when you need specific runtime
behavior like randomness or specialized I/O.</li>
+</ol>
+<hr/>
+<h2 id="debugging-tips">Debugging tips<a class="headerlink"
href="#debugging-tips" title="Permanent link">¶</a></h2>
+<h3 id="print-the-logical-plan">Print the logical plan<a
class="headerlink" href="#print-the-logical-plan" title="Permanent
link">¶</a></h3>
+<pre><code class="language-rust">let df = ctx.sql("SELECT * FROM t
TABLESAMPLE (10 PERCENT)").await?;
+println!("{}", df.logical_plan().display_indent());
+</code></pre>
+<h3 id="use-explain">Use <a
href="https://datafusion.apache.org/user-guide/sql/explain.html"><code>EXPLAIN</code></a><a
class="headerlink" href="#use-explain" title="Permanent
link">¶</a></h3>
+<pre><code class="language-sql">EXPLAIN SELECT * FROM t
TABLESAMPLE (10 PERCENT);
+</code></pre>
+<p>If your extension is not being invoked, it is usually visible in the
logical plan first.</p>
+<hr/>
+<h2 id="when-hooks-arent-enough">When hooks aren't enough<a
class="headerlink" href="#when-hooks-arent-enough" title="Permanent
link">¶</a></h2>
+<p>While these extension points cover the majority of dialect needs,
some deep architectural areas still have limited or no hooks. If you are
working in these parts of the SQL surface area, you may need to contribute
upstream:</p>
+<ul>
+<li>Statement-level planning: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/statement.rs"><code>statement.rs</code></a></li>
+<li>JOIN planning: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/relation/join.rs"><code>relation/join.rs</code></a></li>
+<li>TOP / FETCH clauses: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/select.rs"><code>select.rs</code></a>,
<a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/query.rs"><code>query.rs</code></a></li>
+</ul>
+<hr/>
+<h2 id="ideas-to-try">Ideas to try<a class="headerlink"
href="#ideas-to-try" title="Permanent link">¶</a></h2>
+<p>If you want to experiment with these extension points, here are a few
suggestions:</p>
+<ul>
+<li>Geometry operators (for example <code>@&gt;</code>,
<code>&lt;@</code>) via
<code>ExprPlanner</code></li>
+<li>Oracle <code>NUMBER</code> or SQL Server
<code>MONEY</code> via
<code>TypePlanner</code></li>
+<li><code>JSON_TABLE</code> or semantic-layer style
relations via <code>RelationPlanner</code></li>
+</ul>
+<hr/>
+<h2 id="see-also">See also<a class="headerlink" href="#see-also"
title="Permanent link">¶</a></h2>
+<ul>
+<li>Extending SQL Guide: <a
href="https://datafusion.apache.org/library-user-guide/extending-sql.html">Extending
SQL Guide</a></li>
+<li>Parser wrapping example: <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs"><code>custom_sql_parser.rs</code></a></li>
+<li>
+<p>RelationPlanner examples:</p>
+</li>
+<li>
+<p><code>PIVOT</code> / <code>UNPIVOT</code>:
<a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/pivot_unpivot.rs"><code>pivot_unpivot.rs</code></a></p>
+</li>
+<li>
+<p><code>TABLESAMPLE</code>: <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs"><code>table_sample.rs</code></a></p>
+</li>
+<li>
+<p>ExprPlanner test examples: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/core/tests/user_defined/expr_planner.rs"><code>expr_planner_tests</code></a></p>
+</li>
+</ul>
+<h2 id="acknowledgements">Acknowledgements<a class="headerlink"
href="#acknowledgements" title="Permanent link">¶</a></h2>
+<p>Thank you to <a
href="https://github.com/jayzhan211">@jayzhan211</a> for designing and
implementing the original <code>ExprPlanner</code> API (<a
href="https://github.com/apache/datafusion/pull/11180">#11180</a>), to
<a href="https://github.com/goldmedal">@goldmedal</a> for adding
<code>TypePlanner</code> (<a
href="https://github.com/apache/datafusion/pull/13294">#13294</a>),
and to <a href="https://githu [...]
+<h2 id="get-involved">Get Involved<a class="headerlink"
href="#get-involved" title="Permanent link">¶</a></h2>
+<ul>
+<li><strong>Try it out</strong>: Implement one of the
extension points and share your experience</li>
+<li><strong>File issues or join the conversation</strong>:
<a href="https://github.com/apache/datafusion/">GitHub</a> for bugs
and feature requests, <a
href="https://datafusion.apache.org/contributor-guide/communication.html">Slack
or Discord</a> for discussion</li>
+</ul>
+<!-- Reference links --></content><category
term="blog"></category></entry><entry><title>Optimizing Repartitions in
DataFusion: How I Went From Database Noob to Core Contribution</title><link
href="https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions";
rel="alternate"></link><published>2025-12-15T00:00:00+00:00</published><updated>2025-12-15T00:00:00+00:00</updated><author><name>Gene
Bordegaray</name></author><id>tag:datafusion.apache.org,2025-12-15:/blog/202
[...]
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
diff --git a/output/feeds/geoffrey-claude-datadog.atom.xml
b/output/feeds/geoffrey-claude-datadog.atom.xml
new file mode 100644
index 0000000..4518349
--- /dev/null
+++ b/output/feeds/geoffrey-claude-datadog.atom.xml
@@ -0,0 +1,286 @@
+<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom";><title>Apache DataFusion Blog -
Geoffrey Claude (Datadog)</title><link
href="https://datafusion.apache.org/blog/"; rel="alternate"></link><link
href="https://datafusion.apache.org/blog/feeds/geoffrey-claude-datadog.atom.xml";
rel="self"></link><id>https://datafusion.apache.org/blog/</id><updated>2026-01-12T00:00:00+00:00</updated><subtitle></subtitle><entry><title>Extending
SQL in DataFusion: from ->> to TABLESAMPLE</title><link href="https:/
[...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a
href="https://datafusion.apache.org/">DataFusion</a> in your product,
your users will eventually run SQL that DataFusion does not recognize. Not
because the query is unreasonable, but because SQL in practice includes many
dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to
attach an …</p></summary><content type="html"><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a
href="https://datafusion.apache.org/">DataFusion</a> in your product,
your users will eventually run SQL that DataFusion does not recognize. Not
because the query is unreasonable, but because SQL in practice includes many
dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to
attach an external catalog to query them. DataFusion has <code>CREATE
EXTERNAL TABLE</code> for individual tables, but no built-in equivalent
for catalogs. DuckDB has <code>ATTACH</code>, SQLite has its own
variant, and maybe you really want something even more flexible:</p>
+<pre><code class="language-sql">CREATE EXTERNAL CATALOG my_lake
+STORED AS iceberg
+LOCATION 's3://my-bucket/warehouse'
+OPTIONS ('region' 'eu-west-1');
+</code></pre>
+<p>This syntax does not exist in DataFusion today, but you can add
it.</p>
+<hr/>
+<p>At the same time, many dialect gaps are smaller and show up in
everyday queries:</p>
+<pre><code class="language-sql">-- Postgres-style JSON operators
+SELECT payload-&gt;'user'-&gt;&gt;'id' FROM logs;
+
+-- MySQL-specific types
+SELECT DATETIME '2001-01-01 18:00:00';
+
+-- Statistical sampling
+SELECT * FROM sensor_data TABLESAMPLE BERNOULLI(10 PERCENT);
+</code></pre>
+<p>You can implement all of these <em>without forking</em>
DataFusion:</p>
+<ol>
+<li><strong>Parse</strong> new syntax (custom statements /
dialect quirks)</li>
+<li><strong>Plan</strong> new semantics (expressions, types,
FROM-clause constructs)</li>
+<li><strong>Execute</strong> new operators when rewrites are
not sufficient</li>
+</ol>
+<p>This post explains where and how to hook into each stage. For
complete, working code, see the linked
<code>datafusion-examples</code>.</p>
+<hr/>
+<h2 id="parse-plan-execute">Parse → Plan → Execute<a
class="headerlink" href="#parse-plan-execute" title="Permanent
link">¶</a></h2>
+<p>DataFusion turns SQL into executable work in stages:</p>
+<ol>
+<li><strong>Parse</strong>: SQL text is parsed into an AST
(<a
href="https://docs.rs/sqlparser/latest/sqlparser/ast/enum.Statement.html">Statement</a>
from <a
href="https://github.com/sqlparser-rs/sqlparser-rs">sqlparser-rs</a>)</li>
+<li><strong>Logical planning</strong>: <a
href="https://docs.rs/datafusion/latest/datafusion/sql/planner/struct.SqlToRel.html">SqlToRel</a>
converts the AST into a <a
href="https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html">LogicalPlan</a></li>
+<li><strong>Physical planning</strong>: The <a
href="https://docs.rs/datafusion/latest/datafusion/physical_planner/trait.PhysicalPlanner.html">PhysicalPlanner</a>
turns the logical plan into an <a
href="https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html">ExecutionPlan</a></li>
+</ol>
+<p>Each stage has extension points.</p>
+<figure>
+<img alt="DataFusion SQL processing pipeline: SQL String flows through
Parser to AST, then SqlToRel (with Extension Planners) to LogicalPlan, then
PhysicalPlanner to ExecutionPlan" class="img-responsive"
src="/blog/images/extending-sql/architecture.svg" width="100%"/>
+<figcaption>
+<b>Figure 1:</b> SQL flows through three stages: parsing, logical
planning (via <code>SqlToRel</code>, where the Extension Planners
hook in), and physical planning. Each stage has extension points: wrap the
parser, implement planner traits, or add physical operators.
+ </figcaption>
+</figure>
+<p>To choose the right extension point, look at where the query
fails.</p>
+<table class="table">
+<thead>
+<tr>
+<th>What fails?</th>
+<th>What it looks like</th>
+<th>Where to hook in</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Parsing</td>
+<td><code>Expected: TABLE, found: CATALOG</code></td>
+<td>configure dialect or wrap
<code>DFParser</code></td>
+</tr>
+<tr>
+<td>Planning</td>
+<td><code>This feature is not implemented:
DATETIME</code></td>
+<td><code>ExprPlanner</code>,
<code>TypePlanner</code>,
<code>RelationPlanner</code></td>
+</tr>
+<tr>
+<td>Execution</td>
+<td><code>No physical plan for TableSample</code></td>
+<td><code>ExtensionPlanner</code> (+ physical
operator)</td>
+</tr>
+</tbody>
+</table>
+<p>We will follow that pipeline order.</p>
+<hr/>
+<h2 id="1-extending-parsing-wrapping-dfparser-for-custom-statements">1)
Extending parsing: wrapping <code>DFParser</code> for custom
statements<a class="headerlink"
href="#1-extending-parsing-wrapping-dfparser-for-custom-statements"
title="Permanent link">¶</a></h2>
+<p>The <code>CREATE EXTERNAL CATALOG</code> syntax from the
introduction fails at the parser because DataFusion only recognizes
<code>CREATE EXTERNAL TABLE</code>. To support new statement-level
syntax, you can <strong>wrap
<code>DFParser</code></strong>. Peek ahead <strong>in
the token stream</strong> to detect your custom syntax, handle it
yourself, and delegate everything else to DataFusion.</p>
+<p>The <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs"><code>custom_sql_parser.rs</code></a>
example demonstrates this pattern:</p>
+<pre><code class="language-rust">struct
CustomParser&lt;'a&gt; { df_parser: DFParser&lt;'a&gt; }
+
+impl&lt;'a&gt; CustomParser&lt;'a&gt; {
+ pub fn parse_statement(&amp;mut self) -&gt;
Result&lt;CustomStatement&gt; {
+ // Peek tokens to detect CREATE EXTERNAL CATALOG
+ if self.is_create_external_catalog() {
+ return self.parse_create_external_catalog();
+ }
+ // Delegate everything else to DataFusion
+ Ok(CustomStatement::DFStatement(Box::new(
+ self.df_parser.parse_statement()?,
+ )))
+ }
+}
+</code></pre>
+<p>You do not need to implement a full SQL parser. Reuse DataFusion's
tokenizer and parser helpers to consume tokens, parse identifiers, and handle
options—the example shows how.</p>
+<p>Once parsed, the simplest integration is to treat custom statements
as <strong>application commands</strong>:</p>
+<pre><code class="language-rust">match parser.parse_statement()? {
+ CustomStatement::DFStatement(stmt) =&gt;
ctx.sql(&amp;stmt.to_string()).await?,
+ CustomStatement::CreateExternalCatalog(stmt) =&gt; {
+ handle_create_external_catalog(&amp;ctx, stmt).await?
+ }
+}
+</code></pre>
+<p>This keeps the extension logic in your embedding application. The
example includes a complete
<code>handle_create_external_catalog</code> that registers tables
from a location into a catalog, making them queryable immediately.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs"><code>custom_sql_parser.rs</code></a></p>
+<hr/>
+<h2 id="2-extending-expression-semantics-exprplanner">2) Extending
expression semantics: <code>ExprPlanner</code><a
class="headerlink" href="#2-extending-expression-semantics-exprplanner"
title="Permanent link">¶</a></h2>
+<p>Once SQL <em>parses</em>, the next failure is often that
DataFusion does not know what a particular expression means.</p>
+<p>This is where dialect differences show up in day-to-day queries:
operators like Postgres JSON arrows, vendor-specific functions, or small
syntactic sugar that users expect to keep working when you switch
engines.</p>
+<p><code>ExprPlanner</code> lets you define how specific SQL
expressions become DataFusion <code>Expr</code>. Common
examples:</p>
+<ul>
+<li>Non-standard operators (JSON / geometry / regex operators)</li>
+<li>Custom function syntaxes</li>
+<li>Special identifier behavior</li>
+</ul>
+<h3 id="example-postgres-json-operators-">Example: Postgres JSON
operators (<code>-&gt;</code>,
<code>-&gt;&gt;</code>)<a class="headerlink"
href="#example-postgres-json-operators-" title="Permanent
link">¶</a></h3>
+<p>The Postgres <code>-&gt;</code> operator is a good
illustration because it is widely used and parses only under the PostgreSQL
dialect.</p>
+<p>Configure the dialect:</p>
+<pre><code class="language-rust">let config = SessionConfig::new()
+ .set_str("datafusion.sql_parser.dialect", "postgres");
+let ctx = SessionContext::new_with_config(config);
+</code></pre>
+<p>Then implement <code>ExprPlanner</code> to map the parsed
operator (<code>BinaryOperator::Arrow</code>) to DataFusion
semantics:</p>
+<pre><code class="language-rust">fn plan_binary_op(&amp;self,
expr: RawBinaryExpr, _schema: &amp;DFSchema)
+ -&gt; Result&lt;PlannerResult&lt;RawBinaryExpr&gt;&gt; {
+ match expr.op {
+ BinaryOperator::Arrow =&gt; Ok(Planned(/* your Expr */)),
+ _ =&gt; Ok(Original(expr)),
+ }
+}
+</code></pre>
+<p>Return <code>Planned(...)</code> when you handled the
expression; return <code>Original(...)</code> to pass it to the
next planner.</p>
+<p>For a complete JSON implementation, see <a
href="https://github.com/datafusion-contrib/datafusion-functions-json">datafusion-functions-json</a>.
For a minimal end-to-end example in the DataFusion repo, see <a
href="https://github.com/apache/datafusion/blob/main/datafusion/core/tests/user_defined/expr_planner.rs"><code>expr_planner_tests</code></a>.</p>
+<hr/>
+<h2 id="3-extending-type-support-typeplanner">3) Extending type support:
<code>TypePlanner</code><a class="headerlink"
href="#3-extending-type-support-typeplanner" title="Permanent
link">¶</a></h2>
+<p>After expressions, types are often the next thing to break. Schemas
and DDL may reference types that DataFusion does not support out of the box,
like MySQL's <code>DATETIME</code>.</p>
+<p>Type planning tends to come up when interoperating with other
systems. You want to accept DDL or infer schemas from external catalogs without
forcing users to rewrite types.</p>
+<p><code>TypePlanner</code> maps SQL types to
Arrow/DataFusion types:</p>
+<pre><code class="language-rust">impl TypePlanner for
MyTypePlanner {
+ fn plan_type(&amp;self, sql_type: &amp;ast::DataType) -&gt;
Result&lt;Option&lt;DataType&gt;&gt; {
+ match sql_type {
+ ast::DataType::Datetime(Some(3)) =&gt;
Ok(Some(DataType::Timestamp(TimeUnit::Millisecond, None))),
+ _ =&gt; Ok(None), // let the default planner handle it
+ }
+ }
+}
+</code></pre>
+<p>It is installed when building session state:</p>
+<pre><code class="language-rust">let state =
SessionStateBuilder::new()
+ .with_default_features()
+ .with_type_planner(Arc::new(MyTypePlanner))
+ .build();
+</code></pre>
+<p>Once installed, if your <code>CREATE EXTERNAL
CATALOG</code> statement exposes tables with MySQL types, DataFusion can
interpret them correctly.</p>
+<hr/>
+<h2 id="4-extending-the-from-clause-relationplanner">4) Extending the
FROM clause: <code>RelationPlanner</code><a class="headerlink"
href="#4-extending-the-from-clause-relationplanner" title="Permanent
link">¶</a></h2>
+<p>Some extensions change what a <em>relation</em> means,
not just expressions or types. <code>RelationPlanner</code>
(available starting in DataFusion 52) intercepts FROM-clause constructs while
SQL is being converted into a <code>LogicalPlan</code>.</p>
+<p>Once you have <code>RelationPlanner</code>, there are two
main approaches to implementing your extension.</p>
+<h3 id="strategy-a-rewrite-to-existing-operators-pivot-unpivot">Strategy
A: rewrite to existing operators (PIVOT / UNPIVOT)<a class="headerlink"
href="#strategy-a-rewrite-to-existing-operators-pivot-unpivot" title="Permanent
link">¶</a></h3>
+<p>If you can translate your syntax into relational algebra that
DataFusion already supports, you can implement the feature with
<strong>no custom physical operator</strong>.</p>
+<p><code>PIVOT</code> rotates rows into columns, and
<code>UNPIVOT</code> does the reverse. Neither requires new
execution logic: <code>PIVOT</code> is just <code>GROUP
BY</code> with <code>CASE</code> expressions, and
<code>UNPIVOT</code> is a <code>UNION ALL</code> of
each column. The planner rewrites them accordingly:</p>
+<pre><code class="language-rust">match relation {
+ TableFactor::Pivot { .. } =&gt; /* rewrite to GROUP BY + CASE */,
+ TableFactor::Unpivot { .. } =&gt; /* rewrite to UNION ALL */,
+ other =&gt; Original(other),
+}
+</code></pre>
+<p>Because the output is a standard
<code>LogicalPlan</code>, DataFusion's usual optimization and
physical planning apply automatically.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/pivot_unpivot.rs"><code>pivot_unpivot.rs</code></a></p>
+<h3 id="strategy-b-custom-logical-physical-tablesample">Strategy B:
custom logical + physical (TABLESAMPLE)<a class="headerlink"
href="#strategy-b-custom-logical-physical-tablesample" title="Permanent
link">¶</a></h3>
+<p>Sometimes rewriting is not sufficient.
<code>TABLESAMPLE</code> returns a random subset of rows from a
table and is useful for approximations or debugging on large datasets. Because
it requires runtime randomness, you cannot express it as a rewrite to existing
operators. Instead, you need a custom logical node and physical operator to
execute it.</p>
+<p>The approach (shown in <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs"><code>table_sample.rs</code></a>):</p>
+<ol>
+<li><code>RelationPlanner</code> recognizes
<code>TABLESAMPLE</code> and produces a custom logical
node</li>
+<li>That node gets wrapped in
<code>LogicalPlan::Extension</code></li>
+<li><code>ExtensionPlanner</code> converts it to a custom
<code>ExecutionPlan</code></li>
+</ol>
+<p>In code:</p>
+<pre><code class="language-rust">// Logical planning: FROM t
TABLESAMPLE (...) -&gt; LogicalPlan::Extension(...)
+let plan = LogicalPlan::Extension(Extension { node:
Arc::new(TableSamplePlanNode { /* ... */ }) });
+</code></pre>
+<pre><code class="language-rust">// Physical planning:
TableSamplePlanNode -&gt; SampleExec
+if let Some(sample_node) =
node.as_any().downcast_ref::&lt;TableSamplePlanNode&gt;() {
+ return Ok(Some(Arc::new(SampleExec::try_new(input, /* bounds, seed */)?)));
+}
+</code></pre>
+<p>This is the general pattern for custom FROM constructs that need
runtime behavior.</p>
+<p><strong>Full working example:</strong> <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs"><code>table_sample.rs</code></a></p>
+<h3 id="background-origin-of-the-api">Background: Origin of the API<a
class="headerlink" href="#background-origin-of-the-api" title="Permanent
link">¶</a></h3>
+<p><code>RelationPlanner</code> originally came out of
trying to build <code>MATCH_RECOGNIZE</code> support in DataFusion
as a Datadog hackathon project. <code>MATCH_RECOGNIZE</code> is a
complex SQL feature for detecting patterns in sequences of rows, and it made
sense to prototype as an extension first. At the time, DataFusion had no
extension point at the right stage of SQL-to-rel planning to intercept and
reinterpret relations.</p>
+<p><a href="https://github.com/theirix">@theirix</a>'s
<code>TABLESAMPLE</code> work (<a
href="https://github.com/apache/datafusion/issues/13563">#13563</a>,
<a
href="https://github.com/apache/datafusion/pull/17633">#17633</a>)
demonstrated exactly where the gap was: their extension only worked when
<code>TABLESAMPLE</code> appeared at the query root and any
<code>TABLESAMPLE</code> inside a CTE or JOIN would e [...]
+<p>This is how Datadog approaches compatibility work: build features in
real systems first, then upstream the building blocks. A full
<code>MATCH_RECOGNIZE</code> extension is now in progress, built on
top of <code>RelationPlanner</code>, with the <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/match_recognize.rs"><code>match_recognize.rs</code></a>
example as a starting point.</p>
+<hr/>
+<h2 id="summary-the-extensibility-workflow">Summary: The Extensibility
Workflow<a class="headerlink" href="#summary-the-extensibility-workflow"
title="Permanent link">¶</a></h2>
+<p>DataFusion's SQL extensibility follows its processing pipeline. When
building your own dialect extension, work incrementally:</p>
+<ol>
+<li><strong>Parse</strong>: Use a parser wrapper to
intercept custom syntax in the token stream. Produce either a standard
<code>Statement</code> or your own application-specific
command.</li>
+<li><strong>Plan</strong>: Implement the planning traits
(<code>ExprPlanner</code>, <code>TypePlanner</code>,
<code>RelationPlanner</code>) to give your syntax
meaning.</li>
+<li><strong>Execute</strong>: Prefer rewrites to existing
operators (like <code>PIVOT</code> to
<code>CASE</code>). Only add custom physical operators via
<code>ExtensionPlanner</code> when you need specific runtime
behavior like randomness or specialized I/O.</li>
+</ol>
+<hr/>
+<h2 id="debugging-tips">Debugging tips<a class="headerlink"
href="#debugging-tips" title="Permanent link">¶</a></h2>
+<h3 id="print-the-logical-plan">Print the logical plan<a
class="headerlink" href="#print-the-logical-plan" title="Permanent
link">¶</a></h3>
+<pre><code class="language-rust">let df = ctx.sql("SELECT * FROM t
TABLESAMPLE (10 PERCENT)").await?;
+println!("{}", df.logical_plan().display_indent());
+</code></pre>
+<h3 id="use-explain">Use <a
href="https://datafusion.apache.org/user-guide/sql/explain.html"><code>EXPLAIN</code></a><a
class="headerlink" href="#use-explain" title="Permanent
link">¶</a></h3>
+<pre><code class="language-sql">EXPLAIN SELECT * FROM t
TABLESAMPLE (10 PERCENT);
+</code></pre>
+<p>If your extension is not being invoked, it is usually visible in the
logical plan first.</p>
+<hr/>
+<h2 id="when-hooks-arent-enough">When hooks aren't enough<a
class="headerlink" href="#when-hooks-arent-enough" title="Permanent
link">¶</a></h2>
+<p>While these extension points cover the majority of dialect needs,
some deep architectural areas still have limited or no hooks. If you are
working in these parts of the SQL surface area, you may need to contribute
upstream:</p>
+<ul>
+<li>Statement-level planning: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/statement.rs"><code>statement.rs</code></a></li>
+<li>JOIN planning: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/relation/join.rs"><code>relation/join.rs</code></a></li>
+<li>TOP / FETCH clauses: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/select.rs"><code>select.rs</code></a>,
<a
href="https://github.com/apache/datafusion/blob/main/datafusion/sql/src/query.rs"><code>query.rs</code></a></li>
+</ul>
+<hr/>
+<h2 id="ideas-to-try">Ideas to try<a class="headerlink"
href="#ideas-to-try" title="Permanent link">¶</a></h2>
+<p>If you want to experiment with these extension points, here are a few
suggestions:</p>
+<ul>
+<li>Geometry operators (for example <code>@&gt;</code>,
<code>&lt;@</code>) via
<code>ExprPlanner</code></li>
+<li>Oracle <code>NUMBER</code> or SQL Server
<code>MONEY</code> via
<code>TypePlanner</code></li>
+<li><code>JSON_TABLE</code> or semantic-layer style
relations via <code>RelationPlanner</code></li>
+</ul>
+<hr/>
+<h2 id="see-also">See also<a class="headerlink" href="#see-also"
title="Permanent link">¶</a></h2>
+<ul>
+<li>Extending SQL Guide: <a
href="https://datafusion.apache.org/library-user-guide/extending-sql.html">Extending
SQL Guide</a></li>
+<li>Parser wrapping example: <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs"><code>custom_sql_parser.rs</code></a></li>
+<li>
+<p>RelationPlanner examples:</p>
+</li>
+<li>
+<p><code>PIVOT</code> / <code>UNPIVOT</code>:
<a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/pivot_unpivot.rs"><code>pivot_unpivot.rs</code></a></p>
+</li>
+<li>
+<p><code>TABLESAMPLE</code>: <a
href="https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs"><code>table_sample.rs</code></a></p>
+</li>
+<li>
+<p>ExprPlanner test examples: <a
href="https://github.com/apache/datafusion/blob/main/datafusion/core/tests/user_defined/expr_planner.rs"><code>expr_planner_tests</code></a></p>
+</li>
+</ul>
+<h2 id="acknowledgements">Acknowledgements<a class="headerlink"
href="#acknowledgements" title="Permanent link">¶</a></h2>
+<p>Thank you to <a
href="https://github.com/jayzhan211">@jayzhan211</a> for designing and
implementing the original <code>ExprPlanner</code> API (<a
href="https://github.com/apache/datafusion/pull/11180">#11180</a>), to
<a href="https://github.com/goldmedal">@goldmedal</a> for adding
<code>TypePlanner</code> (<a
href="https://github.com/apache/datafusion/pull/13294">#13294</a>),
and to <a href="https://githu [...]
+<h2 id="get-involved">Get Involved<a class="headerlink"
href="#get-involved" title="Permanent link">¶</a></h2>
+<ul>
+<li><strong>Try it out</strong>: Implement one of the
extension points and share your experience</li>
+<li><strong>File issues or join the conversation</strong>:
<a href="https://github.com/apache/datafusion/">GitHub</a> for bugs
and feature requests, <a
href="https://datafusion.apache.org/contributor-guide/communication.html">Slack
or Discord</a> for discussion</li>
+</ul>
+<!-- Reference links --></content><category
term="blog"></category></entry></feed>
\ No newline at end of file
diff --git a/output/feeds/geoffrey-claude-datadog.rss.xml
b/output/feeds/geoffrey-claude-datadog.rss.xml
new file mode 100644
index 0000000..2ec017c
--- /dev/null
+++ b/output/feeds/geoffrey-claude-datadog.rss.xml
@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="utf-8"?>
+<rss version="2.0"><channel><title>Apache DataFusion Blog - Geoffrey Claude
(Datadog)</title><link>https://datafusion.apache.org/blog/</link><description></description><lastBuildDate>Mon,
12 Jan 2026 00:00:00 +0000</lastBuildDate><item><title>Extending SQL in
DataFusion: from ->> to
TABLESAMPLE</title><link>https://datafusion.apache.org/blog/2026/01/12/extending-sql</link><description><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a
href="https://datafusion.apache.org/">DataFusion</a> in your product,
your users will eventually run SQL that DataFusion does not recognize. Not
because the query is unreasonable, but because SQL in practice includes many
dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to
attach an …</p></description><dc:creator
xmlns:dc="http://purl.org/dc/elements/1.1/";>Geoffrey Claude
(Datadog)</dc:creator><pubDate>Mon, 12 Jan 2026 00:00:00 +0000</pubDate><guid
isPermaLink="false">tag:datafusion.apache.org,2026-01-12:/blog/2026/01/12/extending-sql</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/output/images/extending-sql/architecture.svg
b/output/images/extending-sql/architecture.svg
new file mode 100644
index 0000000..27f5edc
--- /dev/null
+++ b/output/images/extending-sql/architecture.svg
@@ -0,0 +1,151 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1100 383"
style="max-width: 100%; height: auto;" role="img" aria-label="DataFusion SQL
processing pipeline: SQL String flows through Parser to AST, then SqlToRel
(with Extension Planners) to LogicalPlan, then PhysicalPlanner to
ExecutionPlan">
+ <defs>
+ <marker id="arrowhead" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#94a3b8"/>
+ </marker>
+ <marker id="arrowhead-rose" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#c0392b"/>
+ </marker>
+ <marker id="arrowhead-amber" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#e67e22"/>
+ </marker>
+ <marker id="arrowhead-lime" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#f1c40f"/>
+ </marker>
+ <marker id="arrowhead-emerald" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#27ae60"/>
+ </marker>
+ <marker id="arrowhead-sky" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#3498db"/>
+ </marker>
+ <marker id="arrowhead-violet" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#9b59b6"/>
+ </marker>
+ <filter id="shadow" x="-20%" y="-20%" width="140%" height="140%">
+ <feDropShadow dx="2" dy="2" stdDeviation="2" flood-opacity="0.12"/>
+ </filter>
+ </defs>
+
+ <!-- White background -->
+ <rect x="0" y="0" width="1100" height="383" fill="white"/>
+
+ <!-- Background groupings (drawn first, behind everything) -->
+ <rect x="153.75" y="35" width="113.49999999999999" height="200" rx="2"
fill="#e4beba" fill-opacity="0.5"/>
+ <rect x="286" y="55" width="98" height="180" rx="24" fill="#f1dcc9"
fill-opacity="0.5"/>
+ <rect x="402.75" y="35" width="182.5" height="250" rx="2" fill="#f3e9bf"
fill-opacity="0.5"/>
+ <rect x="604" y="55" width="142" height="180" rx="24" fill="#aaddc0"
fill-opacity="0.5"/>
+ <rect x="764.75" y="35" width="159.5" height="200" rx="2" fill="#d2e4f0"
fill-opacity="0.5"/>
+ <rect x="943" y="55" width="142" height="180" rx="24" fill="#e3dbe6"
fill-opacity="0.5"/>
+
+ <!-- Row 1: Main pipeline -->
+
+ <!-- SQL String (DATA) -->
+ <rect id="box-sql-string" x="20" y="60" width="110.00000000000001"
height="50" rx="24" fill="#e2e8f0" stroke="#64748b" stroke-width="2"
filter="url(#shadow)"/>
+ <text id="text-sql-string" x="75" y="90" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#334155">SQL String</text>
+
+ <line id="arrow-sql-to-parser" x1="131" y1="85" x2="156.75" y2="85"
stroke="#c0392b" stroke-width="2" marker-end="url(#arrowhead-rose)"/>
+
+ <!-- Parser (TRANSFORMER) - Slot 1 -->
+ <rect id="box-parser" x="158.75" y="40" width="103.49999999999999"
height="90" rx="2" fill="#e4beba" stroke="#c0392b" stroke-width="2"
filter="url(#shadow)"/>
+ <text id="text-parser-1" x="210.5" y="70" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#43140f">Parser</text>
+ <text id="text-parser-2" x="210.5" y="88" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13"
fill="#7e0d01">(sqlparser-rs)</text>
+ <text id="text-parser-3" x="210.5" y="115" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" font-weight="600"
fill="#7e0d01">Wrap DFParser</text>
+
+ <line id="arrow-parser-to-ast" x1="263.25" y1="85" x2="289" y2="85"
stroke="#e67e22" stroke-width="2" marker-end="url(#arrowhead-amber)"/>
+
+ <!-- DFParser wrapper box (TRANSFORMER) - Slot 1 -->
+ <rect id="box-dfparser" x="158.75" y="179" width="103.49999999999999"
height="50" rx="2" fill="#e4beba" stroke="#c0392b" stroke-width="2"
filter="url(#shadow)"/>
+ <text id="text-dfparser-1" x="210.5" y="202" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#43140f">DFParser</text>
+ <text id="text-dfparser-2" x="210.5" y="217" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13"
fill="#7e0d01">(wrapper)</text>
+
+ <!-- AST (DATA) - Slot 2 -->
+ <rect id="box-ast" x="291" y="60" width="88" height="50" rx="24"
fill="#f1dcc9" stroke="#e67e22" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-ast-1" x="335" y="82" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#64350b">AST</text>
+ <text id="text-ast-2" x="335" y="98" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13"
fill="#803c00">(Statement)</text>
+
+ <line id="arrow-ast-to-sqltorel" x1="380" y1="85" x2="405.75" y2="85"
stroke="#f1c40f" stroke-width="2" marker-end="url(#arrowhead-lime)"/>
+
+ <!-- Custom Statement box (DATA) - Slot 2 -->
+ <rect id="box-custom-statement" x="291" y="179" width="88" height="50"
rx="24" fill="#f1dcc9" stroke="#e67e22" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-custom-statement-1" x="335" y="199" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="14.5" font-weight="600"
fill="#64350b">Custom</text>
+ <text id="text-custom-statement-2" x="335" y="214" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#64350b">Statement</text>
+
+ <!-- Arrow from DFParser to Custom Statement -->
+ <line id="arrow-dfparser-to-custom-statement" x1="263.25" y1="204" x2="289"
y2="204" stroke="#e67e22" stroke-width="2" marker-end="url(#arrowhead-amber)"/>
+
+ <!-- SqlToRel (TRANSFORMER) - Slot 3 -->
+ <rect id="box-sqltorel" x="407.75" y="40" width="172.5" height="90" rx="2"
fill="#f3e9bf" stroke="#f1c40f" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-sqltorel-1" x="494" y="70" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#614f06">SqlToRel</text>
+ <text id="text-sqltorel-2" x="494" y="88" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" fill="#806600">(AST to
Logical)</text>
+ <text id="text-sqltorel-3" x="494" y="115" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" font-weight="600"
fill="#806600">Extension Planners</text>
+
+ <line id="arrow-sqltorel-to-logicalplan" x1="581.25" y1="85" x2="607"
y2="85" stroke="#27ae60" stroke-width="2" marker-end="url(#arrowhead-emerald)"/>
+
+ <!-- Extension Planners container (TRANSFORMER) - Slot 3 -->
+ <rect id="box-extension-planners" x="407.75" y="165" width="172.5"
height="115" rx="2" fill="#f3e9bf" stroke="#f1c40f" stroke-width="2"
filter="url(#shadow)"/>
+ <text id="text-extension-planners-1" x="494" y="183" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="14.5" font-weight="600"
fill="#614f06">Extension Planners</text>
+
+ <!-- RelationPlanner sub-box (top - produces UserDefinedLogicalNode) -->
+ <rect id="box-relation-planner" x="419.25" y="192" width="149.5" height="24"
rx="2" fill="#fbf8e9" stroke="#f1c40f" stroke-width="1"/>
+ <text id="text-relation-planner" x="494" y="208" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="13" font-weight="600"
fill="#614f06">RelationPlanner</text>
+
+ <!-- ExprPlanner sub-box -->
+ <rect id="box-expr-planner" x="419.25" y="220" width="149.5" height="24"
rx="2" fill="#fbf8e9" stroke="#f1c40f" stroke-width="1"/>
+ <text id="text-expr-planner" x="494" y="236" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="13" font-weight="600"
fill="#614f06">ExprPlanner</text>
+
+ <!-- TypePlanner sub-box -->
+ <rect id="box-type-planner" x="419.25" y="248" width="149.5" height="24"
rx="2" fill="#fbf8e9" stroke="#f1c40f" stroke-width="1"/>
+ <text id="text-type-planner" x="494" y="264" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="13" font-weight="600"
fill="#614f06">TypePlanner</text>
+
+ <!-- LogicalPlan (DATA) - Slot 4 -->
+ <rect id="box-logicalplan" x="609" y="60" width="132" height="50" rx="24"
fill="#aaddc0" stroke="#27ae60" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-logicalplan" x="675" y="90" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#0e3e22">LogicalPlan</text>
+
+ <line id="arrow-logicalplan-to-physicalplanner" x1="742" y1="85" x2="767.75"
y2="85" stroke="#3498db" stroke-width="2" marker-end="url(#arrowhead-sky)"/>
+
+ <!-- UserDefinedLogicalNode box (DATA) - Slot 4 -->
+ <rect id="box-userdefined-logical" x="609" y="179" width="132" height="50"
rx="24" fill="#aaddc0" stroke="#27ae60" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-userdefined-logical-1" x="675" y="199" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#0e3e22">UserDefined</text>
+ <text id="text-userdefined-logical-2" x="675" y="214" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#0e3e22">LogicalNode</text>
+
+ <!-- Arrow from RelationPlanner to UserDefinedLogicalNode -->
+ <line id="arrow-relationplanner-to-userdefined" x1="569.75" y1="204"
x2="607" y2="204" stroke="#27ae60" stroke-width="2"
marker-end="url(#arrowhead-emerald)"/>
+
+ <!-- PhysicalPlanner (TRANSFORMER) - Slot 5 -->
+ <rect id="box-physicalplanner" x="769.75" y="40" width="149.5" height="90"
rx="2" fill="#d2e4f0" stroke="#3498db" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-physicalplanner-1" x="844.5" y="70" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#124364">PhysicalPlanner</text>
+ <text id="text-physicalplanner-2" x="844.5" y="88" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" fill="#004c80">(Logical to
Physical)</text>
+ <text id="text-physicalplanner-3" x="844.5" y="115" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" font-weight="600"
fill="#004c80">ExtensionPlanner</text>
+
+ <line id="arrow-physicalplanner-to-executionplan" x1="920.25" y1="85"
x2="946" y2="85" stroke="#9b59b6" stroke-width="2"
marker-end="url(#arrowhead-violet)"/>
+
+ <!-- ExtensionPlanner box (TRANSFORMER) - Slot 5 -->
+ <rect id="box-extensionplanner" x="769.75" y="179" width="149.5" height="50"
rx="2" fill="#d2e4f0" stroke="#3498db" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-extensionplanner-1" x="844.5" y="202" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#124364">ExtensionPlanner</text>
+ <text id="text-extensionplanner-2" x="844.5" y="217" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" fill="#004c80">(trait)</text>
+
+ <!-- ExecutionPlan (DATA) - Slot 6 -->
+ <rect id="box-executionplan" x="948" y="60" width="132" height="50" rx="24"
fill="#e3dbe6" stroke="#9b59b6" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-executionplan" x="1014" y="90" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#452452">ExecutionPlan</text>
+
+ <!-- Custom ExecutionPlan box (DATA) - Slot 6 -->
+ <rect id="box-custom-execution" x="948" y="179" width="132" height="50"
rx="24" fill="#e3dbe6" stroke="#9b59b6" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-custom-execution-1" x="1014" y="199" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="14.5" font-weight="600"
fill="#452452">Custom</text>
+ <text id="text-custom-execution-2" x="1014" y="214" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#452452">ExecutionPlan</text>
+
+ <!-- Arrow from ExtensionPlanner to Custom ExecutionPlan -->
+ <line id="arrow-extensionplanner-to-custom-execution" x1="920.25" y1="204"
x2="946" y2="204" stroke="#9b59b6" stroke-width="2"
marker-end="url(#arrowhead-violet)"/>
+
+ <!-- Legend at bottom (aligned with transformers: Parser@155, SqlToRel@395,
PhysicalPlanner@735) -->
+ <rect x="20" y="313" width="1060" height="55" rx="6" fill="#f8fafc"
stroke="#e2e8f0" stroke-width="1"/>
+ <text x="40" y="338" font-family="system-ui, sans-serif" font-size="15.0"
font-weight="600" fill="#475569">Hook Points:</text>
+ <rect x="158.75" y="327" width="12" height="12" rx="2" fill="#e4beba"
stroke="#c0392b" stroke-width="1"/>
+ <text id="legend-text-1" x="176.75" y="337" font-family="system-ui,
sans-serif" font-size="14" fill="#475569">Parser (wrap DFParser)</text>
+ <rect x="407.75" y="327" width="12" height="12" rx="2" fill="#f3e9bf"
stroke="#f1c40f" stroke-width="1"/>
+ <text id="legend-text-2" x="425.75" y="337" font-family="system-ui,
sans-serif" font-size="14" fill="#475569">SQL Planning (operators, types,
relations)</text>
+ <rect x="769.75" y="327" width="12" height="12" rx="2" fill="#d2e4f0"
stroke="#3498db" stroke-width="1"/>
+ <text id="legend-text-3" x="787.75" y="337" font-family="system-ui,
sans-serif" font-size="14" fill="#475569">Physical Planning (custom
execution)</text>
+ <text id="legend-text-4" x="158.75" y="355" font-family="system-ui,
sans-serif" font-size="14" fill="#64748b">Parse errors: wrap parser</text>
+ <text id="legend-text-5" x="407.75" y="355" font-family="system-ui,
sans-serif" font-size="14" fill="#64748b">Plan errors: implement planners</text>
+ <text id="legend-text-6" x="769.75" y="355" font-family="system-ui,
sans-serif" font-size="14" fill="#64748b">Execute errors: implement
ExtensionPlanner</text>
+</svg>
\ No newline at end of file
diff --git a/output/index.html b/output/index.html
index 2dc3d7f..fe2ebf2 100644
--- a/output/index.html
+++ b/output/index.html
@@ -45,6 +45,45 @@
<p><i>Here you can find the latest updates from DataFusion and
related projects.</i></p>
+ <!-- Post -->
+ <div class="row">
+ <div class="callout">
+ <article class="post">
+ <header>
+ <div class="title">
+ <h1><a href="/blog/2026/01/12/extending-sql">Extending
SQL in DataFusion: from ->> to TABLESAMPLE</a></h1>
+ <p>Posted on: Mon 12 January 2026 by Geoffrey Claude
(Datadog)</p>
+ <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>If you embed <a href="https://datafusion.apache.org/";>DataFusion</a> in
your product, your users will eventually run SQL that DataFusion does not
recognize. Not because the query is unreasonable, but because SQL in practice
includes many dialects and system-specific statements.</p>
+<p>Suppose you store data as Parquet files on S3 and want users to attach an
…</p></p>
+ <footer>
+ <ul class="actions">
+ <div style="text-align: right"><a
href="/blog/2026/01/12/extending-sql" class="button medium">Continue
Reading</a></div>
+ </ul>
+ <ul class="stats">
+ </ul>
+ </footer>
+ </article>
+ </div>
+ </div>
<!-- Post -->
<div class="row">
<div class="callout">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
