This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/main by this push:
new 6e852ae Add blog post on extending SQL in DataFusion (#130)
6e852ae is described below
commit 6e852ae21756485ffdea025bed4a6a0bf8a13c7c
Author: Geoffrey Claude <[email protected]>
AuthorDate: Mon Jan 12 22:20:24 2026 +0100
Add blog post on extending SQL in DataFusion (#130)
* Add Extending SQL blog post
* fix: apply PR review suggestions
* Update publish date to 2026-01-12
* remove initial header
* fix links
---------
Co-authored-by: Andrew Lamb <[email protected]>
---
content/blog/2026-01-12-extending-sql.md | 375 ++++++++++++++++++++++++++
content/images/extending-sql/architecture.svg | 151 +++++++++++
2 files changed, 526 insertions(+)
diff --git a/content/blog/2026-01-12-extending-sql.md
b/content/blog/2026-01-12-extending-sql.md
new file mode 100644
index 0000000..d4d9e7a
--- /dev/null
+++ b/content/blog/2026-01-12-extending-sql.md
@@ -0,0 +1,375 @@
+---
+layout: post
+title: Extending SQL in DataFusion: from ->> to TABLESAMPLE
+date: 2026-01-12
+author: Geoffrey Claude (Datadog)
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+If you embed [DataFusion][apache datafusion] in your product, your users will
eventually run SQL that DataFusion does not recognize. Not because the query is
unreasonable, but because SQL in practice includes many dialects and
system-specific statements.
+
+Suppose you store data as Parquet files on S3 and want users to attach an
external catalog to query them. DataFusion has `CREATE EXTERNAL TABLE` for
individual tables, but no built-in equivalent for catalogs. DuckDB has
`ATTACH`, SQLite has its own variant, and maybe you really want something even
more flexible:
+
+```sql
+CREATE EXTERNAL CATALOG my_lake
+STORED AS iceberg
+LOCATION 's3://my-bucket/warehouse'
+OPTIONS ('region' 'eu-west-1');
+```
+
+This syntax does not exist in DataFusion today, but you can add it.
+
+---
+
+At the same time, many dialect gaps are smaller and show up in everyday
queries:
+
+```sql
+-- Postgres-style JSON operators
+SELECT payload->'user'->>'id' FROM logs;
+
+-- MySQL-specific types
+SELECT DATETIME '2001-01-01 18:00:00';
+
+-- Statistical sampling
+SELECT * FROM sensor_data TABLESAMPLE BERNOULLI(10 PERCENT);
+```
+
+You can implement all of these _without forking_ DataFusion:
+
+1. **Parse** new syntax (custom statements / dialect quirks)
+2. **Plan** new semantics (expressions, types, FROM-clause constructs)
+3. **Execute** new operators when rewrites are not sufficient
+
+This post explains where and how to hook into each stage. For complete,
working code, see the linked `datafusion-examples`.
+
+---
+
+## Parse → Plan → Execute
+
+DataFusion turns SQL into executable work in stages:
+
+1. **Parse**: SQL text is parsed into an AST ([Statement] from [sqlparser-rs])
+2. **Logical planning**: [SqlToRel] converts the AST into a [LogicalPlan]
+3. **Physical planning**: The [PhysicalPlanner] turns the logical plan into an
[ExecutionPlan]
+
+Each stage has extension points.
+
+<figure>
+ <img src="/blog/images/extending-sql/architecture.svg" alt="DataFusion SQL
processing pipeline: SQL String flows through Parser to AST, then SqlToRel
(with Extension Planners) to LogicalPlan, then PhysicalPlanner to
ExecutionPlan" width="100%" class="img-responsive">
+ <figcaption>
+ <b>Figure 1:</b> SQL flows through three stages: parsing, logical planning
(via <code>SqlToRel</code>, where the Extension Planners hook in), and physical
planning. Each stage has extension points: wrap the parser, implement planner
traits, or add physical operators.
+ </figcaption>
+</figure>
+
+To choose the right extension point, look at where the query fails.
+
+| What fails? | What it looks like | Where to hook in
|
+| ----------- | ------------------------------------------- |
----------------------------------------------- |
+| Parsing | `Expected: TABLE, found: CATALOG` | configure
dialect or wrap `DFParser` |
+| Planning | `This feature is not implemented: DATETIME` | `ExprPlanner`,
`TypePlanner`, `RelationPlanner` |
+| Execution | `No physical plan for TableSample` |
`ExtensionPlanner` (+ physical operator) |
+
+We will follow that pipeline order.
+
+---
+
+## 1) Extending parsing: wrapping `DFParser` for custom statements
+
+The `CREATE EXTERNAL CATALOG` syntax from the introduction fails at the parser
because DataFusion only recognizes `CREATE EXTERNAL TABLE`. To support new
statement-level syntax, you can **wrap `DFParser`**. Peek ahead **in the token
stream** to detect your custom syntax, handle it yourself, and delegate
everything else to DataFusion.
+
+The [`custom_sql_parser.rs`][custom_sql_parser.rs] example demonstrates this
pattern:
+
+```rust
+struct CustomParser<'a> { df_parser: DFParser<'a> }
+
+impl<'a> CustomParser<'a> {
+ pub fn parse_statement(&mut self) -> Result<CustomStatement> {
+ // Peek tokens to detect CREATE EXTERNAL CATALOG
+ if self.is_create_external_catalog() {
+ return self.parse_create_external_catalog();
+ }
+ // Delegate everything else to DataFusion
+ Ok(CustomStatement::DFStatement(Box::new(
+ self.df_parser.parse_statement()?,
+ )))
+ }
+}
+```
+
+You do not need to implement a full SQL parser. Reuse DataFusion's tokenizer
and parser helpers to consume tokens, parse identifiers, and handle options—the
example shows how.
+
+Once parsed, the simplest integration is to treat custom statements as
**application commands**:
+
+```rust
+match parser.parse_statement()? {
+ CustomStatement::DFStatement(stmt) => ctx.sql(&stmt.to_string()).await?,
+ CustomStatement::CreateExternalCatalog(stmt) => {
+ handle_create_external_catalog(&ctx, stmt).await?
+ }
+}
+```
+
+This keeps the extension logic in your embedding application. The example
includes a complete `handle_create_external_catalog` that registers tables from
a location into a catalog, making them queryable immediately.
+
+**Full working example:** [`custom_sql_parser.rs`][custom_sql_parser.rs]
+
+---
+
+## 2) Extending expression semantics: `ExprPlanner`
+
+Once SQL _parses_, the next failure is often that DataFusion does not know
what a particular expression means.
+
+This is where dialect differences show up in day-to-day queries: operators
like Postgres JSON arrows, vendor-specific functions, or small syntactic sugar
that users expect to keep working when you switch engines.
+
+`ExprPlanner` lets you define how specific SQL expressions become DataFusion
`Expr`. Common examples:
+
+- Non-standard operators (JSON / geometry / regex operators)
+- Custom function syntaxes
+- Special identifier behavior
+
+### Example: Postgres JSON operators (`->`, `->>`)
+
+The Postgres `->` operator is a good illustration because it is widely used
and parses only under the PostgreSQL dialect.
+
+Configure the dialect:
+
+```rust
+let config = SessionConfig::new()
+ .set_str("datafusion.sql_parser.dialect", "postgres");
+let ctx = SessionContext::new_with_config(config);
+```
+
+Then implement `ExprPlanner` to map the parsed operator
(`BinaryOperator::Arrow`) to DataFusion semantics:
+
+```rust
+fn plan_binary_op(&self, expr: RawBinaryExpr, _schema: &DFSchema)
+ -> Result<PlannerResult<RawBinaryExpr>> {
+ match expr.op {
+ BinaryOperator::Arrow => Ok(Planned(/* your Expr */)),
+ _ => Ok(Original(expr)),
+ }
+}
+```
+
+Return `Planned(...)` when you handled the expression; return `Original(...)`
to pass it to the next planner.
+
+For a complete JSON implementation, see [datafusion-functions-json]. For a
minimal end-to-end example in the DataFusion repo, see
[`expr_planner_tests`][expr_planner_tests].
+
+---
+
+## 3) Extending type support: `TypePlanner`
+
+After expressions, types are often the next thing to break. Schemas and DDL
may reference types that DataFusion does not support out of the box, like
MySQL's `DATETIME`.
+
+Type planning tends to come up when interoperating with other systems. You
want to accept DDL or infer schemas from external catalogs without forcing
users to rewrite types.
+
+`TypePlanner` maps SQL types to Arrow/DataFusion types:
+
+```rust
+impl TypePlanner for MyTypePlanner {
+ fn plan_type(&self, sql_type: &ast::DataType) -> Result<Option<DataType>> {
+ match sql_type {
+ ast::DataType::Datetime(Some(3)) =>
Ok(Some(DataType::Timestamp(TimeUnit::Millisecond, None))),
+ _ => Ok(None), // let the default planner handle it
+ }
+ }
+}
+```
+
+It is installed when building session state:
+
+```rust
+let state = SessionStateBuilder::new()
+ .with_default_features()
+ .with_type_planner(Arc::new(MyTypePlanner))
+ .build();
+```
+
+Once installed, if your `CREATE EXTERNAL CATALOG` statement exposes tables
with MySQL types, DataFusion can interpret them correctly.
+
+---
+
+## 4) Extending the FROM clause: `RelationPlanner`
+
+Some extensions change what a _relation_ means, not just expressions or types.
`RelationPlanner` (available starting in DataFusion 52) intercepts FROM-clause
constructs while SQL is being converted into a `LogicalPlan`.
+
+Once you have `RelationPlanner`, there are two main approaches to implementing
your extension.
+
+### Strategy A: rewrite to existing operators (PIVOT / UNPIVOT)
+
+If you can translate your syntax into relational algebra that DataFusion
already supports, you can implement the feature with **no custom physical
operator**.
+
+`PIVOT` rotates rows into columns, and `UNPIVOT` does the reverse. Neither
requires new execution logic: `PIVOT` is just `GROUP BY` with `CASE`
expressions, and `UNPIVOT` is a `UNION ALL` of each column. The planner
rewrites them accordingly:
+
+```rust
+match relation {
+ TableFactor::Pivot { .. } => /* rewrite to GROUP BY + CASE */,
+ TableFactor::Unpivot { .. } => /* rewrite to UNION ALL */,
+ other => Original(other),
+}
+```
+
+Because the output is a standard `LogicalPlan`, DataFusion's usual
optimization and physical planning apply automatically.
+
+**Full working example:** [`pivot_unpivot.rs`][pivot_unpivot.rs]
+
+### Strategy B: custom logical + physical (TABLESAMPLE)
+
+Sometimes rewriting is not sufficient. `TABLESAMPLE` returns a random subset
of rows from a table and is useful for approximations or debugging on large
datasets. Because it requires runtime randomness, you cannot express it as a
rewrite to existing operators. Instead, you need a custom logical node and
physical operator to execute it.
+
+The approach (shown in [`table_sample.rs`][table_sample.rs]):
+
+1. `RelationPlanner` recognizes `TABLESAMPLE` and produces a custom logical
node
+2. That node gets wrapped in `LogicalPlan::Extension`
+3. `ExtensionPlanner` converts it to a custom `ExecutionPlan`
+
+In code:
+
+```rust
+// Logical planning: FROM t TABLESAMPLE (...) -> LogicalPlan::Extension(...)
+let plan = LogicalPlan::Extension(Extension { node:
Arc::new(TableSamplePlanNode { /* ... */ }) });
+```
+
+```rust
+// Physical planning: TableSamplePlanNode -> SampleExec
+if let Some(sample_node) = node.as_any().downcast_ref::<TableSamplePlanNode>()
{
+ return Ok(Some(Arc::new(SampleExec::try_new(input, /* bounds, seed */)?)));
+}
+```
+
+This is the general pattern for custom FROM constructs that need runtime
behavior.
+
+**Full working example:** [`table_sample.rs`][table_sample.rs]
+
+### Background: Origin of the API
+
+`RelationPlanner` originally came out of trying to build `MATCH_RECOGNIZE`
support in DataFusion as a Datadog hackathon project. `MATCH_RECOGNIZE` is a
complex SQL feature for detecting patterns in sequences of rows, and it made
sense to prototype as an extension first. At the time, DataFusion had no
extension point at the right stage of SQL-to-rel planning to intercept and
reinterpret relations.
+
+[@theirix]'s `TABLESAMPLE` work ([#13563], [#17633]) demonstrated exactly
where the gap was: their extension only worked when `TABLESAMPLE` appeared at
the query root and any `TABLESAMPLE` inside a CTE or JOIN would error. That
limitation motivated [#17843], which introduced `RelationPlanner` to intercept
relations at any nesting level. The same hook now supports `PIVOT`, `UNPIVOT`,
`TABLESAMPLE`, and can translate dialect-specific FROM-clause syntax (for
example, bridging Trino construc [...]
+
+This is how Datadog approaches compatibility work: build features in real
systems first, then upstream the building blocks. A full `MATCH_RECOGNIZE`
extension is now in progress, built on top of `RelationPlanner`, with the
[`match_recognize.rs`][match_recognize.rs] example as a starting point.
+
+---
+
+## Summary: The Extensibility Workflow
+
+DataFusion's SQL extensibility follows its processing pipeline. When building
your own dialect extension, work incrementally:
+
+1. **Parse**: Use a parser wrapper to intercept custom syntax in the token
stream. Produce either a standard `Statement` or your own application-specific
command.
+2. **Plan**: Implement the planning traits (`ExprPlanner`, `TypePlanner`,
`RelationPlanner`) to give your syntax meaning.
+3. **Execute**: Prefer rewrites to existing operators (like `PIVOT` to
`CASE`). Only add custom physical operators via `ExtensionPlanner` when you
need specific runtime behavior like randomness or specialized I/O.
+
+---
+
+## Debugging tips
+
+### Print the logical plan
+
+```rust
+let df = ctx.sql("SELECT * FROM t TABLESAMPLE (10 PERCENT)").await?;
+println!("{}", df.logical_plan().display_indent());
+```
+
+### Use [`EXPLAIN`][EXPLAIN]
+
+```sql
+EXPLAIN SELECT * FROM t TABLESAMPLE (10 PERCENT);
+```
+
+If your extension is not being invoked, it is usually visible in the logical
plan first.
+
+---
+
+## When hooks aren't enough
+
+While these extension points cover the majority of dialect needs, some deep
architectural areas still have limited or no hooks. If you are working in these
parts of the SQL surface area, you may need to contribute upstream:
+
+- Statement-level planning: [`statement.rs`][df_statement_planning]
+- JOIN planning: [`relation/join.rs`][df_join_planning]
+- TOP / FETCH clauses: [`select.rs`][df_select_planning],
[`query.rs`][df_query_planning]
+
+---
+
+## Ideas to try
+
+If you want to experiment with these extension points, here are a few
suggestions:
+
+- Geometry operators (for example `@>`, `<@`) via `ExprPlanner`
+- Oracle `NUMBER` or SQL Server `MONEY` via `TypePlanner`
+- `JSON_TABLE` or semantic-layer style relations via `RelationPlanner`
+
+---
+
+## See also
+
+- Extending SQL Guide: [Extending SQL Guide][extending sql guide]
+- Parser wrapping example: [`custom_sql_parser.rs`][custom_sql_parser.rs]
+- RelationPlanner examples:
+
+ - `PIVOT` / `UNPIVOT`: [`pivot_unpivot.rs`][pivot_unpivot.rs]
+ - `TABLESAMPLE`: [`table_sample.rs`][table_sample.rs]
+
+- ExprPlanner test examples: [`expr_planner_tests`][expr_planner_tests]
+
+## Acknowledgements
+
+Thank you to [@jayzhan211] for designing and implementing the original
`ExprPlanner` API ([#11180]), to [@goldmedal] for adding `TypePlanner`
([#13294]), and to [@theirix] for the `TABLESAMPLE` work ([#13563], [#17633])
that helped shape `RelationPlanner`. Thank you to [@alamb] for driving
DataFusion's extensibility philosophy and for feedback on this post.
+
+## Get Involved
+
+- **Try it out**: Implement one of the extension points and share your
experience
+- **File issues or join the conversation**: [GitHub][datafusion github] for
bugs and feature requests, [Slack or Discord][communication] for discussion
+
+<!-- Reference links -->
+
+[apache datafusion]: https://datafusion.apache.org/
+[datafusion github]: https://github.com/apache/datafusion/
+[sqlparser-rs]: https://github.com/sqlparser-rs/sqlparser-rs
+[extending sql guide]:
https://datafusion.apache.org/library-user-guide/extending-sql.html
+[custom_sql_parser.rs]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_ops/custom_sql_parser.rs
+[pivot_unpivot.rs]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/pivot_unpivot.rs
+[table_sample.rs]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/table_sample.rs
+[match_recognize.rs]:
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/relation_planner/match_recognize.rs
+[expr_planner_tests]:
https://github.com/apache/datafusion/blob/main/datafusion/core/tests/user_defined/expr_planner.rs
+[datafusion-functions-json]:
https://github.com/datafusion-contrib/datafusion-functions-json
+[communication]:
https://datafusion.apache.org/contributor-guide/communication.html
+[@jayzhan211]: https://github.com/jayzhan211
+[@goldmedal]: https://github.com/goldmedal
+[@alamb]: https://github.com/alamb
+[@theirix]: https://github.com/theirix
+[#11180]: https://github.com/apache/datafusion/pull/11180
+[#13294]: https://github.com/apache/datafusion/pull/13294
+[#13563]: https://github.com/apache/datafusion/issues/13563
+[#17633]: https://github.com/apache/datafusion/pull/17633
+[#17843]: https://github.com/apache/datafusion/pull/17843
+[df_statement_planning]:
https://github.com/apache/datafusion/blob/main/datafusion/sql/src/statement.rs
+[df_join_planning]:
https://github.com/apache/datafusion/blob/main/datafusion/sql/src/relation/join.rs
+[df_select_planning]:
https://github.com/apache/datafusion/blob/main/datafusion/sql/src/select.rs
+[df_query_planning]:
https://github.com/apache/datafusion/blob/main/datafusion/sql/src/query.rs
+[Statement]: https://docs.rs/sqlparser/latest/sqlparser/ast/enum.Statement.html
+[SqlToRel]:
https://docs.rs/datafusion/latest/datafusion/sql/planner/struct.SqlToRel.html
+[LogicalPlan]:
https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html
+[PhysicalPlanner]:
https://docs.rs/datafusion/latest/datafusion/physical_planner/trait.PhysicalPlanner.html
+[ExecutionPlan]:
https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html
+[EXPLAIN]: https://datafusion.apache.org/user-guide/sql/explain.html
diff --git a/content/images/extending-sql/architecture.svg
b/content/images/extending-sql/architecture.svg
new file mode 100644
index 0000000..27f5edc
--- /dev/null
+++ b/content/images/extending-sql/architecture.svg
@@ -0,0 +1,151 @@
+<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1100 383"
style="max-width: 100%; height: auto;" role="img" aria-label="DataFusion SQL
processing pipeline: SQL String flows through Parser to AST, then SqlToRel
(with Extension Planners) to LogicalPlan, then PhysicalPlanner to
ExecutionPlan">
+ <defs>
+ <marker id="arrowhead" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#94a3b8"/>
+ </marker>
+ <marker id="arrowhead-rose" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#c0392b"/>
+ </marker>
+ <marker id="arrowhead-amber" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#e67e22"/>
+ </marker>
+ <marker id="arrowhead-lime" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#f1c40f"/>
+ </marker>
+ <marker id="arrowhead-emerald" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#27ae60"/>
+ </marker>
+ <marker id="arrowhead-sky" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#3498db"/>
+ </marker>
+ <marker id="arrowhead-violet" markerWidth="8" markerHeight="5.6" refX="7"
refY="2.8" orient="auto">
+ <polygon points="0 0, 8 2.8, 0 5.6" fill="#9b59b6"/>
+ </marker>
+ <filter id="shadow" x="-20%" y="-20%" width="140%" height="140%">
+ <feDropShadow dx="2" dy="2" stdDeviation="2" flood-opacity="0.12"/>
+ </filter>
+ </defs>
+
+ <!-- White background -->
+ <rect x="0" y="0" width="1100" height="383" fill="white"/>
+
+ <!-- Background groupings (drawn first, behind everything) -->
+ <rect x="153.75" y="35" width="113.49999999999999" height="200" rx="2"
fill="#e4beba" fill-opacity="0.5"/>
+ <rect x="286" y="55" width="98" height="180" rx="24" fill="#f1dcc9"
fill-opacity="0.5"/>
+ <rect x="402.75" y="35" width="182.5" height="250" rx="2" fill="#f3e9bf"
fill-opacity="0.5"/>
+ <rect x="604" y="55" width="142" height="180" rx="24" fill="#aaddc0"
fill-opacity="0.5"/>
+ <rect x="764.75" y="35" width="159.5" height="200" rx="2" fill="#d2e4f0"
fill-opacity="0.5"/>
+ <rect x="943" y="55" width="142" height="180" rx="24" fill="#e3dbe6"
fill-opacity="0.5"/>
+
+ <!-- Row 1: Main pipeline -->
+
+ <!-- SQL String (DATA) -->
+ <rect id="box-sql-string" x="20" y="60" width="110.00000000000001"
height="50" rx="24" fill="#e2e8f0" stroke="#64748b" stroke-width="2"
filter="url(#shadow)"/>
+ <text id="text-sql-string" x="75" y="90" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#334155">SQL String</text>
+
+ <line id="arrow-sql-to-parser" x1="131" y1="85" x2="156.75" y2="85"
stroke="#c0392b" stroke-width="2" marker-end="url(#arrowhead-rose)"/>
+
+ <!-- Parser (TRANSFORMER) - Slot 1 -->
+ <rect id="box-parser" x="158.75" y="40" width="103.49999999999999"
height="90" rx="2" fill="#e4beba" stroke="#c0392b" stroke-width="2"
filter="url(#shadow)"/>
+ <text id="text-parser-1" x="210.5" y="70" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#43140f">Parser</text>
+ <text id="text-parser-2" x="210.5" y="88" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13"
fill="#7e0d01">(sqlparser-rs)</text>
+ <text id="text-parser-3" x="210.5" y="115" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" font-weight="600"
fill="#7e0d01">Wrap DFParser</text>
+
+ <line id="arrow-parser-to-ast" x1="263.25" y1="85" x2="289" y2="85"
stroke="#e67e22" stroke-width="2" marker-end="url(#arrowhead-amber)"/>
+
+ <!-- DFParser wrapper box (TRANSFORMER) - Slot 1 -->
+ <rect id="box-dfparser" x="158.75" y="179" width="103.49999999999999"
height="50" rx="2" fill="#e4beba" stroke="#c0392b" stroke-width="2"
filter="url(#shadow)"/>
+ <text id="text-dfparser-1" x="210.5" y="202" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#43140f">DFParser</text>
+ <text id="text-dfparser-2" x="210.5" y="217" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13"
fill="#7e0d01">(wrapper)</text>
+
+ <!-- AST (DATA) - Slot 2 -->
+ <rect id="box-ast" x="291" y="60" width="88" height="50" rx="24"
fill="#f1dcc9" stroke="#e67e22" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-ast-1" x="335" y="82" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#64350b">AST</text>
+ <text id="text-ast-2" x="335" y="98" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13"
fill="#803c00">(Statement)</text>
+
+ <line id="arrow-ast-to-sqltorel" x1="380" y1="85" x2="405.75" y2="85"
stroke="#f1c40f" stroke-width="2" marker-end="url(#arrowhead-lime)"/>
+
+ <!-- Custom Statement box (DATA) - Slot 2 -->
+ <rect id="box-custom-statement" x="291" y="179" width="88" height="50"
rx="24" fill="#f1dcc9" stroke="#e67e22" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-custom-statement-1" x="335" y="199" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="14.5" font-weight="600"
fill="#64350b">Custom</text>
+ <text id="text-custom-statement-2" x="335" y="214" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#64350b">Statement</text>
+
+ <!-- Arrow from DFParser to Custom Statement -->
+ <line id="arrow-dfparser-to-custom-statement" x1="263.25" y1="204" x2="289"
y2="204" stroke="#e67e22" stroke-width="2" marker-end="url(#arrowhead-amber)"/>
+
+ <!-- SqlToRel (TRANSFORMER) - Slot 3 -->
+ <rect id="box-sqltorel" x="407.75" y="40" width="172.5" height="90" rx="2"
fill="#f3e9bf" stroke="#f1c40f" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-sqltorel-1" x="494" y="70" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#614f06">SqlToRel</text>
+ <text id="text-sqltorel-2" x="494" y="88" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" fill="#806600">(AST to
Logical)</text>
+ <text id="text-sqltorel-3" x="494" y="115" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" font-weight="600"
fill="#806600">Extension Planners</text>
+
+ <line id="arrow-sqltorel-to-logicalplan" x1="581.25" y1="85" x2="607"
y2="85" stroke="#27ae60" stroke-width="2" marker-end="url(#arrowhead-emerald)"/>
+
+ <!-- Extension Planners container (TRANSFORMER) - Slot 3 -->
+ <rect id="box-extension-planners" x="407.75" y="165" width="172.5"
height="115" rx="2" fill="#f3e9bf" stroke="#f1c40f" stroke-width="2"
filter="url(#shadow)"/>
+ <text id="text-extension-planners-1" x="494" y="183" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="14.5" font-weight="600"
fill="#614f06">Extension Planners</text>
+
+ <!-- RelationPlanner sub-box (top - produces UserDefinedLogicalNode) -->
+ <rect id="box-relation-planner" x="419.25" y="192" width="149.5" height="24"
rx="2" fill="#fbf8e9" stroke="#f1c40f" stroke-width="1"/>
+ <text id="text-relation-planner" x="494" y="208" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="13" font-weight="600"
fill="#614f06">RelationPlanner</text>
+
+ <!-- ExprPlanner sub-box -->
+ <rect id="box-expr-planner" x="419.25" y="220" width="149.5" height="24"
rx="2" fill="#fbf8e9" stroke="#f1c40f" stroke-width="1"/>
+ <text id="text-expr-planner" x="494" y="236" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="13" font-weight="600"
fill="#614f06">ExprPlanner</text>
+
+ <!-- TypePlanner sub-box -->
+ <rect id="box-type-planner" x="419.25" y="248" width="149.5" height="24"
rx="2" fill="#fbf8e9" stroke="#f1c40f" stroke-width="1"/>
+ <text id="text-type-planner" x="494" y="264" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="13" font-weight="600"
fill="#614f06">TypePlanner</text>
+
+ <!-- LogicalPlan (DATA) - Slot 4 -->
+ <rect id="box-logicalplan" x="609" y="60" width="132" height="50" rx="24"
fill="#aaddc0" stroke="#27ae60" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-logicalplan" x="675" y="90" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#0e3e22">LogicalPlan</text>
+
+ <line id="arrow-logicalplan-to-physicalplanner" x1="742" y1="85" x2="767.75"
y2="85" stroke="#3498db" stroke-width="2" marker-end="url(#arrowhead-sky)"/>
+
+ <!-- UserDefinedLogicalNode box (DATA) - Slot 4 -->
+ <rect id="box-userdefined-logical" x="609" y="179" width="132" height="50"
rx="24" fill="#aaddc0" stroke="#27ae60" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-userdefined-logical-1" x="675" y="199" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#0e3e22">UserDefined</text>
+ <text id="text-userdefined-logical-2" x="675" y="214" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#0e3e22">LogicalNode</text>
+
+ <!-- Arrow from RelationPlanner to UserDefinedLogicalNode -->
+ <line id="arrow-relationplanner-to-userdefined" x1="569.75" y1="204"
x2="607" y2="204" stroke="#27ae60" stroke-width="2"
marker-end="url(#arrowhead-emerald)"/>
+
+ <!-- PhysicalPlanner (TRANSFORMER) - Slot 5 -->
+ <rect id="box-physicalplanner" x="769.75" y="40" width="149.5" height="90"
rx="2" fill="#d2e4f0" stroke="#3498db" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-physicalplanner-1" x="844.5" y="70" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#124364">PhysicalPlanner</text>
+ <text id="text-physicalplanner-2" x="844.5" y="88" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" fill="#004c80">(Logical to
Physical)</text>
+ <text id="text-physicalplanner-3" x="844.5" y="115" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" font-weight="600"
fill="#004c80">ExtensionPlanner</text>
+
+ <line id="arrow-physicalplanner-to-executionplan" x1="920.25" y1="85"
x2="946" y2="85" stroke="#9b59b6" stroke-width="2"
marker-end="url(#arrowhead-violet)"/>
+
+ <!-- ExtensionPlanner box (TRANSFORMER) - Slot 5 -->
+ <rect id="box-extensionplanner" x="769.75" y="179" width="149.5" height="50"
rx="2" fill="#d2e4f0" stroke="#3498db" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-extensionplanner-1" x="844.5" y="202" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#124364">ExtensionPlanner</text>
+ <text id="text-extensionplanner-2" x="844.5" y="217" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="13" fill="#004c80">(trait)</text>
+
+ <!-- ExecutionPlan (DATA) - Slot 6 -->
+ <rect id="box-executionplan" x="948" y="60" width="132" height="50" rx="24"
fill="#e3dbe6" stroke="#9b59b6" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-executionplan" x="1014" y="90" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#452452">ExecutionPlan</text>
+
+ <!-- Custom ExecutionPlan box (DATA) - Slot 6 -->
+ <rect id="box-custom-execution" x="948" y="179" width="132" height="50"
rx="24" fill="#e3dbe6" stroke="#9b59b6" stroke-width="2" filter="url(#shadow)"/>
+ <text id="text-custom-execution-1" x="1014" y="199" text-anchor="middle"
font-family="system-ui, sans-serif" font-size="14.5" font-weight="600"
fill="#452452">Custom</text>
+ <text id="text-custom-execution-2" x="1014" y="214" text-anchor="middle"
font-family="ui-monospace, monospace" font-size="14.5" font-weight="600"
fill="#452452">ExecutionPlan</text>
+
+ <!-- Arrow from ExtensionPlanner to Custom ExecutionPlan -->
+ <line id="arrow-extensionplanner-to-custom-execution" x1="920.25" y1="204"
x2="946" y2="204" stroke="#9b59b6" stroke-width="2"
marker-end="url(#arrowhead-violet)"/>
+
+ <!-- Legend at bottom (aligned with transformers: Parser@155, SqlToRel@395,
PhysicalPlanner@735) -->
+ <rect x="20" y="313" width="1060" height="55" rx="6" fill="#f8fafc"
stroke="#e2e8f0" stroke-width="1"/>
+ <text x="40" y="338" font-family="system-ui, sans-serif" font-size="15.0"
font-weight="600" fill="#475569">Hook Points:</text>
+ <rect x="158.75" y="327" width="12" height="12" rx="2" fill="#e4beba"
stroke="#c0392b" stroke-width="1"/>
+ <text id="legend-text-1" x="176.75" y="337" font-family="system-ui,
sans-serif" font-size="14" fill="#475569">Parser (wrap DFParser)</text>
+ <rect x="407.75" y="327" width="12" height="12" rx="2" fill="#f3e9bf"
stroke="#f1c40f" stroke-width="1"/>
+ <text id="legend-text-2" x="425.75" y="337" font-family="system-ui,
sans-serif" font-size="14" fill="#475569">SQL Planning (operators, types,
relations)</text>
+ <rect x="769.75" y="327" width="12" height="12" rx="2" fill="#d2e4f0"
stroke="#3498db" stroke-width="1"/>
+ <text id="legend-text-3" x="787.75" y="337" font-family="system-ui,
sans-serif" font-size="14" fill="#475569">Physical Planning (custom
execution)</text>
+ <text id="legend-text-4" x="158.75" y="355" font-family="system-ui,
sans-serif" font-size="14" fill="#64748b">Parse errors: wrap parser</text>
+ <text id="legend-text-5" x="407.75" y="355" font-family="system-ui,
sans-serif" font-size="14" fill="#64748b">Plan errors: implement planners</text>
+ <text id="legend-text-6" x="769.75" y="355" font-family="system-ui,
sans-serif" font-size="14" fill="#64748b">Execute errors: implement
ExtensionPlanner</text>
+</svg>
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
