maver1ck opened a new issue, #26084:
URL: https://github.com/apache/datafusion/issues/26084
### Is your feature request related to a problem or challenge?
I would like downstream applications to register custom top-level SQL
statements and use them through DataFusion’s normal session APIs, without
maintaining a separate SQL dispatcher or adding a command-specific variant to
DataFusion core for each feature.
For example, an extension could implement Snowflake-style CREATE SEMANTIC
VIEW. The request is not to implement semantic views in core, but to provide
the extension points needed to carry a custom statement from parsing through
planning to execution.
### Describe the solution you'd like
Provide three complementary extension points:
1. Statement parser: Recognize custom statements and return a typed
extension AST, reusing existing tokenization, identifier/expression parsing,
and diagnostics. A nonmatching extension must leave the token position
unchanged; malformed recognized syntax should report its error rather than
silently fall back. Extension precedence should be explicit.
2. Statement planner: Convert the extension AST into a logical plan or typed
DDL command. Provide access to normal name resolution, expression planning,
type checking, and dependency discovery. Parsing and planning must not execute
mutations.
3. DDL executor: Execute custom commands asynchronously through the standard
session execution path. Custom DDL must respect restrictions such as
with_allow_ddl(false) and execute once per explicit command execution—not per
batch or partition. Preparing, describing, or explaining a command must not
trigger its mutations.
These extensions should be registered through parser/session configuration
and support an extensible statement representation, rather than requiring a new
core enum variant for every downstream command.
For example, a downstream extension could accept:
```
CREATE SEMANTIC VIEW analytics.sales
TABLES (
orders AS public.orders PRIMARY KEY (id)
)
DIMENSIONS (
orders.country AS orders.country
)
METRICS (
orders.revenue AS SUM(orders.amount)
);
```
The intended path is:
```
SQL → registered parser → extension AST
→ registered planner → logical command
→ registered executor
```
The semantic implementation would remain downstream. With no extensions
registered, existing SQL behavior should remain unchanged.
### Describe alternatives you've considered
1. Application-level parser wrapper and dispatcher: Provides a workable
integration, but requires each application to maintain routing outside the
standard session SQL path.
2. Command-specific changes to core: Solve an individual use case but
require further core changes for each additional statement.
3. External-table syntax or side-effecting functions: May suit narrower use
cases, but do not provide a general extension point for new statements with a
distinct DDL lifecycle.
### Additional context
The initial scope is top-level statements—not arbitrary grammar injection, a
semantic-view implementation, a new execution engine, or a stable binary plugin
ABI.
The architectural inspiration is Calcite-style parser/DDL extensibility, not
a requirement to reproduce its parser generator or APIs. Trait names and
registration details are open for discussion.
Would maintainers prefer parser/planner extensibility and DDL execution as
separate proposals or PRs? Is there an existing mechanism that covers this
complete path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
