morrySnow commented on code in PR #3997:
URL: https://github.com/apache/doris-website/pull/3997#discussion_r3602591894
##########
docs/sql-manual/sql-statements/data-governance/data-lineage.md:
##########
@@ -0,0 +1,336 @@
+---
+{
+ "title": "Data Lineage Technical Guide",
+ "sidebar_label": "Data Lineage",
+ "language": "en",
+ "description": "Collect table-level and column-level lineage from
supported Doris DML statements through an external LineagePlugin.",
+ "keywords": ["Apache Doris", "data lineage", "data governance",
"LineagePlugin", "column-level lineage"]
+}
+---
+
+<!-- Knowledge type: Feature overview + technical design + user guide -->
+<!-- Use cases: Data governance integration / impact analysis / compliance
traceability -->
+
+Data Lineage extracts table-level and column-level dependencies from supported
Doris DML statements and sends them to external governance systems through a
`LineagePlugin`. Doris is the lineage producer. It does not retain events,
expose a lineage query API, or provide lineage visualization.
+
+> This capability is available in the Dev version. It was first released in
Apache Doris 4.0.6.
+
+## Capabilities and limitations
+
+The framework generates an event only after one of the following statements
succeeds:
+
+| Supported statement | Event behavior |
+| --- | --- |
+| `INSERT INTO ... SELECT` | Generates one event after the Insert succeeds. |
+| `INSERT OVERWRITE ... SELECT` | Generates one event after the Overwrite
succeeds. |
+| `CREATE TABLE AS SELECT` | Generates one event after the internal Insert
succeeds. |
+
+The following operations do not generate an event: `SELECT`, `UPDATE`,
`DELETE`, load jobs, `VALUES`-only inserts, and writes whose target is
`__internal_schema`.
+
+:::caution Delivery guarantee
+
Review Comment:
The SQL syntax used throughout this document (`INSERT OVERWRITE ... SELECT`)
drops the `TABLE` keyword. The standard Doris SQL syntax is `INSERT OVERWRITE
TABLE table_name ... SELECT ...`. The command class in the code is
`InsertOverwriteTableCommand`. While this may be intended as shorthand in
prose, the supported-statements table should use the precise syntax. The SVG
diagrams (`lineage-architecture.svg`, `lineage-event-model.svg`) also use
`INSERT OVERWRITE SELECT` without `TABLE`.
##########
community/developer-guide/data-lineage-plugin-development.md:
##########
@@ -0,0 +1,604 @@
+---
+title: Data Lineage Plugin Development
+language: en
+description: Build, package, deploy, and verify an Apache Doris data lineage
plugin that logs lineage events or forwards them to an external governance
system.
+keywords:
+ - Apache Doris
+ - data lineage
+ - LineagePlugin
+ - LineagePluginFactory
+ - ServiceLoader
+ - plugin development
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<!-- Knowledge type: Extension development -->
+<!-- Applicable scenario: Data governance integration / Kernel extension
development -->
+
+# Data Lineage Plugin Development
+
+This document describes how to develop an external data lineage plugin for
Apache Doris. The plugin receives a `LineageInfo` event after a supported DML
statement succeeds, then converts the event and writes it to logs, a messaging
system, or an external metadata governance platform.
+
+> Applicable version: Apache Doris community edition 4.0.6 and later. Compile
the plugin against the exact Doris community release or source revision
deployed on the FE.
+
+## Scope and lifecycle
+
+The lineage framework is an FE-side SPI. It does not store lineage data,
provide a query API, or render a lineage graph. A plugin converts the in-memory
Java objects in `LineageInfo` to the format and transport protocol required by
its downstream system.
+
+The framework generates events only for successful `INSERT INTO ... SELECT`,
`INSERT OVERWRITE ... SELECT`, and `CREATE TABLE AS SELECT` statements.
`VALUES`-only writes and writes whose target is `__internal_schema` are
skipped. `SELECT`, `UPDATE`, `DELETE`, and load jobs are outside the scope of
this framework.
+
+
+
+`eventFilter()` is called once before lineage extraction on the DML query path
and again before worker dispatch. Return `false` when the plugin should not
receive events. The method is called concurrently from query threads and the
worker thread, so it must be thread-safe. `exec()` is called by one worker
thread, but can run concurrently with `eventFilter()`.
+
+At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE
then calls the Factory's `create(context)` method and calls
`initialize(context)` on the returned plugin. The default
`LineagePluginFactory.create(context)` implementation delegates to the
no-argument `create()`, so the minimal example in this guide implements only
`create()`. `PluginContext` contains `plugin.name` and `plugin.path`. A
production plugin that reads a configuration file from its plugin directory can
override `initialize(context)`, locate the file through `plugin.path`, and
initialize its resources there.
+
+## Lineage event model
+
+`LineageInfo` contains a target table, target output columns, source tables,
direct lineage, indirect lineage, and a `LineageContext`.
+
+| Part | API | Meaning |
Review Comment:
The phrase "`PluginContext` contains `plugin.name` and `plugin.path`" is
imprecise. Looking at the `PluginContext` class
(`fe-extension-spi/.../PluginContext.java`), it only exposes `getProperties()`
which returns a `Map<String, String>`. The properties `"plugin.name"` and
`"plugin.path"` are stored as map entries (set in `discoverPlugins()` via
`props.put("plugin.path", ...)` and `props.put("plugin.name", ...)`). A
developer reading this would look for `context.getPluginName()` or
`context.pluginName` which don't exist. Please clarify: accessed via
`context.getProperties().get("plugin.name")` and
`context.getProperties().get("plugin.path")`. Same issue exists in the Chinese
version at the corresponding position.
##########
docs/sql-manual/sql-statements/data-governance/data-lineage.md:
##########
@@ -0,0 +1,336 @@
+---
+{
+ "title": "Data Lineage Technical Guide",
+ "sidebar_label": "Data Lineage",
+ "language": "en",
+ "description": "Collect table-level and column-level lineage from
supported Doris DML statements through an external LineagePlugin.",
+ "keywords": ["Apache Doris", "data lineage", "data governance",
"LineagePlugin", "column-level lineage"]
+}
+---
+
+<!-- Knowledge type: Feature overview + technical design + user guide -->
+<!-- Use cases: Data governance integration / impact analysis / compliance
traceability -->
+
+Data Lineage extracts table-level and column-level dependencies from supported
Doris DML statements and sends them to external governance systems through a
`LineagePlugin`. Doris is the lineage producer. It does not retain events,
expose a lineage query API, or provide lineage visualization.
+
+> This capability is available in the Dev version. It was first released in
Apache Doris 4.0.6.
+
+## Capabilities and limitations
+
+The framework generates an event only after one of the following statements
succeeds:
+
+| Supported statement | Event behavior |
+| --- | --- |
+| `INSERT INTO ... SELECT` | Generates one event after the Insert succeeds. |
+| `INSERT OVERWRITE ... SELECT` | Generates one event after the Overwrite
succeeds. |
+| `CREATE TABLE AS SELECT` | Generates one event after the internal Insert
succeeds. |
+
+The following operations do not generate an event: `SELECT`, `UPDATE`,
`DELETE`, load jobs, `VALUES`-only inserts, and writes whose target is
`__internal_schema`.
+
+:::caution Delivery guarantee
+
+Lineage delivery is asynchronous and best effort. A successful DML statement
is not rolled back when event collection or plugin delivery fails. Events can
be discarded when the queue is full, and Doris does not retry or persist them.
+
+:::
+
+## Technical design
+
+### Collection flow
+
+For a supported statement, Doris records the analyzed Nereids logical plan in
an `afterAnalyze` planner hook. After the DML succeeds, Doris extracts lineage
from that plan and submits the event to an FE-local queue. A single daemon
worker dispatches the event to every active plugin.
+
+
+
+Before extraction, Doris calls `eventFilter()` on the loaded plugins. If no
plugin is willing to receive events, extraction is skipped. The worker
evaluates `eventFilter()` again before dispatch.
+
+### Lineage content
+
+Each `LineageInfo` event contains table lineage, direct column lineage, two
kinds of indirect lineage, and query context. The complete SQL example below
establishes the source tables, target table, and result used by the following
sections.
+
+#### Example SQL and result
+
+The current user must have privileges to create a database and tables, write
data, and query data, and the FE must have loaded a lineage plugin. The example
uses two source tables and creates a customer summary through a CTE, Join,
filters, aggregation, a window function, a conditional expression, and sorting.
+
+```sql
+CREATE DATABASE IF NOT EXISTS lineage_demo;
+USE lineage_demo;
+
+CREATE TABLE lineage_orders (
+ order_id BIGINT,
+ customer_id BIGINT,
+ region VARCHAR(16),
+ amount DECIMAL(18, 2),
+ status VARCHAR(16)
+)
+DUPLICATE KEY(order_id)
+DISTRIBUTED BY HASH(order_id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+
+CREATE TABLE lineage_customers (
+ customer_id BIGINT,
+ customer_name VARCHAR(64),
+ customer_level VARCHAR(16)
+)
+DUPLICATE KEY(customer_id)
+DISTRIBUTED BY HASH(customer_id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+
+CREATE TABLE lineage_customer_summary (
+ customer_id BIGINT,
+ region_label VARCHAR(16),
+ total_amount DECIMAL(18, 2),
+ customer_seq BIGINT,
+ region_group VARCHAR(16)
+)
+DUPLICATE KEY(customer_id)
+DISTRIBUTED BY HASH(customer_id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+
+INSERT INTO lineage_customers VALUES
+ (1, 'Alice', 'VIP'),
+ (2, 'Bob', 'VIP'),
+ (3, 'Carol', 'REGULAR');
+
+INSERT INTO lineage_orders VALUES
+ (101, 1, 'east', 100.00, 'PAID'),
+ (102, 1, 'east', 50.00, 'PAID'),
+ (103, 2, 'west', 80.00, 'PAID'),
+ (104, 2, 'west', 30.00, 'CANCELLED'),
+ (105, 3, 'north', 200.00, 'PAID');
+
+INSERT INTO lineage_customer_summary
+WITH customer_totals AS (
+ SELECT
+ o.customer_id,
+ UPPER(o.region) AS region_label,
+ SUM(o.amount) AS total_amount
+ FROM lineage_orders o
+ JOIN lineage_customers c
+ ON o.customer_id = c.customer_id
+ WHERE o.status = 'PAID'
+ AND c.customer_level = 'VIP'
+ GROUP BY o.customer_id, UPPER(o.region)
+ HAVING SUM(o.amount) >= 50
+)
+SELECT
+ customer_id,
+ region_label,
+ total_amount,
+ ROW_NUMBER() OVER (ORDER BY customer_id) AS customer_seq,
+ CASE
+ WHEN region_label = 'EAST' THEN 'CORE'
+ ELSE 'OTHER'
+ END AS region_group
+FROM customer_totals
+ORDER BY total_amount DESC;
+
+SELECT customer_id, region_label, total_amount, customer_seq, region_group
+FROM lineage_customer_summary
+ORDER BY customer_id;
+```
+
+The query returns:
+
+```text
++-------------+--------------+--------------+--------------+--------------+
+| customer_id | region_label | total_amount | customer_seq | region_group |
++-------------+--------------+--------------+--------------+--------------+
+| 1 | EAST | 150.00 | 1 | CORE |
+| 2 | WEST | 80.00 | 2 | OTHER |
++-------------+--------------+--------------+--------------+--------------+
+```
+
+The two `VALUES`-only inserts do not generate lineage events. The `INSERT INTO
lineage_customer_summary ... SELECT` statement generates one `LineageInfo`
event, whose contents are described below.
+
+#### Table lineage
+
+Table lineage records one target table and every source table scanned by the
analyzed logical plan. The query that writes `lineage_customer_summary` scans
both `lineage_orders` and `lineage_customers`, so both source tables have a
table-level relationship with the target. A CTE is resolved to its underlying
tables, and every `UNION` branch contributes to the source-table set.
+
+This example produces the following table lineage:
+
+| Source table | Target table |
+| --- | --- |
+| `lineage_orders` | `lineage_customer_summary` |
+| `lineage_customers` | `lineage_customer_summary` |
+
+#### Direct column lineage
+
+Direct column lineage describes which source expression produces each
target-column value. After resolving aliases and CTEs, Doris classifies
expressions in this order: an expression containing an aggregate function is
`AGGREGATION`; a plain source-column reference is `IDENTITY`; every other
expression is `TRANSFORMATION`.
+
+| Type | Meaning | Example |
+| --- | --- | --- |
+| `IDENTITY` | The target value is taken directly from a source column without
a function or operation. | `lineage_customer_summary.customer_id` comes
directly from `lineage_orders.customer_id`. |
+| `TRANSFORMATION` | A non-aggregate expression transforms the target value,
such as arithmetic, a string function, a window function, or a conditional
expression. | `region_label` comes from `UPPER(lineage_orders.region)`;
`customer_seq` comes from `ROW_NUMBER()`. |
+| `AGGREGATION` | The target value is produced by an expression that contains
an aggregate function. It remains this type even when another expression wraps
the aggregate. | `total_amount` comes from `SUM(lineage_orders.amount)`. |
+
+This example produces the following direct column lineage:
+
+| Target column | Source expression | Type |
+| --- | --- | --- |
+| `lineage_customer_summary.customer_id` | `lineage_orders.customer_id` |
`IDENTITY` |
+| `lineage_customer_summary.region_label` | `UPPER(lineage_orders.region)` |
`TRANSFORMATION` |
+| `lineage_customer_summary.total_amount` | `SUM(lineage_orders.amount)` |
`AGGREGATION` |
+| `lineage_customer_summary.customer_seq` | `ROW_NUMBER() OVER (ORDER BY
lineage_orders.customer_id)` | `TRANSFORMATION` |
+| `lineage_customer_summary.region_group` | `CASE WHEN
UPPER(lineage_orders.region) = 'EAST' ... END` | `TRANSFORMATION` |
+
+#### Dataset indirect lineage
+
+Dataset indirect lineage records expressions that affect the complete result
set without directly producing a target-column value. These dependencies are
stored in an event-level map and can be applied to every target output column
when needed.
+
+| Type | Meaning | Example |
+| --- | --- | --- |
+| `JOIN` | Join conditions determine which source rows can be combined. |
`lineage_orders.customer_id = lineage_customers.customer_id`. |
+| `FILTER` | `WHERE` or `HAVING` conditions determine which rows or groups
enter the result. | `status = 'PAID'`, `customer_level = 'VIP'`, and `HAVING
SUM(amount) >= 50`. |
+| `GROUP_BY` | Grouping expressions determine the granularity of aggregate
results. | Group by `customer_id` and `UPPER(region)`. |
+| `SORT` | `ORDER BY` or TopN expressions determine result order. | Sort by
`total_amount DESC`, which resolves to `SUM(amount)`. |
+
+This example produces the following dataset indirect lineage:
+
+| Type | Example source expression in the event | Effect |
+| --- | --- | --- |
+| `JOIN` | `lineage_orders.customer_id = lineage_customers.customer_id` |
Determines how orders match customers. |
+| `FILTER` | `status = 'PAID'`, `customer_level = 'VIP'`, and `SUM(amount) >=
50` | The `WHERE` and `HAVING` predicates jointly determine which rows and
groups enter the result. |
+| `GROUP_BY` | `lineage_orders.customer_id` and `UPPER(lineage_orders.region)`
| Determine the aggregation granularity for each customer and region. |
+| `SORT` | `total_amount DESC`, resolved to `SUM(lineage_orders.amount)` |
Determines the order of the Insert query result. |
+
+#### Output indirect lineage
+
+Output indirect lineage is attached only to the affected target column and
does not apply to the other output columns.
+
+| Type | Meaning | Example |
+| --- | --- | --- |
+| `WINDOW` | The `PARTITION BY` and `ORDER BY` inputs of a window function
affect its output column. | `customer_seq` is produced by `ROW_NUMBER() OVER
(ORDER BY customer_id)`, whose `WINDOW` dependency is `customer_id`. |
+| `CONDITIONAL` | A `CASE` or `IF` condition, or the candidate expressions of
`COALESCE`, affect its output column. | `region_group` is determined by `CASE
WHEN region_label = 'EAST' ...`, whose `CONDITIONAL` dependency is that
predicate. |
+
+The window or conditional expression itself still appears in direct column
lineage. For example, the direct type of `customer_seq` is `TRANSFORMATION`,
and it also has `WINDOW` indirect lineage. The direct type of `region_group` is
also `TRANSFORMATION`, and it also has `CONDITIONAL` indirect lineage.
+
+This example produces the following output indirect lineage:
+
+| Target column | Type | Source expression in the event | Effect |
+| --- | --- | --- | --- |
+| `customer_seq` | `WINDOW` | `lineage_orders.customer_id` | Serves as the
window ordering input to `ROW_NUMBER()` and affects only `customer_seq`. |
+| `region_group` | `CONDITIONAL` | `UPPER(lineage_orders.region) = 'EAST'` |
Serves as the `CASE WHEN` condition and affects only `region_group`. |
+
+The other three target columns have no output indirect lineage. Dataset-level
`JOIN`, `FILTER`, `GROUP_BY`, and `SORT` dependencies still affect the complete
result set.
+
+#### Query context
+
+Query context identifies who produced the lineage event, in which session, and
through which DML statement.
+
+| Field | Example or meaning |
+| --- | --- |
+| Source Command | `InsertIntoTableCommand`, indicating that the event comes
from an Insert. |
+| Query ID and original SQL | The Query ID and the complete `INSERT INTO
lineage_customer_summary ...` text. |
+| User and client IP | For example, `lineage_user` and `192.0.2.10`. |
+| Session database and catalog | `lineage_demo` and `internal` in this
example. They describe session context and do not necessarily identify the
target-table location. |
+| Execution state | A successful DML statement normally records `OK`. |
+| Timestamp and duration | The event creation time and the milliseconds from
query start to event creation. The value is `-1` when unavailable. |
+| External catalog properties | Non-sensitive properties from external
catalogs referenced by the query. Passwords, keys, and hidden properties are
removed. The map is empty when only the `internal` catalog is used. |
+
+For this example, the original SQL is the complex Insert above, the Source
Command is `InsertIntoTableCommand`, the session database is `lineage_demo`,
the session catalog is `internal`, and the successful state is `OK`. The Query
ID, user, client IP, timestamp, and duration use values from the actual
execution context. Because the example uses only the Internal Catalog, the
external catalog properties map is empty.
+
+Confirm in the plugin's downstream system that the event was received. During
initial deployment, also check `fe.log` for `Loaded lineage plugin`,
plugin-specific errors, and queue-full warnings.
+
+
+
+The extractor resolves CTE producer expressions and expands `UNION` branches
before the event is delivered. A downstream plugin must convert Doris Java
objects such as `TableIf`, `SlotReference`, and `Expression` to stable
identifiers or its own event schema.
+
+### Event processing behavior
+
+`lineage_event_queue_size` sets the maximum number of lineage events waiting
in each FE-local queue. It counts events, not bytes. When the queue is full,
the new event is discarded and the DML continues normally. The worker invokes
plugins serially, so a slow plugin delays later events and can cause drops
under sustained load. Exceptions thrown by a plugin are logged and do not stop
the worker or affect the DML.
+
+Plugin discovery and initialization occur only when FE starts. Dynamic reload
and unload are not supported.
+
+## Configure and use
+
+### Prerequisites
+
+1. Prepare an external lineage plugin. Doris does not ship a built-in sink.
For the SPI contract, JAR packaging, and a complete minimal plugin, see [Data
Lineage Plugin
Development](/community/developer-guide/data-lineage-plugin-development).
+2. Copy the plugin JAR and any required third-party dependency JARs to every
FE.
+3. If the plugin sends events to an external governance service, make sure
that service is reachable from every FE that can execute the DML.
+
+### Deploy the plugin
+
+`FE_HOME` denotes a single FE installation directory that contains `bin/`,
`conf/`, and `lib/`. Use the following layout on every FE. The loader scans
only direct child directories under `lineage/`, plus JARs in each plugin
directory and its `lib/` directory.
+
+```text
+$FE_HOME/plugins/
+└── lineage/
+ └── example-lineage/
+ ├── example-lineage-plugin.jar
+ └── lib/
+ └── downstream-client.jar
+```
+
+Configure `fe.conf` on every FE. Replace `example-lineage` with the value
returned by the plugin factory's `name()` method:
+
Review Comment:
The JAR filename `example-lineage-plugin.jar` is inconsistent with the
developer guide
(`community/developer-guide/data-lineage-plugin-development.md`), where the
Maven `<finalName>` produces `example-lineage.jar`. Users who follow both
guides may be confused about which JAR name is correct. Please align the JAR
name across both documents — either change this to `example-lineage.jar` or
update the dev guide's `pom.xml` `<finalName>` and build output references.
##########
community/developer-guide/data-lineage-plugin-development.md:
##########
@@ -0,0 +1,604 @@
+---
+title: Data Lineage Plugin Development
+language: en
+description: Build, package, deploy, and verify an Apache Doris data lineage
plugin that logs lineage events or forwards them to an external governance
system.
+keywords:
+ - Apache Doris
+ - data lineage
+ - LineagePlugin
+ - LineagePluginFactory
+ - ServiceLoader
+ - plugin development
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<!-- Knowledge type: Extension development -->
+<!-- Applicable scenario: Data governance integration / Kernel extension
development -->
+
+# Data Lineage Plugin Development
+
+This document describes how to develop an external data lineage plugin for
Apache Doris. The plugin receives a `LineageInfo` event after a supported DML
statement succeeds, then converts the event and writes it to logs, a messaging
system, or an external metadata governance platform.
+
+> Applicable version: Apache Doris community edition 4.0.6 and later. Compile
the plugin against the exact Doris community release or source revision
deployed on the FE.
+
+## Scope and lifecycle
+
+The lineage framework is an FE-side SPI. It does not store lineage data,
provide a query API, or render a lineage graph. A plugin converts the in-memory
Java objects in `LineageInfo` to the format and transport protocol required by
its downstream system.
+
+The framework generates events only for successful `INSERT INTO ... SELECT`,
`INSERT OVERWRITE ... SELECT`, and `CREATE TABLE AS SELECT` statements.
`VALUES`-only writes and writes whose target is `__internal_schema` are
skipped. `SELECT`, `UPDATE`, `DELETE`, and load jobs are outside the scope of
this framework.
+
+
+
+`eventFilter()` is called once before lineage extraction on the DML query path
and again before worker dispatch. Return `false` when the plugin should not
receive events. The method is called concurrently from query threads and the
worker thread, so it must be thread-safe. `exec()` is called by one worker
thread, but can run concurrently with `eventFilter()`.
+
+At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE
then calls the Factory's `create(context)` method and calls
`initialize(context)` on the returned plugin. The default
`LineagePluginFactory.create(context)` implementation delegates to the
no-argument `create()`, so the minimal example in this guide implements only
`create()`. `PluginContext` contains `plugin.name` and `plugin.path`. A
production plugin that reads a configuration file from its plugin directory can
override `initialize(context)`, locate the file through `plugin.path`, and
initialize its resources there.
Review Comment:
Same `INSERT OVERWRITE` issue as the user guide — the `TABLE` keyword is
missing. Standard Doris syntax is `INSERT OVERWRITE TABLE ... SELECT ...`. The
command class name `InsertOverwriteTableCommand` also confirms this. Please use
the precise syntax throughout (applies to all files: EN user guide, ZH user
guide, ZH dev guide, and all versioned copies).
##########
docs/sql-manual/sql-statements/data-governance/data-lineage.md:
##########
@@ -0,0 +1,336 @@
+---
+{
+ "title": "Data Lineage Technical Guide",
+ "sidebar_label": "Data Lineage",
+ "language": "en",
+ "description": "Collect table-level and column-level lineage from
supported Doris DML statements through an external LineagePlugin.",
+ "keywords": ["Apache Doris", "data lineage", "data governance",
"LineagePlugin", "column-level lineage"]
+}
+---
+
+<!-- Knowledge type: Feature overview + technical design + user guide -->
+<!-- Use cases: Data governance integration / impact analysis / compliance
traceability -->
+
+Data Lineage extracts table-level and column-level dependencies from supported
Doris DML statements and sends them to external governance systems through a
`LineagePlugin`. Doris is the lineage producer. It does not retain events,
expose a lineage query API, or provide lineage visualization.
+
+> This capability is available in the Dev version. It was first released in
Apache Doris 4.0.6.
+
+## Capabilities and limitations
+
+The framework generates an event only after one of the following statements
succeeds:
+
+| Supported statement | Event behavior |
+| --- | --- |
+| `INSERT INTO ... SELECT` | Generates one event after the Insert succeeds. |
+| `INSERT OVERWRITE ... SELECT` | Generates one event after the Overwrite
succeeds. |
+| `CREATE TABLE AS SELECT` | Generates one event after the internal Insert
succeeds. |
+
+The following operations do not generate an event: `SELECT`, `UPDATE`,
`DELETE`, load jobs, `VALUES`-only inserts, and writes whose target is
`__internal_schema`.
+
+:::caution Delivery guarantee
+
+Lineage delivery is asynchronous and best effort. A successful DML statement
is not rolled back when event collection or plugin delivery fails. Events can
be discarded when the queue is full, and Doris does not retry or persist them.
+
+:::
+
+## Technical design
+
+### Collection flow
+
+For a supported statement, Doris records the analyzed Nereids logical plan in
an `afterAnalyze` planner hook. After the DML succeeds, Doris extracts lineage
from that plan and submits the event to an FE-local queue. A single daemon
worker dispatches the event to every active plugin.
+
+
+
+Before extraction, Doris calls `eventFilter()` on the loaded plugins. If no
plugin is willing to receive events, extraction is skipped. The worker
evaluates `eventFilter()` again before dispatch.
+
+### Lineage content
+
+Each `LineageInfo` event contains table lineage, direct column lineage, two
kinds of indirect lineage, and query context. The complete SQL example below
establishes the source tables, target table, and result used by the following
sections.
+
+#### Example SQL and result
+
+The current user must have privileges to create a database and tables, write
data, and query data, and the FE must have loaded a lineage plugin. The example
uses two source tables and creates a customer summary through a CTE, Join,
filters, aggregation, a window function, a conditional expression, and sorting.
+
+```sql
+CREATE DATABASE IF NOT EXISTS lineage_demo;
+USE lineage_demo;
+
+CREATE TABLE lineage_orders (
+ order_id BIGINT,
+ customer_id BIGINT,
+ region VARCHAR(16),
+ amount DECIMAL(18, 2),
+ status VARCHAR(16)
+)
+DUPLICATE KEY(order_id)
+DISTRIBUTED BY HASH(order_id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+
+CREATE TABLE lineage_customers (
+ customer_id BIGINT,
+ customer_name VARCHAR(64),
+ customer_level VARCHAR(16)
+)
+DUPLICATE KEY(customer_id)
+DISTRIBUTED BY HASH(customer_id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+
+CREATE TABLE lineage_customer_summary (
+ customer_id BIGINT,
+ region_label VARCHAR(16),
+ total_amount DECIMAL(18, 2),
+ customer_seq BIGINT,
+ region_group VARCHAR(16)
+)
+DUPLICATE KEY(customer_id)
+DISTRIBUTED BY HASH(customer_id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+
+INSERT INTO lineage_customers VALUES
+ (1, 'Alice', 'VIP'),
+ (2, 'Bob', 'VIP'),
+ (3, 'Carol', 'REGULAR');
+
+INSERT INTO lineage_orders VALUES
+ (101, 1, 'east', 100.00, 'PAID'),
+ (102, 1, 'east', 50.00, 'PAID'),
+ (103, 2, 'west', 80.00, 'PAID'),
+ (104, 2, 'west', 30.00, 'CANCELLED'),
+ (105, 3, 'north', 200.00, 'PAID');
+
+INSERT INTO lineage_customer_summary
+WITH customer_totals AS (
+ SELECT
+ o.customer_id,
+ UPPER(o.region) AS region_label,
+ SUM(o.amount) AS total_amount
+ FROM lineage_orders o
+ JOIN lineage_customers c
+ ON o.customer_id = c.customer_id
+ WHERE o.status = 'PAID'
+ AND c.customer_level = 'VIP'
+ GROUP BY o.customer_id, UPPER(o.region)
+ HAVING SUM(o.amount) >= 50
+)
+SELECT
+ customer_id,
+ region_label,
+ total_amount,
+ ROW_NUMBER() OVER (ORDER BY customer_id) AS customer_seq,
+ CASE
+ WHEN region_label = 'EAST' THEN 'CORE'
+ ELSE 'OTHER'
+ END AS region_group
+FROM customer_totals
+ORDER BY total_amount DESC;
+
+SELECT customer_id, region_label, total_amount, customer_seq, region_group
+FROM lineage_customer_summary
+ORDER BY customer_id;
+```
+
+The query returns:
+
+```text
++-------------+--------------+--------------+--------------+--------------+
+| customer_id | region_label | total_amount | customer_seq | region_group |
++-------------+--------------+--------------+--------------+--------------+
+| 1 | EAST | 150.00 | 1 | CORE |
+| 2 | WEST | 80.00 | 2 | OTHER |
++-------------+--------------+--------------+--------------+--------------+
+```
+
+The two `VALUES`-only inserts do not generate lineage events. The `INSERT INTO
lineage_customer_summary ... SELECT` statement generates one `LineageInfo`
event, whose contents are described below.
+
+#### Table lineage
+
+Table lineage records one target table and every source table scanned by the
analyzed logical plan. The query that writes `lineage_customer_summary` scans
both `lineage_orders` and `lineage_customers`, so both source tables have a
table-level relationship with the target. A CTE is resolved to its underlying
tables, and every `UNION` branch contributes to the source-table set.
+
+This example produces the following table lineage:
+
+| Source table | Target table |
+| --- | --- |
+| `lineage_orders` | `lineage_customer_summary` |
+| `lineage_customers` | `lineage_customer_summary` |
+
+#### Direct column lineage
+
+Direct column lineage describes which source expression produces each
target-column value. After resolving aliases and CTEs, Doris classifies
expressions in this order: an expression containing an aggregate function is
`AGGREGATION`; a plain source-column reference is `IDENTITY`; every other
expression is `TRANSFORMATION`.
+
+| Type | Meaning | Example |
+| --- | --- | --- |
+| `IDENTITY` | The target value is taken directly from a source column without
a function or operation. | `lineage_customer_summary.customer_id` comes
directly from `lineage_orders.customer_id`. |
+| `TRANSFORMATION` | A non-aggregate expression transforms the target value,
such as arithmetic, a string function, a window function, or a conditional
expression. | `region_label` comes from `UPPER(lineage_orders.region)`;
`customer_seq` comes from `ROW_NUMBER()`. |
+| `AGGREGATION` | The target value is produced by an expression that contains
an aggregate function. It remains this type even when another expression wraps
the aggregate. | `total_amount` comes from `SUM(lineage_orders.amount)`. |
+
+This example produces the following direct column lineage:
+
+| Target column | Source expression | Type |
+| --- | --- | --- |
+| `lineage_customer_summary.customer_id` | `lineage_orders.customer_id` |
`IDENTITY` |
+| `lineage_customer_summary.region_label` | `UPPER(lineage_orders.region)` |
`TRANSFORMATION` |
+| `lineage_customer_summary.total_amount` | `SUM(lineage_orders.amount)` |
`AGGREGATION` |
+| `lineage_customer_summary.customer_seq` | `ROW_NUMBER() OVER (ORDER BY
lineage_orders.customer_id)` | `TRANSFORMATION` |
+| `lineage_customer_summary.region_group` | `CASE WHEN
UPPER(lineage_orders.region) = 'EAST' ... END` | `TRANSFORMATION` |
+
+#### Dataset indirect lineage
+
+Dataset indirect lineage records expressions that affect the complete result
set without directly producing a target-column value. These dependencies are
stored in an event-level map and can be applied to every target output column
when needed.
+
+| Type | Meaning | Example |
+| --- | --- | --- |
+| `JOIN` | Join conditions determine which source rows can be combined. |
`lineage_orders.customer_id = lineage_customers.customer_id`. |
+| `FILTER` | `WHERE` or `HAVING` conditions determine which rows or groups
enter the result. | `status = 'PAID'`, `customer_level = 'VIP'`, and `HAVING
SUM(amount) >= 50`. |
+| `GROUP_BY` | Grouping expressions determine the granularity of aggregate
results. | Group by `customer_id` and `UPPER(region)`. |
+| `SORT` | `ORDER BY` or TopN expressions determine result order. | Sort by
`total_amount DESC`, which resolves to `SUM(amount)`. |
+
+This example produces the following dataset indirect lineage:
+
+| Type | Example source expression in the event | Effect |
+| --- | --- | --- |
+| `JOIN` | `lineage_orders.customer_id = lineage_customers.customer_id` |
Determines how orders match customers. |
+| `FILTER` | `status = 'PAID'`, `customer_level = 'VIP'`, and `SUM(amount) >=
50` | The `WHERE` and `HAVING` predicates jointly determine which rows and
groups enter the result. |
+| `GROUP_BY` | `lineage_orders.customer_id` and `UPPER(lineage_orders.region)`
| Determine the aggregation granularity for each customer and region. |
+| `SORT` | `total_amount DESC`, resolved to `SUM(lineage_orders.amount)` |
Determines the order of the Insert query result. |
+
+#### Output indirect lineage
+
+Output indirect lineage is attached only to the affected target column and
does not apply to the other output columns.
+
+| Type | Meaning | Example |
+| --- | --- | --- |
+| `WINDOW` | The `PARTITION BY` and `ORDER BY` inputs of a window function
affect its output column. | `customer_seq` is produced by `ROW_NUMBER() OVER
(ORDER BY customer_id)`, whose `WINDOW` dependency is `customer_id`. |
+| `CONDITIONAL` | A `CASE` or `IF` condition, or the candidate expressions of
`COALESCE`, affect its output column. | `region_group` is determined by `CASE
WHEN region_label = 'EAST' ...`, whose `CONDITIONAL` dependency is that
predicate. |
+
+The window or conditional expression itself still appears in direct column
lineage. For example, the direct type of `customer_seq` is `TRANSFORMATION`,
and it also has `WINDOW` indirect lineage. The direct type of `region_group` is
also `TRANSFORMATION`, and it also has `CONDITIONAL` indirect lineage.
+
+This example produces the following output indirect lineage:
+
+| Target column | Type | Source expression in the event | Effect |
+| --- | --- | --- | --- |
+| `customer_seq` | `WINDOW` | `lineage_orders.customer_id` | Serves as the
window ordering input to `ROW_NUMBER()` and affects only `customer_seq`. |
+| `region_group` | `CONDITIONAL` | `UPPER(lineage_orders.region) = 'EAST'` |
Serves as the `CASE WHEN` condition and affects only `region_group`. |
+
+The other three target columns have no output indirect lineage. Dataset-level
`JOIN`, `FILTER`, `GROUP_BY`, and `SORT` dependencies still affect the complete
result set.
+
+#### Query context
+
+Query context identifies who produced the lineage event, in which session, and
through which DML statement.
+
+| Field | Example or meaning |
+| --- | --- |
+| Source Command | `InsertIntoTableCommand`, indicating that the event comes
from an Insert. |
+| Query ID and original SQL | The Query ID and the complete `INSERT INTO
lineage_customer_summary ...` text. |
+| User and client IP | For example, `lineage_user` and `192.0.2.10`. |
+| Session database and catalog | `lineage_demo` and `internal` in this
example. They describe session context and do not necessarily identify the
target-table location. |
+| Execution state | A successful DML statement normally records `OK`. |
+| Timestamp and duration | The event creation time and the milliseconds from
query start to event creation. The value is `-1` when unavailable. |
+| External catalog properties | Non-sensitive properties from external
catalogs referenced by the query. Passwords, keys, and hidden properties are
removed. The map is empty when only the `internal` catalog is used. |
+
+For this example, the original SQL is the complex Insert above, the Source
Command is `InsertIntoTableCommand`, the session database is `lineage_demo`,
the session catalog is `internal`, and the successful state is `OK`. The Query
ID, user, client IP, timestamp, and duration use values from the actual
execution context. Because the example uses only the Internal Catalog, the
external catalog properties map is empty.
+
+Confirm in the plugin's downstream system that the event was received. During
initial deployment, also check `fe.log` for `Loaded lineage plugin`,
plugin-specific errors, and queue-full warnings.
+
+
+
+The extractor resolves CTE producer expressions and expands `UNION` branches
before the event is delivered. A downstream plugin must convert Doris Java
objects such as `TableIf`, `SlotReference`, and `Expression` to stable
identifiers or its own event schema.
+
+### Event processing behavior
+
+`lineage_event_queue_size` sets the maximum number of lineage events waiting
in each FE-local queue. It counts events, not bytes. When the queue is full,
the new event is discarded and the DML continues normally. The worker invokes
plugins serially, so a slow plugin delays later events and can cause drops
under sustained load. Exceptions thrown by a plugin are logged and do not stop
the worker or affect the DML.
+
+Plugin discovery and initialization occur only when FE starts. Dynamic reload
and unload are not supported.
+
+## Configure and use
+
+### Prerequisites
+
+1. Prepare an external lineage plugin. Doris does not ship a built-in sink.
For the SPI contract, JAR packaging, and a complete minimal plugin, see [Data
Lineage Plugin
Development](/community/developer-guide/data-lineage-plugin-development).
+2. Copy the plugin JAR and any required third-party dependency JARs to every
FE.
+3. If the plugin sends events to an external governance service, make sure
that service is reachable from every FE that can execute the DML.
+
+### Deploy the plugin
+
+`FE_HOME` denotes a single FE installation directory that contains `bin/`,
`conf/`, and `lib/`. Use the following layout on every FE. The loader scans
only direct child directories under `lineage/`, plus JARs in each plugin
directory and its `lib/` directory.
+
+```text
+$FE_HOME/plugins/
+└── lineage/
+ └── example-lineage/
+ ├── example-lineage-plugin.jar
+ └── lib/
+ └── downstream-client.jar
+```
+
+Configure `fe.conf` on every FE. Replace `example-lineage` with the value
returned by the plugin factory's `name()` method:
+
+```text
+plugin_dir = /opt/apache-doris/fe/plugins
+activate_lineage_plugin = example-lineage
+lineage_event_queue_size = 50000
+```
+
+| Configuration | Type and default | Required | Description |
+| --- | --- | --- | --- |
+| `plugin_dir` | String; `$FE_HOME/plugins` | No | Plugin root directory. The
lineage loader scans its direct children under `lineage/`. Omit this setting
when using the default directory. |
+| `activate_lineage_plugin` | String array; empty | Explicit configuration
recommended | Comma-separated Factory names to instantiate. An empty value
disables name filtering and instantiates all discovered factories. |
+| `lineage_event_queue_size` | Positive integer; `50000` | No | Maximum
pending lineage events on the current FE. New events are discarded when the
queue is full. |
+
+All three settings are configured in `$FE_HOME/conf/fe.conf` on each FE and
apply only to that FE process. They are read only at FE startup, so restart the
corresponding FE after changing them.
+
+#### Activate plugins
+
+Configure `activate_lineage_plugin` in `$FE_HOME/conf/fe.conf` on every FE. It
controls which discovered factories are instantiated. It does not discover JARs
or control queue capacity. Each name is case-sensitive and must exactly match
`LineagePluginFactory.name()`. The `LineagePlugin.name()` implementation should
return the same name. The plugin directory name is not used for matching.
Separate multiple plugin names with commas; FE trims surrounding spaces:
+
+```text
+activate_lineage_plugin = example-lineage,governance-lineage
+```
+
+At startup, FE discovers built-in and external factories, then creates
instances only for the configured names. Each FE reads its own configuration,
so every FE that can execute DML must use the same plugin set. Restart FE after
a change. This setting cannot be changed dynamically through `ADMIN SET
FRONTEND CONFIG`.
Review Comment:
This paragraph says: "Separate multiple plugin names with commas; FE trims
surrounding spaces." Looking at the code, `Config.activate_lineage_plugin` is a
`String[]`, and `discoverPlugins()` uses
`Arrays.asList(Config.activate_lineage_plugin)` followed by `new
HashSet<>(...)`. The raw `Arrays.asList()` does NOT trim whitespace from
individual elements. Unless the Doris Config framework's String[] parser does
the trimming (please verify), this claim may be incorrect. If a user writes
`activate_lineage_plugin = example-lineage, governance-lineage`, the second
entry would be `" governance-lineage"` (with leading space) and would fail name
matching because the comparison is case-sensitive `HashSet.contains()`.
##########
static/images/data-lineage/lineage-architecture.svg:
##########
@@ -0,0 +1,65 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="1440" height="620" viewBox="0 0
1440 620" role="img" aria-labelledby="title desc">
+ <title id="title">Apache Doris data lineage collection architecture</title>
+ <desc id="desc">Supported DML is analyzed by Nereids and, after successful
execution, produces a LineageInfo event. The event enters a bounded FE queue,
is processed by one worker, and is sent by a lineage plugin to an external
governance system.</desc>
+ <defs>
+ <marker id="arrow" markerWidth="10" markerHeight="10" refX="9" refY="5"
orient="auto"><path d="M0,0 L10,5 L0,10 z" fill="#4b6472"/></marker>
+ <style>
+ .title { font: 700 28px Arial, sans-serif; fill: #173b4d; }
+ .label { font: 700 18px Arial, sans-serif; fill: #173b4d; }
+ .body { font: 16px Arial, sans-serif; fill: #3b5360; }
+ .small { font: 14px Arial, sans-serif; fill: #526b77; }
+ .edge { fill: none; stroke: #4b6472; stroke-width: 3; marker-end:
url(#arrow); }
+ .edge-label { font: 14px Arial, sans-serif; fill: #526b77; }
+ </style>
+ </defs>
+ <rect width="1440" height="620" fill="#f7faf9"/>
+ <text x="72" y="70" class="title">Data Lineage Collection and Delivery</text>
+ <text x="72" y="102" class="body">Collection occurs after a supported DML
statement succeeds. Delivery is asynchronous and best effort.</text>
+
+ <rect x="72" y="186" width="230" height="174" rx="8" fill="#e7f4f1"
stroke="#2f8f83" stroke-width="2"/>
+ <text x="96" y="226" class="label">Supported DML</text>
+ <text x="96" y="262" class="body">INSERT INTO SELECT</text>
+ <text x="96" y="290" class="body">INSERT OVERWRITE SELECT</text>
+ <text x="96" y="318" class="body">CREATE TABLE AS SELECT</text>
Review Comment:
The SVG text uses `INSERT OVERWRITE SELECT` without the `TABLE` keyword. The
standard Doris SQL syntax is `INSERT OVERWRITE TABLE ... SELECT ...`. Please
add `TABLE` to match the actual syntax. Affects both `lineage-architecture.svg`
and its zh-CN variant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
