seawinde commented on code in PR #3997: URL: https://github.com/apache/doris-website/pull/3997#discussion_r3628212103
########## static/images/data-lineage/lineage-architecture.svg: ########## @@ -0,0 +1,65 @@ +<svg xmlns="http://www.w3.org/2000/svg"; width="1440" height="620" viewBox="0 0 1440 620" role="img" aria-labelledby="title desc"> + <title id="title">Apache Doris data lineage collection architecture</title> + <desc id="desc">Supported DML is analyzed by Nereids and, after successful execution, produces a LineageInfo event. The event enters a bounded FE queue, is processed by one worker, and is sent by a lineage plugin to an external governance system.</desc> + <defs> + <marker id="arrow" markerWidth="10" markerHeight="10" refX="9" refY="5" orient="auto"><path d="M0,0 L10,5 L0,10 z" fill="#4b6472"/></marker> + <style> + .title { font: 700 28px Arial, sans-serif; fill: #173b4d; } + .label { font: 700 18px Arial, sans-serif; fill: #173b4d; } + .body { font: 16px Arial, sans-serif; fill: #3b5360; } + .small { font: 14px Arial, sans-serif; fill: #526b77; } + .edge { fill: none; stroke: #4b6472; stroke-width: 3; marker-end: url(#arrow); } + .edge-label { font: 14px Arial, sans-serif; fill: #526b77; } + </style> + </defs> + <rect width="1440" height="620" fill="#f7faf9"/> + <text x="72" y="70" class="title">Data Lineage Collection and Delivery</text> + <text x="72" y="102" class="body">Collection occurs after a supported DML statement succeeds. Delivery is asynchronous and best effort.</text> + + <rect x="72" y="186" width="230" height="174" rx="8" fill="#e7f4f1" stroke="#2f8f83" stroke-width="2"/> + <text x="96" y="226" class="label">Supported DML</text> + <text x="96" y="262" class="body">INSERT INTO SELECT</text> + <text x="96" y="290" class="body">INSERT OVERWRITE SELECT</text> + <text x="96" y="318" class="body">CREATE TABLE AS SELECT</text> Review Comment: 已经修复 ########## community/developer-guide/data-lineage-plugin-development.md: ########## @@ -0,0 +1,604 @@ +--- +title: Data Lineage Plugin Development +language: en +description: Build, package, deploy, and verify an Apache Doris data lineage plugin that logs lineage events or forwards them to an external governance system. +keywords: + - Apache Doris + - data lineage + - LineagePlugin + - LineagePluginFactory + - ServiceLoader + - plugin development +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +<!-- Knowledge type: Extension development --> +<!-- Applicable scenario: Data governance integration / Kernel extension development --> + +# Data Lineage Plugin Development + +This document describes how to develop an external data lineage plugin for Apache Doris. The plugin receives a `LineageInfo` event after a supported DML statement succeeds, then converts the event and writes it to logs, a messaging system, or an external metadata governance platform. + +> Applicable version: Apache Doris community edition 4.0.6 and later. Compile the plugin against the exact Doris community release or source revision deployed on the FE. + +## Scope and lifecycle + +The lineage framework is an FE-side SPI. It does not store lineage data, provide a query API, or render a lineage graph. A plugin converts the in-memory Java objects in `LineageInfo` to the format and transport protocol required by its downstream system. + +The framework generates events only for successful `INSERT INTO ... SELECT`, `INSERT OVERWRITE ... SELECT`, and `CREATE TABLE AS SELECT` statements. `VALUES`-only writes and writes whose target is `__internal_schema` are skipped. `SELECT`, `UPDATE`, `DELETE`, and load jobs are outside the scope of this framework. + + + +`eventFilter()` is called once before lineage extraction on the DML query path and again before worker dispatch. Return `false` when the plugin should not receive events. The method is called concurrently from query threads and the worker thread, so it must be thread-safe. `exec()` is called by one worker thread, but can run concurrently with `eventFilter()`. + +At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` contains `plugin.name` and `plugin.path`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through `plugin.path`, and initialize its resources there. Review Comment: 已经修复 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
