FANNG1 commented on code in PR #6946:
URL: https://github.com/apache/gravitino/pull/6946#discussion_r2046962433
##########
docs/lineage/gravitino-server-lineage.md:
##########
@@ -0,0 +1,56 @@
+---
+title: "Gravitino server Lineage support"
+slug: /lineage/gravitino-server-lineage
+keyword: Gravitino OpenLineage
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Overview
+
+Gravitino server provides a pluginable lineage framework to receive, process,
and sink OpenLineage events. By leveraging this, you could do custom process
for the lineage event and sink to your dedicated systems.
+
+## Lineage Configuration
+
+| Configuration item | Description
| Default value | Required |
Since Version |
+|-----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|----------|---------------|
+| `gravitino.lineage.source` | The name of lineage event
source. The default `http` event source will .
| http | No
| 0.9.0 |
+| `gravitino.lineage.${sourceName}.sourceClass` | The name of the lineage
source class which should implement
`org.apache.gravitino.lineage.source.LineageSource` interface.
| (none)
| No | 0.9.0 |
+| `gravitino.lineage.processorClass` | The name of the lineage
processor class which should implement
`org.apache.gravitino.lineage.processor.LineageProcessor` interface. The
default noop processor will do nothing about the run event. |
`org.apache.gravitino.lineage.processor.NoopProcessor` | No | 0.9.0
|
+| `gravitino.lineage.sinks` | The name of lineage event
sinks.
| log | No
| 0.9.0 |
+| `gravitino.lineage.${sinkName}.sinkClass` | The name of the lineage sink
class which should implement `org.apache.gravitino.lineage.sink.LineageSink`
interface.
| (none) | No
| 0.9.0 |
+| `gravitino.lineage.queueCapacity` | The total capacity of
lineage event queues. If there are multi lineage sinks, the sinks will use an
isolated event queue with the capacity of `gravitino.lineage.queueCapacity` div
the num of sinks. | 10000 | No
| 0.9.0 |
+
+## Lineage http source
+
+Http source provides an endpoint which follows [OpenLineage API
spec](https://openlineage.io/apidocs/openapi/) to receive OpenLineage run
event. The following use example:
+
+```shell
+curl -X POST http://localhost:8090/api/lineage \
+ -i -H 'Content-Type: application/json' \
+ -d '{
+ "eventType": "START",
+ "eventTime": "2023-10-28T19:52:00.001+10:00",
+ "run": {
+ "runId": "0176a8c2-fe01-7439-87e6-56a1a1b4029f"
+ },
+ "job": {
+ "namespace": "gravitino-namespace",
+ "name": "gravitino-job1"
+ },
+ "inputs": [{
+ "namespace": "gravitino-namespace",
+ "name": "gravitino-table-identifier"
+ }],
+ "producer":
"https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client";,
+ "schemaURL":
"https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent";
+ }'
Review Comment:
Could you tell me the benefit of writing the json data to file first, the
former solution seems more simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
