Loveca commented on issue #5042:
URL: https://github.com/apache/inlong/issues/5042#issuecomment-1198030907
# Motivition
Add Apache Doris Extract Node for Agent
# About Doris Data Export
Data Export is a function provided by Doris to Export data. This function
exports data in a table or partition specified by users to a remote storage
system, such as HDFS/BOS, in text format through the Broker process. You can
also export it locally.
## Export To HDFS
```
EXPORT TABLE db1.tbl1
PARTITION (p1,p2)
[WHERE [expr]]
TO "hdfs://host/path/to/export/"
PROPERTIES
(
"label" = "mylabel",
"column_separator"=",",
"columns" = "col1,col2",
"exec_mem_limit"="2147483648",
"timeout" = "3600"
)
WITH BROKER "hdfs"
(
"username" = "user",
"password" = "passwd"
);
```
## Export To Local
```
EXPORT TABEL tablename TO "file:///local_file_path"
```
# Design
1.Doris exports the data to the HDFS data file or local data file through
Export command
2. The InLong Agent reads the corresponding data file
# Implementation
- Reader: Implements DorisReader to read the data files exported by Doris
- Source: Implements DorisSource, implements Split logic, and returns Reader
list
- Sink: Use ProxySink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
