[
https://issues.apache.org/jira/browse/NIFI-12296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiyang updated NIFI-12296:
--------------------------
Description:
A processor was added to write data to Apache Doris as a StreamLoad. Unlike the
others, StreamLoad is a high performance, low latency data import mode, and
supports transactions (controlled by labels). This processor solves several
problems:
1. The CaptureChangeMySQL processor can be used directly to read the data out,
and the data can be written to Doris after simple processing
2. Support large Batch writes (increase Batch Size and Buffer Size)
3. Support dynamic delete operation (if parse to op operation DELETE case)
4. Support the operation of multi-database and multi-table
Deficiencies:
1. Do not implement all the test cases (section have been implemented, but CI |
CD after the merger will have conflict, there is no solution, need advice or
Suggestions)
2. The data reading operation needs to be reviewed or provided by the
management (not sure whether it is reasonable)
3. As for the suggestion of using HttpClient part, I may not know much about
WebClientServiceProvider and I need some time to understand this implementation
was:
A processor was added to write data to Apache Doris as a StreamLoad. Unlike the
others, StreamLoad is a high performance, low latency data import mode, and
supports transactions (controlled by labels). This processor solves several
problems:
1. The CaptureChangeMySQL processor can be used directly to read the data out,
and the data can be written to Doris after simple processing
2. Support large Batch writes (increase Batch Size and Buffer Size)
3. Support dynamic delete operation (if parse to op operation DELETE case)
4. Support the operation of multi-database and multi-table
Deficiencies:
1. Do not implement all the test cases (section have been implemented, but CI |
CD after the merger will have conflict, there is no solution, need advice or
Suggestions)
2. The data reading operation needs to be reviewed or provided by the
management (not sure whether it is reasonable)
> Write the data to Apache Doris in a StreamLoad fashion
> ------------------------------------------------------
>
> Key: NIFI-12296
> URL: https://issues.apache.org/jira/browse/NIFI-12296
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Reporter: xiyang
> Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> A processor was added to write data to Apache Doris as a StreamLoad. Unlike
> the others, StreamLoad is a high performance, low latency data import mode,
> and supports transactions (controlled by labels). This processor solves
> several problems:
> 1. The CaptureChangeMySQL processor can be used directly to read the data
> out, and the data can be written to Doris after simple processing
> 2. Support large Batch writes (increase Batch Size and Buffer Size)
> 3. Support dynamic delete operation (if parse to op operation DELETE case)
> 4. Support the operation of multi-database and multi-table
> Deficiencies:
> 1. Do not implement all the test cases (section have been implemented, but CI
> | CD after the merger will have conflict, there is no solution, need advice
> or Suggestions)
> 2. The data reading operation needs to be reviewed or provided by the
> management (not sure whether it is reasonable)
> 3. As for the suggestion of using HttpClient part, I may not know much about
> WebClientServiceProvider and I need some time to understand this
> implementation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
