[
https://issues.apache.org/jira/browse/NIFI-12296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiyang updated NIFI-12296:
--------------------------
Description:
A processor was added to write data to Apache Doris as a StreamLoad. Unlike the
others, StreamLoad is a high performance, low latency data import mode, and
supports transactions (controlled by labels). This processor solves several
problems:
1. The CaptureChangeMySQL processor can be used directly to read the data out,
and the data can be written to Doris after simple processing
2. Support large Batch writes (increase Batch Size and Buffer Size)
3. Support dynamic delete operation (if parse to op operation DELETE case)
4. Support the operation of multi-database and multi-table
Deficiencies:
1. Do not implement all the test cases (section have been implemented, but CI |
CD after the merger will have conflict, there is no solution, need advice or
Suggestions)
2. The data reading operation needs to be reviewed or provided by the
management (not sure whether it is reasonable)
was:
https://github.com/apache/nifi/pull/7959
As shown in the link, the basic functionality is implemented and some
decorative functionality is missing, but I'm not sure if my code is spec
compliant or if something is missing. What should I do next to make this
work.Looking forward to your reply.
best regards
> Write the data to Apache Doris in a StreamLoad fashion
> ------------------------------------------------------
>
> Key: NIFI-12296
> URL: https://issues.apache.org/jira/browse/NIFI-12296
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Reporter: xiyang
> Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> A processor was added to write data to Apache Doris as a StreamLoad. Unlike
> the others, StreamLoad is a high performance, low latency data import mode,
> and supports transactions (controlled by labels). This processor solves
> several problems:
> 1. The CaptureChangeMySQL processor can be used directly to read the data
> out, and the data can be written to Doris after simple processing
> 2. Support large Batch writes (increase Batch Size and Buffer Size)
> 3. Support dynamic delete operation (if parse to op operation DELETE case)
> 4. Support the operation of multi-database and multi-table
> Deficiencies:
> 1. Do not implement all the test cases (section have been implemented, but CI
> | CD after the merger will have conflict, there is no solution, need advice
> or Suggestions)
> 2. The data reading operation needs to be reviewed or provided by the
> management (not sure whether it is reasonable)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
