HappenLee opened a new issue #5764:
URL: https://github.com/apache/incubator-doris/issues/5764
## Motivation
Currently, Doris support insert data to External Table of ODBC table.
but it's too slow to wait time write to External Table of ODBC table, So we
should speed up it to make it more useful.
## Implementation
### 1. Batch Insert
Now, not only ODBC table or MySQL table insert data to External Table tuple
by tuple, which cause too many rpc between Doris and RDMS. If we use Batch
insert, which will speed up the bulk data load of ODBC table, **but there need
some memory to buffer the intermediate data.**
After change the insert row by one insert stmt to many row by one insert
stmt，insert 6W5 rows tuple to ODBC table
| | one tuple | batch tuple |
| ---- | ---- | ---- |
| ResultSendTime | 1min52s | 2s482ms |
### 2 More Effective Serialization Of Insert Stmt
like issue#5554
use `fmt` to replace `stringstream`, insert 6W5 rows tuple to ODBC table
| | stringstream | fmt |
| ---- | ---- | ---- |
| TupleConvertTime | 345ms | 160ms |
Speed Up 2x
### 3. More Profile
Add thress new profile couter for `ODBC_TABLE_SINK`
- ResultSendTime: rpc procss use in odbc driver
- TupleConvertTime: insert stmt serialization
- NumSentRows: num of rows process by `ODBC_TABLE_SINK`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
