AlexRiedler opened a new pull request, #619:
URL: https://github.com/apache/doris-flink-connector/pull/619
# Proposed changes
<!-- Issue Number: close #xxx -->
BUGFIX: Since arrow do not have a line delimiter, this buffer merging
operation results in a null pointer exception.
```
Caused by: java.lang.NullPointerException: Cannot read the array length
because the return value of
"org.apache.doris.flink.sink.batch.BatchRecordBuffer.getLineDelimiter()" is null
at
org.apache.doris.flink.sink.batch.DorisBatchStreamLoad.merge(DorisBatchStreamLoad.java:386)
at
org.apache.doris.flink.sink.batch.DorisBatchStreamLoad.mergeBuffer(DorisBatchStreamLoad.java:366)
at
org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.run(DorisBatchStreamLoad.java:429)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
```
We can see in the `DorisBatchStreamLoad(...)` constructor that:
```
if (loadProps.getProperty(FORMAT_KEY, CSV).equals(ARROW)) {
this.lineDelimiter = null;
} else {
this.lineDelimiter =
EscapeHandler.escapeString(
loadProps.getProperty(
LINE_DELIMITER_KEY,
LINE_DELIMITER_DEFAULT))
.getBytes();
}
```
which is used when building BatchRecordBuffer's in the class.
the delimiter is explicitly set to null in this case because Arrow does not
have delimiters as it encodes a length field.
## Problem Summary:
Null pointer exception when using arrow for batch-stream-loading during
buffer merge operation.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
Yes
2. Has unit tests been added: (Yes/No/No Need)
Yes (although might need higher level somewhere else for arrow)
3. Has document been added or modified: (Yes/No/No Need)
No Need
4. Does it need to update dependencies: (Yes/No)
No
5. Are there any changes that cannot be rolled back: (Yes/No)
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
