caiconghui commented on a change in pull request #5774:
URL: https://github.com/apache/incubator-doris/pull/5774#discussion_r632246796
##########
File path: be/src/exec/json_scanner.cpp
##########
@@ -270,27 +339,20 @@ void JsonReader::_close() {
if (_closed) {
return;
}
- if (typeid(*_file_reader) == typeid(doris::BrokerReader) ||
- typeid(*_file_reader) == typeid(doris::LocalFileReader)) {
- _file_reader->close();
- delete _file_reader;
- }
_closed = true;
}
-// read one json string from file read and parse it to json doc.
+// read one json string from line read and parse it to json doc.
// return Status::DataQualityError() if data has quality error.
// return other error if encounter other problemes.
// return Status::OK() if parse succeed or reach EOF.
-Status JsonReader::_parse_json_doc(bool* eof) {
- // read a whole message, must be delete json_str by `delete[]`
+Status JsonReader::_parse_json_doc(size_t* size, bool* eof) {
+ // read a whole message
SCOPED_TIMER(_file_read_timer);
- std::unique_ptr<uint8_t[]> json_str;
- size_t length = 0;
- RETURN_IF_ERROR(_file_reader->read_one_message(&json_str, &length));
- _bytes_read_counter += length;
- if (length == 0) {
- *eof = true;
+ const uint8_t* json_str = nullptr;
+ RETURN_IF_ERROR(_line_reader->read_line(&json_str, size, eof));
Review comment:
In fact, JSON usually serialized format does not contain line breaks.
what you give above is JSON display format. If we encounter it, we will set the
line_delimiter to other value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
