lgbo-ustc commented on code in PR #8558:
URL: https://github.com/apache/incubator-gluten/pull/8558#discussion_r1952057787
##########
cpp-ch/local-engine/Functions/SparkFunctionGetJsonObject.h:
##########
@@ -679,15 +720,63 @@ class FlattenJSONStringOnRequiredFunction : public
DB::IFunction
return is_doc_ok;
}
+private:
+ DB::ContextPtr context;
+ /// If too many rows cannot be parsed by simdjson directly, we will
normalize the json text at first;
+ mutable bool is_most_normal_json_text = true;
+ mutable size_t total_parsed_rows = 0;
+ mutable size_t total_normalized_rows = 0;
+
+ template<typename JSONParser, typename JSONStringSerializer>
+ void insertResultToColumn(
Review Comment:
Should explain whic case it is for each branch
##########
cpp-ch/local-engine/Functions/SparkFunctionGetJsonObject.h:
##########
@@ -679,15 +720,63 @@ class FlattenJSONStringOnRequiredFunction : public
DB::IFunction
return is_doc_ok;
}
+private:
+ DB::ContextPtr context;
+ /// If too many rows cannot be parsed by simdjson directly, we will
normalize the json text at first;
+ mutable bool is_most_normal_json_text = true;
+ mutable size_t total_parsed_rows = 0;
+ mutable size_t total_normalized_rows = 0;
+
+ template<typename JSONParser, typename JSONStringSerializer>
+ void insertResultToColumn(
Review Comment:
It seems to be complex, I guess there should be a simpler implement with
less branches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
