zhztheplayer commented on code in PR #5952:
URL: https://github.com/apache/incubator-gluten/pull/5952#discussion_r1689359964
##########
cpp/core/jni/JniWrapper.cc:
##########
@@ -530,16 +533,36 @@
Java_org_apache_gluten_vectorized_NativeColumnarToRowJniWrapper_nativeColumnarTo
JNIEnv* env,
jobject wrapper,
jlong c2rHandle,
- jlong batchHandle) {
+ jlong batchHandle,
+ jlong rowId) {
JNI_METHOD_START
auto columnarToRowConverter =
ObjectStore::retrieve<ColumnarToRowConverter>(c2rHandle);
auto cb = ObjectStore::retrieve<ColumnarBatch>(batchHandle);
columnarToRowConverter->convert(cb);
+ auto ctx = gluten::getRuntime(env, wrapper);
+
+ auto& conf = ctx->getConfMap();
+ int64_t column2RowMemThreshold;
+ auto it = conf.find(kColumnToRowMemoryThreshold);
+ bool confIsLeagal =
+ ((it == conf.end()) ? false : std::all_of(it->second.begin(),
it->second.end(), [](unsigned char c) {
+ return std::isdigit(c);
+ }));
+ if (confIsLeagal) {
+ column2RowMemThreshold = std::stoll(it->second);
+ } else {
+ LOG(INFO)
+ << "Because the spark.gluten.sql.columnToRowMemoryThreshold
configuration item is invalid, the kColumnToRowMemoryDefaultThreshold default
value is used, which is "
+ << kColumnToRowMemoryDefaultThreshold << " byte";
+ column2RowMemThreshold = std::stoll(kColumnToRowMemoryDefaultThreshold);
+ }
+
+ columnarToRowConverter->convert(cb, rowId, column2RowMemThreshold);
Review Comment:
The operations could become expensive given that the JNI function is called
for every single row?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
