marin-ma commented on code in PR #11083:
URL:
https://github.com/apache/incubator-gluten/pull/11083#discussion_r2541180332
##########
backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala:
##########
@@ -203,7 +205,7 @@ class ColumnarShuffleWriter[K, V](
val columnarBatchHandle =
ColumnarBatches.getNativeHandle(BackendsApiManager.getBackendName,
cb)
val startTime = System.nanoTime()
- shuffleWriterJniWrapper.write(
+ val curBytesWritten = shuffleWriterJniWrapper.write(
Review Comment:
```
JNIEXPORT jlong JNICALL
Java_org_apache_gluten_vectorized_ShuffleWriterJniWrapper_write( // NOLINT
JNIEnv* env,
jobject wrapper,
jlong shuffleWriterHandle,
jint numRows,
jlong batchHandle,
jlong memLimit) {
JNI_METHOD_START
auto shuffleWriter =
ObjectStore::retrieve<ShuffleWriter>(shuffleWriterHandle);
if (!shuffleWriter) {
std::string errorMessage = "Invalid shuffle writer handle " +
std::to_string(shuffleWriterHandle);
throw GlutenException(errorMessage);
}
// The column batch maybe VeloxColumnBatch or
ArrowCStructColumnarBatch(FallbackRangeShuffleWriter)
auto batch = ObjectStore::retrieve<ColumnarBatch>(batchHandle);
auto numBytes = batch->numBytes();
arrowAssertOkOrThrow(shuffleWriter->write(batch, memLimit), "Native write:
shuffle writer failed");
return numBytes;
JNI_METHOD_END(kInvalidObjectHandle)
}
```
Seems like it only returns one input batch size. Am I misunderstanding
something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
