summaryzb commented on code in PR #950:
URL: https://github.com/apache/incubator-uniffle/pull/950#discussion_r1360528164
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java:
##########
@@ -151,25 +154,20 @@ public WriteBufferManager(
this.memorySpillTimeoutSec =
rssConf.get(RssSparkConfig.RSS_MEMORY_SPILL_TIMEOUT);
}
- public List<ShuffleBlockInfo> addRecord(int partitionId, Object key, Object
value) {
- final long start = System.currentTimeMillis();
- arrayOutputStream.reset();
- if (key != null) {
- serializeStream.writeKey(key, ClassTag$.MODULE$.apply(key.getClass()));
- } else {
- serializeStream.writeKey(null, ManifestFactory$.MODULE$.Null());
- }
- if (value != null) {
- serializeStream.writeValue(value,
ClassTag$.MODULE$.apply(value.getClass()));
- } else {
- serializeStream.writeValue(null, ManifestFactory$.MODULE$.Null());
+ /**
+ * add serialized columnar data directly when integrate with gluten
+ */
+ public List<ShuffleBlockInfo> addPartitionData(int partitionId, byte[]
serializedData) {
+ return addPartitionData(partitionId, serializedData, 0, 0L);
Review Comment:
@loukey-lj The record and bytes metric is already counted in gluten side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
