jerqi commented on code in PR #950:
URL: https://github.com/apache/incubator-uniffle/pull/950#discussion_r1229509127
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java:
##########
@@ -151,25 +154,20 @@ public WriteBufferManager(
this.memorySpillTimeoutSec =
rssConf.get(RssSparkConfig.RSS_MEMORY_SPILL_TIMEOUT);
}
- public List<ShuffleBlockInfo> addRecord(int partitionId, Object key, Object
value) {
- final long start = System.currentTimeMillis();
- arrayOutputStream.reset();
- if (key != null) {
- serializeStream.writeKey(key, ClassTag$.MODULE$.apply(key.getClass()));
- } else {
- serializeStream.writeKey(null, ManifestFactory$.MODULE$.Null());
- }
- if (value != null) {
- serializeStream.writeValue(value,
ClassTag$.MODULE$.apply(value.getClass()));
- } else {
- serializeStream.writeValue(null, ManifestFactory$.MODULE$.Null());
+ /**
+ * add serialized columnar data directly when integrate with gluten
+ */
+ public List<ShuffleBlockInfo> addPartitionData(int partitionId, byte[]
serializedData) {
+ return addPartitionData(partitionId, serializedData, 0, 0L);
+ }
+
+ public List<ShuffleBlockInfo> addPartitionData(
+ int partitionId, byte[] serializedData, int serializedDataLength,
long start) {
+ if (start == 0L) {
+ start = System.currentTimeMillis();
Review Comment:
Could we assign `start` variable in upper method? It's weird to assign a
value here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
