If we introduce Netty, data copy when scaling a bytebuf is not what we want. Can we use compositeByteBuf to replace it and meanwhile enjoy the benefit of pooledByteBuf?
----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao <qiaojia...@apache.org> 于2022年5月16日周一 12:35写道： > Hi, > > The serialization interface needs to be refactored afterward. > > Before that, using ByteArrayOutputStream is easier. > > Thanks, > ————————————————— > Jialin Qiao > Apache IoTDB PMC > > > 李思佳 <lisi...@360.cn> 于2022年5月16日周一 11:44写道： > > > Hi all, > > > > When I was developing the snapshot interface for the configNode module, I > > noticed that the parameters received by the serialization interface were > > all defined as ByteBuffer, which seemed to have some problems. For > example, > > the external main process has no way of knowing how big the buffer will > be. > > We can only estimate a large value to allocate memory. > > > > Then I looked at the serialization interfaces of other modules, and it > > seemed that most modules did the same thing. This could be a problem once > > the actual size of the buffer exceeds our estimate. So I did a quick > survey > > of Netty's byteBuf last week, and here's the Chinese version of the > results< > > https://apache-iotdb.feishu.cn/docs/doccnW1EFoyLOScys9GTOuaEUbh>. > > > > At the same time, we found that the consensus module also has some > > ByteBuf requirements. But byteBuf doesn't seem to be enough to give us > > precise control over the size of the memory pool, and we may need to wrap > > it if we decide to use it. > > > > Finally, we decided to use stream type instead of byteBuffer in > > configNode for the time being. I will start this work to see if this is > the > > better way this week. If any idea, please let me know. > > > > By the way, Netty’s ByteBuf provides powerful tool operations that we > > will not discard outright, but rather as an option. > > > > BR, > > ----------------------------------- > > Sijia Li > > > > >
