Re: About replacing byteBuffer

[李思佳]

In fact, we can do that. There's no copy if bytebuf passed to the 
compositeByteBuf are wrapped by ByteBuffer.

The problem is that we might not force every module to do this(may be checked 
only by review code). If anyone uses Netty's byteBuf directly and does not 
manage memory properly(and we don't have good surveillance to find this 
problem), this could result in a memory leak.  

Maybe, how to make better use of Netty, provide more monitoring capabilities, 
and constrain the development standards of all modules, this may need further 
consideration.  

BR,
---
Sijia Li

-邮件原件-
发件人: Xiangdong Huang  
发送时间: 2022年5月17日 8:57
收件人: dev 
主题:Re: About replacing byteBuffer

If we introduce Netty, data copy when scaling a bytebuf is not what we want.
Can we use compositeByteBuf to replace it and meanwhile enjoy the benefit of 
pooledByteBuf?

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Jialin Qiao  于2022年5月16日周一 12:35写道：

> Hi,
>
> The serialization interface needs to be refactored afterward.
>
> Before that, using ByteArrayOutputStream is easier.
>
> Thanks,
> —
> Jialin Qiao
> Apache IoTDB PMC
>
>
> 李思佳  于2022年5月16日周一 11:44写道：
>
> > Hi all,
> >
> > When I was developing the snapshot interface for the configNode 
> > module, I noticed that the parameters received by the serialization 
> > interface were all defined as ByteBuffer, which seemed to have some 
> > problems. For
> example,
> > the external main process has no way of knowing how big the buffer 
> > will
> be.
> > We can only estimate a large value to allocate memory.
> >
> >  Then I looked at the serialization interfaces of other modules, and 
> > it seemed that most modules did the same thing. This could be a 
> > problem once the actual size of the buffer exceeds our estimate. So 
> > I did a quick
> survey
> > of Netty's byteBuf last week, and here's the Chinese version of the
> results<
> > https://apache-iotdb.feishu.cn/docs/doccnW1EFoyLOScys9GTOuaEUbh>.
> >
> >   At the same time, we found that the consensus module also has some 
> > ByteBuf requirements. But byteBuf doesn't seem to be enough to give 
> > us precise control over the size of the memory pool, and we may need 
> > to wrap it if we decide to use it.
> >
> >   Finally, we decided to use stream type instead of byteBuffer in 
> > configNode for the time being. I will start this work to see if this 
> > is
> the
> > better way this week. If any idea, please let me know.
> >
> >   By the way, Netty’s ByteBuf provides powerful tool operations that 
> > we will not discard outright, but rather as an option.
> >
> > BR,
> > ---
> > Sijia Li
> >
> >
>

Re: About replacing byteBuffer

[Xiangdong Huang]

If we introduce Netty, data copy when scaling a bytebuf is not what we
want.
Can we use compositeByteBuf to replace it and meanwhile enjoy the benefit
of pooledByteBuf?

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Jialin Qiao  于2022年5月16日周一 12:35写道：

> Hi,
>
> The serialization interface needs to be refactored afterward.
>
> Before that, using ByteArrayOutputStream is easier.
>
> Thanks,
> —
> Jialin Qiao
> Apache IoTDB PMC
>
>
> 李思佳  于2022年5月16日周一 11:44写道：
>
> > Hi all,
> >
> > When I was developing the snapshot interface for the configNode module, I
> > noticed that the parameters received by the serialization interface were
> > all defined as ByteBuffer, which seemed to have some problems. For
> example,
> > the external main process has no way of knowing how big the buffer will
> be.
> > We can only estimate a large value to allocate memory.
> >
> >  Then I looked at the serialization interfaces of other modules, and it
> > seemed that most modules did the same thing. This could be a problem once
> > the actual size of the buffer exceeds our estimate. So I did a quick
> survey
> > of Netty's byteBuf last week, and here's the Chinese version of the
> results<
> > https://apache-iotdb.feishu.cn/docs/doccnW1EFoyLOScys9GTOuaEUbh>.
> >
> >   At the same time, we found that the consensus module also has some
> > ByteBuf requirements. But byteBuf doesn't seem to be enough to give us
> > precise control over the size of the memory pool, and we may need to wrap
> > it if we decide to use it.
> >
> >   Finally, we decided to use stream type instead of byteBuffer in
> > configNode for the time being. I will start this work to see if this is
> the
> > better way this week. If any idea, please let me know.
> >
> >   By the way, Netty’s ByteBuf provides powerful tool operations that we
> > will not discard outright, but rather as an option.
> >
> > BR,
> > ---
> > Sijia Li
> >
> >
>

Re: About replacing byteBuffer

[Jialin Qiao]

Hi,

The serialization interface needs to be refactored afterward.

Before that, using ByteArrayOutputStream is easier.

Thanks,
—
Jialin Qiao
Apache IoTDB PMC


李思佳  于2022年5月16日周一 11:44写道：

> Hi all,
>
> When I was developing the snapshot interface for the configNode module, I
> noticed that the parameters received by the serialization interface were
> all defined as ByteBuffer, which seemed to have some problems. For example,
> the external main process has no way of knowing how big the buffer will be.
> We can only estimate a large value to allocate memory.
>
>  Then I looked at the serialization interfaces of other modules, and it
> seemed that most modules did the same thing. This could be a problem once
> the actual size of the buffer exceeds our estimate. So I did a quick survey
> of Netty's byteBuf last week, and here's the Chinese version of the results<
> https://apache-iotdb.feishu.cn/docs/doccnW1EFoyLOScys9GTOuaEUbh>.
>
>   At the same time, we found that the consensus module also has some
> ByteBuf requirements. But byteBuf doesn't seem to be enough to give us
> precise control over the size of the memory pool, and we may need to wrap
> it if we decide to use it.
>
>   Finally, we decided to use stream type instead of byteBuffer in
> configNode for the time being. I will start this work to see if this is the
> better way this week. If any idea, please let me know.
>
>   By the way, Netty’s ByteBuf provides powerful tool operations that we
> will not discard outright, but rather as an option.
>
> BR,
> ---
> Sijia Li
>
>

About replacing byteBuffer

[李思佳]

Hi all,

When I was developing the snapshot interface for the configNode module, I 
noticed that the parameters received by the serialization interface were all 
defined as ByteBuffer, which seemed to have some problems. For example, the 
external main process has no way of knowing how big the buffer will be. We can 
only estimate a large value to allocate memory.

 Then I looked at the serialization interfaces of other modules, and it seemed 
that most modules did the same thing. This could be a problem once the actual 
size of the buffer exceeds our estimate. So I did a quick survey of Netty's 
byteBuf last week, and here's the Chinese version of the 
results.

  At the same time, we found that the consensus module also has some ByteBuf 
requirements. But byteBuf doesn't seem to be enough to give us precise control 
over the size of the memory pool, and we may need to wrap it if we decide to 
use it.

  Finally, we decided to use stream type instead of byteBuffer in configNode 
for the time being. I will start this work to see if this is the better way 
this week. If any idea, please let me know.

  By the way, Netty’s ByteBuf provides powerful tool operations that we will 
not discard outright, but rather as an option.

BR,
---
Sijia Li