RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[Dexuan Cui]

> From: David Miller [mailto:da...@davemloft.net]
> Sent: Thursday, May 19, 2016 12:13
> To: Dexuan Cui <de...@microsoft.com>
> Cc: KY Srinivasan <k...@microsoft.com>; o...@aepfle.de;
> gre...@linuxfoundation.org; jasow...@redhat.com; linux-
> ker...@vger.kernel.org; j...@perches.com; net...@vger.kernel.org;
> a...@canonical.com; de...@linuxdriverproject.org; Haiyang Zhang
> <haiya...@microsoft.com>
> Subject: Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
> 
> 
> I'm travelling and very busy with the merge window.  So sorry I won't be able
> to think about this for some time.

David, 
Sure, I understand.

Please let me recap my last mail:

1)  I'll replace my statically-allocated per-connection "send/recv bufs" with
dynamically ones, so no buf is used when there is no traffic.

2) Another kind of bufs i.e., the  multi-page "VMBus send/recv ringbuffer", is
a must IMO due to the host side's design of the feature: every connection needs
its own ringbuffer, which takes several pages (2~3 pages at least. And, 5 pages
should suffice for good performance). The ringbuffer can be accessed by the
host at any time, so IMO the pages can't be swappable.

I understand net-next is closed now. I'm going to post the next version
after 4.7-rc1 is out in several weeks.

If you could give me some suggestions, I would be definitely happy to take.

Thanks!
-- Dexuan

RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[Dexuan Cui]

> From: David Miller [mailto:da...@davemloft.net]
> Sent: Thursday, May 19, 2016 12:13
> To: Dexuan Cui 
> Cc: KY Srinivasan ; o...@aepfle.de;
> gre...@linuxfoundation.org; jasow...@redhat.com; linux-
> ker...@vger.kernel.org; j...@perches.com; net...@vger.kernel.org;
> a...@canonical.com; de...@linuxdriverproject.org; Haiyang Zhang
> 
> Subject: Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
> 
> 
> I'm travelling and very busy with the merge window.  So sorry I won't be able
> to think about this for some time.

David, 
Sure, I understand.

Please let me recap my last mail:

1)  I'll replace my statically-allocated per-connection "send/recv bufs" with
dynamically ones, so no buf is used when there is no traffic.

2) Another kind of bufs i.e., the  multi-page "VMBus send/recv ringbuffer", is
a must IMO due to the host side's design of the feature: every connection needs
its own ringbuffer, which takes several pages (2~3 pages at least. And, 5 pages
should suffice for good performance). The ringbuffer can be accessed by the
host at any time, so IMO the pages can't be swappable.

I understand net-next is closed now. I'm going to post the next version
after 4.7-rc1 is out in several weeks.

If you could give me some suggestions, I would be definitely happy to take.

Thanks!
-- Dexuan

Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[David Miller]

I'm travelling and very busy with the merge window.  So sorry I won't be able
to think about this for some time.

Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[David Miller]

I'm travelling and very busy with the merge window.  So sorry I won't be able
to think about this for some time.

Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[gre...@linuxfoundation.org]

On Thu, May 19, 2016 at 12:59:09AM +, Dexuan Cui wrote:
> > From: devel [mailto:driverdev-devel-boun...@linuxdriverproject.org] On 
> > Behalf
> > Of Dexuan Cui
> > Sent: Tuesday, May 17, 2016 10:46
> > To: David Miller <da...@davemloft.net>
> > Cc: o...@aepfle.de; gre...@linuxfoundation.org; jasow...@redhat.com;
> > linux-kernel@vger.kernel.org; j...@perches.com; net...@vger.kernel.org;
> > a...@canonical.com; de...@linuxdriverproject.org; Haiyang Zhang
> > <haiya...@microsoft.com>
> > Subject: RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
> > 
> > > From: David Miller [mailto:da...@davemloft.net]
> > > Sent: Monday, May 16, 2016 1:16
> > > To: Dexuan Cui <de...@microsoft.com>
> > > Cc: gre...@linuxfoundation.org; net...@vger.kernel.org; linux-
> > > ker...@vger.kernel.org; de...@linuxdriverproject.org; o...@aepfle.de;
> > > a...@canonical.com; jasow...@redhat.com; cav...@redhat.com; KY
> > > Srinivasan <k...@microsoft.com>; Haiyang Zhang <haiya...@microsoft.com>;
> > > j...@perches.com; vkuzn...@redhat.com
> > > Subject: Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM 
> > > Sockets(hv_sock)
> > >
> > > From: Dexuan Cui <de...@microsoft.com>
> > > Date: Sun, 15 May 2016 09:52:42 -0700
> > >
> > > > Changes since v10
> > > >
> > > > 1) add module params: send_ring_page, recv_ring_page. They can be used
> > to
> > > > enlarge the ringbuffer size to get better performance, e.g.,
> > > > # modprobe hv_sock  recv_ring_page=16 send_ring_page=16
> > > > By default, recv_ring_page is 3 and send_ring_page is 2.
> > > >
> > > > 2) add module param max_socket_number (the default is 1024).
> > > > A user can enlarge the number to create more than 1024 hv_sock sockets.
> > > > By default, 1024 sockets take about 1024 * (3+2+1+1) * 4KB = 28M bytes.
> > > > (Here 1+1 means 1 page for send/recv buffers per connection, 
> > > > respectively.)
> > >
> > > This is papering around my objections, and create module parameters which
> > > I am fundamentally against.
> > >
> > > You're making the facility unusable by default, just to work around my
> > > memory consumption concerns.
> > >
> > > What will end up happening is that everyone will simply increase the
> > > values.
> > >
> > > You're not really addressing the core issue, and I will be ignoring you
> > > future submissions of this change until you do.
> > 
> > David,
> > I am sorry I came across as ignoring your feedback; that was not my 
> > intention.
> > The current host side design for this feature is such that each socket 
> > connection
> > needs its own channel, which consists of
> > 
> > 1.A ring buffer for host to guest communication
> > 2.A ring buffer for guest to host communication
> > 
> > The memory for the ring buffers has to be pinned down as this will be 
> > accessed
> > both from interrupt level in Linux guest and from the host OS at any time.
> > 
> > To address your concerns, I am planning to re-implement both the receive 
> > path
> > and the send path so that no additional pinned memory will be needed.
> > 
> > Receive Path:
> > When the application does a read on the socket, we will dynamically allocate
> > the buffer and perform the read operation on the incoming ring buffer. Since
> > we will be in the process context, we can sleep here and will set the
> > "GFP_KERNEL | __GFP_NOFAIL" flags. This buffer will be freed once the
> > application consumes all the data.
> > 
> > Send Path:
> > On the send side, we will construct the payload to be sent directly on the
> > outgoing ringbuffer.
> > 
> > So, with these changes, the only memory that will be pinned down will be the
> > memory for the ring buffers on a per-connection basis and this memory will 
> > be
> > pinned down until the connection is torn down.
> > 
> > Please let me know if this addresses your concerns.
> > 
> > -- Dexuan
> 
> Hi David,
> Ping. Really appreciate your comment.

Don't wait for people to respond to random design questions, go work on
the code and figure out if it is workable or not yourself.  Then post
patches.  We aren't responsible for your work, you are.

greg k-h

Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[gre...@linuxfoundation.org]

On Thu, May 19, 2016 at 12:59:09AM +, Dexuan Cui wrote:
> > From: devel [mailto:driverdev-devel-boun...@linuxdriverproject.org] On 
> > Behalf
> > Of Dexuan Cui
> > Sent: Tuesday, May 17, 2016 10:46
> > To: David Miller 
> > Cc: o...@aepfle.de; gre...@linuxfoundation.org; jasow...@redhat.com;
> > linux-kernel@vger.kernel.org; j...@perches.com; net...@vger.kernel.org;
> > a...@canonical.com; de...@linuxdriverproject.org; Haiyang Zhang
> > 
> > Subject: RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
> > 
> > > From: David Miller [mailto:da...@davemloft.net]
> > > Sent: Monday, May 16, 2016 1:16
> > > To: Dexuan Cui 
> > > Cc: gre...@linuxfoundation.org; net...@vger.kernel.org; linux-
> > > ker...@vger.kernel.org; de...@linuxdriverproject.org; o...@aepfle.de;
> > > a...@canonical.com; jasow...@redhat.com; cav...@redhat.com; KY
> > > Srinivasan ; Haiyang Zhang ;
> > > j...@perches.com; vkuzn...@redhat.com
> > > Subject: Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM 
> > > Sockets(hv_sock)
> > >
> > > From: Dexuan Cui 
> > > Date: Sun, 15 May 2016 09:52:42 -0700
> > >
> > > > Changes since v10
> > > >
> > > > 1) add module params: send_ring_page, recv_ring_page. They can be used
> > to
> > > > enlarge the ringbuffer size to get better performance, e.g.,
> > > > # modprobe hv_sock  recv_ring_page=16 send_ring_page=16
> > > > By default, recv_ring_page is 3 and send_ring_page is 2.
> > > >
> > > > 2) add module param max_socket_number (the default is 1024).
> > > > A user can enlarge the number to create more than 1024 hv_sock sockets.
> > > > By default, 1024 sockets take about 1024 * (3+2+1+1) * 4KB = 28M bytes.
> > > > (Here 1+1 means 1 page for send/recv buffers per connection, 
> > > > respectively.)
> > >
> > > This is papering around my objections, and create module parameters which
> > > I am fundamentally against.
> > >
> > > You're making the facility unusable by default, just to work around my
> > > memory consumption concerns.
> > >
> > > What will end up happening is that everyone will simply increase the
> > > values.
> > >
> > > You're not really addressing the core issue, and I will be ignoring you
> > > future submissions of this change until you do.
> > 
> > David,
> > I am sorry I came across as ignoring your feedback; that was not my 
> > intention.
> > The current host side design for this feature is such that each socket 
> > connection
> > needs its own channel, which consists of
> > 
> > 1.A ring buffer for host to guest communication
> > 2.A ring buffer for guest to host communication
> > 
> > The memory for the ring buffers has to be pinned down as this will be 
> > accessed
> > both from interrupt level in Linux guest and from the host OS at any time.
> > 
> > To address your concerns, I am planning to re-implement both the receive 
> > path
> > and the send path so that no additional pinned memory will be needed.
> > 
> > Receive Path:
> > When the application does a read on the socket, we will dynamically allocate
> > the buffer and perform the read operation on the incoming ring buffer. Since
> > we will be in the process context, we can sleep here and will set the
> > "GFP_KERNEL | __GFP_NOFAIL" flags. This buffer will be freed once the
> > application consumes all the data.
> > 
> > Send Path:
> > On the send side, we will construct the payload to be sent directly on the
> > outgoing ringbuffer.
> > 
> > So, with these changes, the only memory that will be pinned down will be the
> > memory for the ring buffers on a per-connection basis and this memory will 
> > be
> > pinned down until the connection is torn down.
> > 
> > Please let me know if this addresses your concerns.
> > 
> > -- Dexuan
> 
> Hi David,
> Ping. Really appreciate your comment.

Don't wait for people to respond to random design questions, go work on
the code and figure out if it is workable or not yourself.  Then post
patches.  We aren't responsible for your work, you are.

greg k-h

RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[Dexuan Cui]

> From: devel [mailto:driverdev-devel-boun...@linuxdriverproject.org] On Behalf
> Of Dexuan Cui
> Sent: Tuesday, May 17, 2016 10:46
> To: David Miller <da...@davemloft.net>
> Cc: o...@aepfle.de; gre...@linuxfoundation.org; jasow...@redhat.com;
> linux-kernel@vger.kernel.org; j...@perches.com; net...@vger.kernel.org;
> a...@canonical.com; de...@linuxdriverproject.org; Haiyang Zhang
> <haiya...@microsoft.com>
> Subject: RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
> 
> > From: David Miller [mailto:da...@davemloft.net]
> > Sent: Monday, May 16, 2016 1:16
> > To: Dexuan Cui <de...@microsoft.com>
> > Cc: gre...@linuxfoundation.org; net...@vger.kernel.org; linux-
> > ker...@vger.kernel.org; de...@linuxdriverproject.org; o...@aepfle.de;
> > a...@canonical.com; jasow...@redhat.com; cav...@redhat.com; KY
> > Srinivasan <k...@microsoft.com>; Haiyang Zhang <haiya...@microsoft.com>;
> > j...@perches.com; vkuzn...@redhat.com
> > Subject: Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
> >
> > From: Dexuan Cui <de...@microsoft.com>
> > Date: Sun, 15 May 2016 09:52:42 -0700
> >
> > > Changes since v10
> > >
> > > 1) add module params: send_ring_page, recv_ring_page. They can be used
> to
> > > enlarge the ringbuffer size to get better performance, e.g.,
> > > # modprobe hv_sock  recv_ring_page=16 send_ring_page=16
> > > By default, recv_ring_page is 3 and send_ring_page is 2.
> > >
> > > 2) add module param max_socket_number (the default is 1024).
> > > A user can enlarge the number to create more than 1024 hv_sock sockets.
> > > By default, 1024 sockets take about 1024 * (3+2+1+1) * 4KB = 28M bytes.
> > > (Here 1+1 means 1 page for send/recv buffers per connection, 
> > > respectively.)
> >
> > This is papering around my objections, and create module parameters which
> > I am fundamentally against.
> >
> > You're making the facility unusable by default, just to work around my
> > memory consumption concerns.
> >
> > What will end up happening is that everyone will simply increase the
> > values.
> >
> > You're not really addressing the core issue, and I will be ignoring you
> > future submissions of this change until you do.
> 
> David,
> I am sorry I came across as ignoring your feedback; that was not my intention.
> The current host side design for this feature is such that each socket 
> connection
> needs its own channel, which consists of
> 
> 1.A ring buffer for host to guest communication
> 2.A ring buffer for guest to host communication
> 
> The memory for the ring buffers has to be pinned down as this will be accessed
> both from interrupt level in Linux guest and from the host OS at any time.
> 
> To address your concerns, I am planning to re-implement both the receive path
> and the send path so that no additional pinned memory will be needed.
> 
> Receive Path:
> When the application does a read on the socket, we will dynamically allocate
> the buffer and perform the read operation on the incoming ring buffer. Since
> we will be in the process context, we can sleep here and will set the
> "GFP_KERNEL | __GFP_NOFAIL" flags. This buffer will be freed once the
> application consumes all the data.
> 
> Send Path:
> On the send side, we will construct the payload to be sent directly on the
> outgoing ringbuffer.
> 
> So, with these changes, the only memory that will be pinned down will be the
> memory for the ring buffers on a per-connection basis and this memory will be
> pinned down until the connection is torn down.
> 
> Please let me know if this addresses your concerns.
> 
> -- Dexuan

Hi David,
Ping. Really appreciate your comment.

 Thanks,
-- Dexuan

RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[Dexuan Cui]

> From: devel [mailto:driverdev-devel-boun...@linuxdriverproject.org] On Behalf
> Of Dexuan Cui
> Sent: Tuesday, May 17, 2016 10:46
> To: David Miller 
> Cc: o...@aepfle.de; gre...@linuxfoundation.org; jasow...@redhat.com;
> linux-kernel@vger.kernel.org; j...@perches.com; net...@vger.kernel.org;
> a...@canonical.com; de...@linuxdriverproject.org; Haiyang Zhang
> 
> Subject: RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
> 
> > From: David Miller [mailto:da...@davemloft.net]
> > Sent: Monday, May 16, 2016 1:16
> > To: Dexuan Cui 
> > Cc: gre...@linuxfoundation.org; net...@vger.kernel.org; linux-
> > ker...@vger.kernel.org; de...@linuxdriverproject.org; o...@aepfle.de;
> > a...@canonical.com; jasow...@redhat.com; cav...@redhat.com; KY
> > Srinivasan ; Haiyang Zhang ;
> > j...@perches.com; vkuzn...@redhat.com
> > Subject: Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
> >
> > From: Dexuan Cui 
> > Date: Sun, 15 May 2016 09:52:42 -0700
> >
> > > Changes since v10
> > >
> > > 1) add module params: send_ring_page, recv_ring_page. They can be used
> to
> > > enlarge the ringbuffer size to get better performance, e.g.,
> > > # modprobe hv_sock  recv_ring_page=16 send_ring_page=16
> > > By default, recv_ring_page is 3 and send_ring_page is 2.
> > >
> > > 2) add module param max_socket_number (the default is 1024).
> > > A user can enlarge the number to create more than 1024 hv_sock sockets.
> > > By default, 1024 sockets take about 1024 * (3+2+1+1) * 4KB = 28M bytes.
> > > (Here 1+1 means 1 page for send/recv buffers per connection, 
> > > respectively.)
> >
> > This is papering around my objections, and create module parameters which
> > I am fundamentally against.
> >
> > You're making the facility unusable by default, just to work around my
> > memory consumption concerns.
> >
> > What will end up happening is that everyone will simply increase the
> > values.
> >
> > You're not really addressing the core issue, and I will be ignoring you
> > future submissions of this change until you do.
> 
> David,
> I am sorry I came across as ignoring your feedback; that was not my intention.
> The current host side design for this feature is such that each socket 
> connection
> needs its own channel, which consists of
> 
> 1.A ring buffer for host to guest communication
> 2.A ring buffer for guest to host communication
> 
> The memory for the ring buffers has to be pinned down as this will be accessed
> both from interrupt level in Linux guest and from the host OS at any time.
> 
> To address your concerns, I am planning to re-implement both the receive path
> and the send path so that no additional pinned memory will be needed.
> 
> Receive Path:
> When the application does a read on the socket, we will dynamically allocate
> the buffer and perform the read operation on the incoming ring buffer. Since
> we will be in the process context, we can sleep here and will set the
> "GFP_KERNEL | __GFP_NOFAIL" flags. This buffer will be freed once the
> application consumes all the data.
> 
> Send Path:
> On the send side, we will construct the payload to be sent directly on the
> outgoing ringbuffer.
> 
> So, with these changes, the only memory that will be pinned down will be the
> memory for the ring buffers on a per-connection basis and this memory will be
> pinned down until the connection is torn down.
> 
> Please let me know if this addresses your concerns.
> 
> -- Dexuan

Hi David,
Ping. Really appreciate your comment.

 Thanks,
-- Dexuan

RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[Dexuan Cui]

> From: David Miller [mailto:da...@davemloft.net]
> Sent: Monday, May 16, 2016 1:16
> To: Dexuan Cui <de...@microsoft.com>
> Cc: gre...@linuxfoundation.org; net...@vger.kernel.org; linux-
> ker...@vger.kernel.org; de...@linuxdriverproject.org; o...@aepfle.de;
> a...@canonical.com; jasow...@redhat.com; cav...@redhat.com; KY
> Srinivasan <k...@microsoft.com>; Haiyang Zhang <haiya...@microsoft.com>;
> j...@perches.com; vkuzn...@redhat.com
> Subject: Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
>
> From: Dexuan Cui <de...@microsoft.com>
> Date: Sun, 15 May 2016 09:52:42 -0700
>
> > Changes since v10
> >
> > 1) add module params: send_ring_page, recv_ring_page. They can be used to
> > enlarge the ringbuffer size to get better performance, e.g.,
> > # modprobe hv_sock  recv_ring_page=16 send_ring_page=16
> > By default, recv_ring_page is 3 and send_ring_page is 2.
> >
> > 2) add module param max_socket_number (the default is 1024).
> > A user can enlarge the number to create more than 1024 hv_sock sockets.
> > By default, 1024 sockets take about 1024 * (3+2+1+1) * 4KB = 28M bytes.
> > (Here 1+1 means 1 page for send/recv buffers per connection, respectively.)
>
> This is papering around my objections, and create module parameters which
> I am fundamentally against.
>
> You're making the facility unusable by default, just to work around my
> memory consumption concerns.
>
> What will end up happening is that everyone will simply increase the
> values.
>
> You're not really addressing the core issue, and I will be ignoring you
> future submissions of this change until you do.

David,
I am sorry I came across as ignoring your feedback; that was not my intention.
The current host side design for this feature is such that each socket 
connection
needs its own channel, which consists of

1.A ring buffer for host to guest communication
2.A ring buffer for guest to host communication

The memory for the ring buffers has to be pinned down as this will be accessed
both from interrupt level in Linux guest and from the host OS at any time.

To address your concerns, I am planning to re-implement both the receive path
and the send path so that no additional pinned memory will be needed.

Receive Path:
When the application does a read on the socket, we will dynamically allocate
the buffer and perform the read operation on the incoming ring buffer. Since
we will be in the process context, we can sleep here and will set the
"GFP_KERNEL | __GFP_NOFAIL" flags. This buffer will be freed once the
application consumes all the data.

Send Path:
On the send side, we will construct the payload to be sent directly on the
outgoing ringbuffer.

So, with these changes, the only memory that will be pinned down will be the
memory for the ring buffers on a per-connection basis and this memory will be
pinned down until the connection is torn down.

Please let me know if this addresses your concerns.

 Thanks,
-- Dexuan

RE: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[Dexuan Cui]

> From: David Miller [mailto:da...@davemloft.net]
> Sent: Monday, May 16, 2016 1:16
> To: Dexuan Cui 
> Cc: gre...@linuxfoundation.org; net...@vger.kernel.org; linux-
> ker...@vger.kernel.org; de...@linuxdriverproject.org; o...@aepfle.de;
> a...@canonical.com; jasow...@redhat.com; cav...@redhat.com; KY
> Srinivasan ; Haiyang Zhang ;
> j...@perches.com; vkuzn...@redhat.com
> Subject: Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)
>
> From: Dexuan Cui 
> Date: Sun, 15 May 2016 09:52:42 -0700
>
> > Changes since v10
> >
> > 1) add module params: send_ring_page, recv_ring_page. They can be used to
> > enlarge the ringbuffer size to get better performance, e.g.,
> > # modprobe hv_sock  recv_ring_page=16 send_ring_page=16
> > By default, recv_ring_page is 3 and send_ring_page is 2.
> >
> > 2) add module param max_socket_number (the default is 1024).
> > A user can enlarge the number to create more than 1024 hv_sock sockets.
> > By default, 1024 sockets take about 1024 * (3+2+1+1) * 4KB = 28M bytes.
> > (Here 1+1 means 1 page for send/recv buffers per connection, respectively.)
>
> This is papering around my objections, and create module parameters which
> I am fundamentally against.
>
> You're making the facility unusable by default, just to work around my
> memory consumption concerns.
>
> What will end up happening is that everyone will simply increase the
> values.
>
> You're not really addressing the core issue, and I will be ignoring you
> future submissions of this change until you do.

David,
I am sorry I came across as ignoring your feedback; that was not my intention.
The current host side design for this feature is such that each socket 
connection
needs its own channel, which consists of

1.A ring buffer for host to guest communication
2.A ring buffer for guest to host communication

The memory for the ring buffers has to be pinned down as this will be accessed
both from interrupt level in Linux guest and from the host OS at any time.

To address your concerns, I am planning to re-implement both the receive path
and the send path so that no additional pinned memory will be needed.

Receive Path:
When the application does a read on the socket, we will dynamically allocate
the buffer and perform the read operation on the incoming ring buffer. Since
we will be in the process context, we can sleep here and will set the
"GFP_KERNEL | __GFP_NOFAIL" flags. This buffer will be freed once the
application consumes all the data.

Send Path:
On the send side, we will construct the payload to be sent directly on the
outgoing ringbuffer.

So, with these changes, the only memory that will be pinned down will be the
memory for the ring buffers on a per-connection basis and this memory will be
pinned down until the connection is torn down.

Please let me know if this addresses your concerns.

 Thanks,
-- Dexuan

Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[David Miller]

From: Dexuan Cui 
Date: Sun, 15 May 2016 09:52:42 -0700

> Changes since v10
> 
> 1) add module params: send_ring_page, recv_ring_page. They can be used to
> enlarge the ringbuffer size to get better performance, e.g.,
> # modprobe hv_sock  recv_ring_page=16 send_ring_page=16
> By default, recv_ring_page is 3 and send_ring_page is 2.
> 
> 2) add module param max_socket_number (the default is 1024).
> A user can enlarge the number to create more than 1024 hv_sock sockets.
> By default, 1024 sockets take about 1024 * (3+2+1+1) * 4KB = 28M bytes.
> (Here 1+1 means 1 page for send/recv buffers per connection, respectively.)

This is papering around my objections, and create module parameters which
I am fundamentally against.

You're making the facility unusable by default, just to work around my
memory consumption concerns.

What will end up happening is that everyone will simply increase the
values.

You're not really addressing the core issue, and I will be ignoring you
future submissions of this change until you do.

Re: [PATCH v11 net-next 0/1] introduce Hyper-V VM Sockets(hv_sock)

[David Miller]

From: Dexuan Cui 
Date: Sun, 15 May 2016 09:52:42 -0700

> Changes since v10
> 
> 1) add module params: send_ring_page, recv_ring_page. They can be used to
> enlarge the ringbuffer size to get better performance, e.g.,
> # modprobe hv_sock  recv_ring_page=16 send_ring_page=16
> By default, recv_ring_page is 3 and send_ring_page is 2.
> 
> 2) add module param max_socket_number (the default is 1024).
> A user can enlarge the number to create more than 1024 hv_sock sockets.
> By default, 1024 sockets take about 1024 * (3+2+1+1) * 4KB = 28M bytes.
> (Here 1+1 means 1 page for send/recv buffers per connection, respectively.)

This is papering around my objections, and create module parameters which
I am fundamentally against.

You're making the facility unusable by default, just to work around my
memory consumption concerns.

What will end up happening is that everyone will simply increase the
values.

You're not really addressing the core issue, and I will be ignoring you
future submissions of this change until you do.