Re: DPDK for File Transfer and Writing
On Wed, 11 Jun 2025 08:08:20 + (UTC) Isaac Lim wrote: > Dear Stephen, > Thank you so much. > Regarding SPDX, I did a Google search. Is it Software Package Data Exchange? > Best Regards,Isaac > > On Tuesday, June 10, 2025 at 09:36:45 PM GMT+8, Stephen Hemminger > wrote: > > On Tue, 10 Jun 2025 08:49:02 + (UTC) > Isaac Lim wrote: > > > Dear all, > > We are using DPDK to transfer files and write to SSD on the receiver side. > > On the sender side, we are splitting the files into 1024-byte packets > > before sending over to the receiver. > > Currently, on the sender side, we are using a worker lcore to retrieve the > > packets and storing the file data in a buffer while the main lcore writes > > the data to file using write(). > > However, we are realizing that the write speed is not fast enough to keep > > up with the number of packets being received. > > We have tried using io_uring to aid speeding up the writing but it seems to > > be even slower than just writing the file normally. > > We would like to check with the community if you have any DPDK compatible > > suggestions that would help improve file transfer and write performance? > > Thank you very much in advance. > > Best Regards,Isaac LIM > > You could try SPDX to get faster storage, but networking is faster than > storage. > > At some point you need a full network stack with real protocols like TCP. > SPDK https://spdk.io/
Re: DPDK for File Transfer and Writing
Dear Stephen, Thank you so much. Regarding SPDX, I did a Google search. Is it Software Package Data Exchange? Best Regards,Isaac On Tuesday, June 10, 2025 at 09:36:45 PM GMT+8, Stephen Hemminger wrote: On Tue, 10 Jun 2025 08:49:02 + (UTC) Isaac Lim wrote: > Dear all, > We are using DPDK to transfer files and write to SSD on the receiver side. > On the sender side, we are splitting the files into 1024-byte packets before > sending over to the receiver. > Currently, on the sender side, we are using a worker lcore to retrieve the > packets and storing the file data in a buffer while the main lcore writes the > data to file using write(). > However, we are realizing that the write speed is not fast enough to keep up > with the number of packets being received. > We have tried using io_uring to aid speeding up the writing but it seems to > be even slower than just writing the file normally. > We would like to check with the community if you have any DPDK compatible > suggestions that would help improve file transfer and write performance? > Thank you very much in advance. > Best Regards,Isaac LIM You could try SPDX to get faster storage, but networking is faster than storage. At some point you need a full network stack with real protocols like TCP.
Re: DPDK for File Transfer and Writing
On Tue, 10 Jun 2025 08:49:02 + (UTC) Isaac Lim wrote: > Dear all, > We are using DPDK to transfer files and write to SSD on the receiver side. > On the sender side, we are splitting the files into 1024-byte packets before > sending over to the receiver. > Currently, on the sender side, we are using a worker lcore to retrieve the > packets and storing the file data in a buffer while the main lcore writes the > data to file using write(). > However, we are realizing that the write speed is not fast enough to keep up > with the number of packets being received. > We have tried using io_uring to aid speeding up the writing but it seems to > be even slower than just writing the file normally. > We would like to check with the community if you have any DPDK compatible > suggestions that would help improve file transfer and write performance? > Thank you very much in advance. > Best Regards,Isaac LIM You could try SPDX to get faster storage, but networking is faster than storage. At some point you need a full network stack with real protocols like TCP.
