And fcp? On 6/1/22, Bakul Shah <ba...@iitbombay.org> wrote: > On May 31, 2022, at 9:14 AM, ron minnich <rminn...@gmail.com> wrote: >> >> On Mon, May 30, 2022 at 12:21 AM Bakul Shah <ba...@iitbombay.org> wrote: >>> 9p itself is low performance but that is a separate issue. >> >> Bakul, what are the units? It might be helpful to quantify this >> statement. Are you possibly conflating Plan 9 file systems being slow >> and 9p being slow? > > I did a quick test: > > From a 9front VM to another machine I get about 11.7 MBps > caching. The first time around it was close to 7.3 MBps). > > From an Ubuntu VM to another machine I get about 111 MBps > (cached. The first time around it was close to 62 MBps). > > Both VMs run on the same host. Test copies to the same target > machine. I used 9p read for 9front, scp for Linux, copy to > /dev/null. The target machine is freebsd. The VMs talk to > the target over a 1Gbps ethernet (so 111 MBps is the wirespeed > limit). > > 9front uses hjfs. Ubuntu uses ext4. On the host I give a file > as the guest "disk", using 'nvme' type device on bhyve to each > VM. Both 9front and ubuntu are 64 bit kernels. > > This is a very rough measurement as there are many differences > between the systems. The filesystem overhead is clearly an issue > but 10 times worse? > ----- > Looking at the protocol: > > For read/write 9p uses 4 byte for size so in theory you can send > very large packets but then you have to buffer up a lot of data. > Ideally you want streaming (some sort of sliding window). May be > you can use the tag field to do something more intelligent. Not > sure any implementations do so. You also have head of line blocking > if you can have only one TCP connection to a server. > >> As Rob pointed out in 2013, "If go install is slow on Plan 9, it's >> because Plan 9's file system is >> slow (which it is and always has been)", so slowness in Plan 9 file >> systems is to be expected. >> >> 9p itself does have its limits, which is why Bell Labs Antwerp started >> an effort in 2011 to replace it, but the new work never went very far. >> >> I also know of a number of efforts in the virtualization world where >> 9p was discarded for performance reasons. It's hard to argue with the >> 100x performance improvement that comes with virtiofs, for example. > > > Why is virtiofs 100x faster? Just lot of hardwork and tuning? > May be that is good place to look to learn what needs to change > (in case someone wants to replace 9p with something else)? > >> Gvisor is replacing 9p: https://github.com/google/gvisor/milestone/6. >> Although, in the latter case, I would argue the problem is more with >> Linux limitations than 9p limitations -- linux can't seem to walk more >> than one pathname component at a time, for example, since it has the >> old school namei loop. >> >> But I'm wondering if you have a measurement with numbers. >> >> For rough order of magnitude, HPC file systems can deliver 10 Gbytes/ >> second for file reads nowadays, but getting there took 20 years of >> work. When we ran Plan 9 on Blue Gene, with the 6 Gbyte/second >> toroidal mesh connect for each node, we never came remotely close to >> that figure. > > Given that experience, why do you need "numbers"? :-) > > Running 10Gbps links even @ home is quite doable now. With TCP you > can achieve decent performance if not quite wirespeed. NVMe "disks" > are pretty damn fast - you can easily get 2-4 GBps. But I think at > remote filesystem protocol level you'd have to optimize multiple > things in order to get close to wirespeed performance. Minimize > copying, increase concurrency, reduce overhead in frequently used > common path code, reduce user/kernel crossings etc. I think rdma and > mmap will probably get used a lot too (obviously on non-plan9 OSes!). > May be if you pushed 9p knowledge down to a smart NIC, it can map a > tag value to compute location where the data needs to go. > > But all this is just handwaving. Without a real project and funding > it is hard to get sufficiently motivated to do more. >
------------------------------------------ 9fans: 9fans Permalink: https://9fans.topicbox.com/groups/9fans/T769854fafd2b7d35-M9900ebe3ebf76b5d4d4426bd Delivery options: https://9fans.topicbox.com/groups/9fans/subscription
