On Sat, Sep 18, 2021 at 8:21 PM Lohit Valleru via Beowulf <beowulf@beowulf.org> wrote: > > Hello Everyone, > > I am trying to find answers to an age old question of NFS vs Parallel file > systems. Specifically - Isilon oneFS vs parallel filesystems.Specifically > looking for any technical articles or papers that can help me understand what > exactly will not work on oneFS. > I understand that at the end - it all depends on workloads. > But at what capacity of metadata io or a particular io pattern is bad in > NFS.Would just getting a beefy isilon NFS HDD based storage - resolve most of > the issues? > I am trying to find sources that can say that no matter how beefy an NFS > server can get with HDDs as backed - it will not be as good as parallel > filesystems for so and so workload. > If possible - Can anyone point me to experiences or technical papers that > mention so and so do not work with NFS. > > Does it have to be that at the end - i will have to test my workloads across > both NFS/OneFS and Parallel File systems and then see what would not work? > > I am concerned that any test case might not be valid, compared to real shared > workloads where performance might lag once the storage reaches PBs in scale > and millions of files.
For one thing NFS is not cache coherent, but rather implements a looser form of consistency called close-to-open consistency. See e.g. the spec at https://datatracker.ietf.org/doc/html/rfc7530#section-1.4.6 One case in which this matters is if you have a workload where multiple nodes concurrently write to a shared file. E.g. with the ever-popular IOR benchmarking tool, a slurm batch file like #SBATCH -N 2 # 2 nodes #SBATCH --ntasks-per-node=1 # 1 MPI task per node SEGMENTCOUNT=100 #Offset must be equal to ntasks-per-node OFFSET=1 srun IOR -a POSIX -t 1000 -b 1000 -s $SEGMENTCOUNT -C -Q $OFFSET -e -i 5 -d 10 -v -w -r -W -R -g -u -q -o testfile This should fail due to corruption within minutes if the testfile is on NFS. Not saying any parallel filesystem will handle this either. Some will. -- Janne Blomqvist _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
