Andrew Savchenko <bircoph <at> gentoo.org> writes:
> While it is good to have another solution available, I don't see > any real benefits of FhgFS/BeeGFS compared to Lustre these days. > At the time where FhgFS was created, Lustre indeed was unable to > use multiple metadata servers, so this was a bottleneck. But now > Lustre also supports distributed metadata, so they should on par in > this matter. Interesting thesis. I only have anecdotal information, from those I've encountered who are willing to converse, privately. Many more sites exist than are publicized as I think most (scientific) groups have a keen interest in distributed processing, in an open source semantic. I did notice the '9999' version of lustre in portage (science overlay), but reading elsewhere I did not know it was still being actively developed? > On the other hand, Lustre has much larger community (e.g. see > TOP-500 list) and is much better tested (and even under such > conditions it has problems in some corner cases). Thus I see no > advantage in FhgFS for HPC setups. Strangely, the folks I have chatted up do not publish their test results as that would be quite a large undertaking to assure critics that the tests are fair and equivalent, with the only thing different being the local and cluster file systems. Lustre seems to have a bad rap, but that may be due to folks testing much earlier versions. I'm no authority on the subject; just trying to ferret out pathways for robust cluster computing on gentoo; although containers are useful, my focus is on the leanest/fastest bare metal HPC Opensource approach. to clusters on gentoo. > Of course world of parallel distributed file systems is very > versatile, so for different tasks/workloads different file systems > are the most suitable, but for typical IB-based HPC storage I see > no better solution than Lustre at this moment. YES. But also these test/benchmarks should include Cephfs, gluster, and tachyon if not many others. [1] Perhaps we should encourage some of our gentoo-devs, to put up a wiki for gentoo-HPC, with at least a working framework of packages suggested, including all the DFS tricks therein ? Me, I'm just stumbling my way around to try to figure out a resonable pathway to HPC on gentoo. I thought that systemd was going to dominate these cluster-container wars until I started reading up on Docker's acquisition of the main dev at Alpine linux and the rapid movement of Docker to 'subsume' Alpine linux as it's distro for releases [2]. Alpine leverages OpenRC and eudev and Docker is preparing for battle with other container offerings, commercially, so this does suggest that the performance battle with clusters is now openly challenging the systemd proponents for performance bragging rights. Combined with the question of the DFS, it does lsuggest some publish test comparing these different approaches would be of keen interest to a wide audience. The only test code I am aware of for HPC on gentoo is sys-cluster/hpl and I'm not sure how well that will exercise the DFS performance questions. > Best regards, > Andrew Savchenko James [1] http://www.datanami.com/2016/02/23/meet-alluxio-the-distributed-file-system-formerly-known-as-tachyon/ [2] https://www.brianchristner.io/docker-is-moving-to-alpine-linux/
