Hi I have generated this problem myself by tweaking the MTU of my 8 node Raspberry Pi 4 cluster to 9000 bytes, but I would be grateful for any ideas/suggestions on how to relate the Open-MPI ORTE message to my tweaking.
When I run HPL Linpack using my “improved” cluster, it runs quite happily for 2 hours with P=1 & Q=32 using 80% of memory, and this give me a 7% performance increase to 97 Gflops. And I can quite happily Iperf 1GB of data between nodes with an improved bandwidth of 980Mb/s. So, the MTU tweak appears to be relatively robust. However, as soon as the HPL.dat parameters change to P=2 & Q=16, from within the same HPL.dat file, I get the following message... -------------------------------------------------------------------------- ORTE has lost communication with a remote daemon. HNP daemon : [[19859,0],0] on node node1 Remote daemon: [[19859,0],5] on node node6 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. ————————————————————————————————————— …and the affected node becomes uncontactable. I’m thinking the Open-MPI message sizes with P=2 & Q=16 are not working with my imperfect MTU tweak, and I’m corrupting the TCP stack somehow. My tweak consisted of the following kernel changes: 1.) include/linux/if_vlan.h #define VLAN_ETH_DATA_LEN 9000 #define VLAN_ETH_FRAME_LEN 9018 2.) include/uapi/linux/if_ether.h #define ETH_DATA_LEN 9000 #define ETH_FRAME_LEN 9014 3.) drivers/net/ethernet/broadcom/genet/bcmgenet.c #define RX_BUF_LENGTH 10240 The Raspberry Pi 4 ethernet driver does not expose many knobs to turn, most ethtool options are not available, and there is no publicly available NIC documentation, so my tweaks are educated guesswork based upon Raspberry Pi forum threads. Any ideas/suggestions would be much appreciated. With P=2 & Q=16 prior to my tweak I can achieve 100 Gflops, a potential increase to 107 Gflops is not to be sniffed at. Best regards
