John,
I am not sure you will get much help here with a kernel crash caused
by a tweaked driver.
About HPL, you are more likely to get better performance with P and Q
closer (e.g. 4x8 is likely better then 2x16 or 1x32).
Also, HPL might have better performance with one MPI task per node and
a multithreaded BLAS
(e.g. PxQ = 2x4 and 4 OpenMP threads per MPI task)
Cheers,
Gilles
On Mon, Aug 10, 2020 at 3:31 AM John Duffy via users
wrote:
>
> Hi
>
> I have generated this problem myself by tweaking the MTU of my 8 node
> Raspberry Pi 4 cluster to 9000 bytes, but I would be grateful for any
> ideas/suggestions on how to relate the Open-MPI ORTE message to my tweaking.
>
> When I run HPL Linpack using my “improved” cluster, it runs quite happily for
> 2 hours with P=1 & Q=32 using 80% of memory, and this give me a 7%
> performance increase to 97 Gflops. And I can quite happily Iperf 1GB of data
> between nodes with an improved bandwidth of 980Mb/s. So, the MTU tweak
> appears to be relatively robust.
>
> However, as soon as the HPL.dat parameters change to P=2 & Q=16, from within
> the same HPL.dat file, I get the following message...
>
> --
> ORTE has lost communication with a remote daemon.
>
> HNP daemon : [[19859,0],0] on node node1
> Remote daemon: [[19859,0],5] on node node6
>
> This is usually due to either a failure of the TCP network
> connection to the node, or possibly an internal failure of
> the daemon itself. We cannot recover from this failure, and
> therefore will terminate the job.
> —
>
> …and the affected node becomes uncontactable.
>
> I’m thinking the Open-MPI message sizes with P=2 & Q=16 are not working with
> my imperfect MTU tweak, and I’m corrupting the TCP stack somehow.
>
> My tweak consisted of the following kernel changes:
>
> 1.) include/linux/if_vlan.h
>
> #define VLAN_ETH_DATA_LEN 9000
> #define VLAN_ETH_FRAME_LEN 9018
>
> 2.) include/uapi/linux/if_ether.h
>
> #define ETH_DATA_LEN 9000
> #define ETH_FRAME_LEN 9014
>
> 3.) drivers/net/ethernet/broadcom/genet/bcmgenet.c
>
> #define RX_BUF_LENGTH 10240
>
> The Raspberry Pi 4 ethernet driver does not expose many knobs to turn, most
> ethtool options are not available, and there is no publicly available NIC
> documentation, so my tweaks are educated guesswork based upon Raspberry Pi
> forum threads.
>
> Any ideas/suggestions would be much appreciated. With P=2 & Q=16 prior to my
> tweak I can achieve 100 Gflops, a potential increase to 107 Gflops is not to
> be sniffed at.
>
> Best regards
>
