Dear Wojtek, Thanks for your reply. I did not mention, but I am aware of the REAL TIME CLOCK settings. Thanks for pointing out how it is then related to the phc and /dev/ptp files Just for information we are using Connect X6-Dx cards with REAL_TIME_CLOCK enabled.
What you pointed out with PCIe bus read latency is an interesting thought. As I mentioned wIth my test software I can measure two clock with clock_gettime(phcclockID, ts) and show their difference. I observed the difference between mlnx1 and mlnx3 and it is close to 1450 +- 50 ns. The jitter is stable when there is no PCIe traffic (no IP input/output streams) Still my original question remains, if there is some other/better way to synchronize the hardware clocks on multiple NICs other than starting instances of phc2sys per NIC ? Another Idea/question : Is it possible that the cards talk over PCIe without intervention from CPU to sync their hardware clocks, e.g. Mellanox NIC 1 has PTP (hardware clock syned via ptp4l) and other mellanox NIC cards are synced over PCIe in the FPGA. Cheers Prankur On Mon, Feb 13, 2023 at 1:23 PM Wojtek Wasko <wwa...@nvidia.com> wrote: > > There is a caveat here : I assume both the ports on the same NIC (mlnx1 > and mlnx2 for example) > > have the same hardware clock source even though under /dev/ they are > mapped to different ptp > > files i.e /dev/ptp8 and /dev/ptp9 respectively. > > Depending on the configuration of the Mellanox NIC, the PHC exposed for > different ports will either be the same PHC (in Real-Time Clock mode) - > though exposed through two /dev/ files or (in non-Real-Time Clock mode) > driver will construct separate PHCs for each of the ports. > You can find some instructions on how to configure it here: > https://docs.nvidia.com/networking/display/NVIDIA5TTechnologyUserManualv10/Real-Time+Clock > And some general documentation on the Real-Time Clock here: > https://docs.nvidia.com/networking/display/NVIDIA5TTechnologyUserManualv10/Real+Time+Clock > > > There are some jitters / unexpected outliers which I attribute to > measurement uncertainties / syscall jitter. > 99% of what you're seeing is likely PCIe read latency jitter. This is the > reason why phc2sys has the '-N' option to mitigate the problem: > > -N phc-num > > Specify the number of master clock readings per one slave clock update. > > Only the fastest reading is used to update the slave clock, this is > useful > > to minimize the error caused by random delays in scheduling and bus > utilization. > > W > -- Cheers Prankur
_______________________________________________ Linuxptp-users mailing list Linuxptp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-users
