Dear Arun, thank you for the report. I agree with the analysis, detectCores() will only report logical processors in the NUMA group in which R is running. I don't have a system to test on, could you please check these workarounds for me on your systems?
# number of logical processors - what detectCores() should return out <- system("wmic cpu get numberoflogicalprocessors", intern=TRUE) sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))) # number of cores - what detectCores(FALSE) should return out <- system("wmic cpu get numberofcores", intern=TRUE) sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))) # number of physical processors - as a sanity check system("wmic computersystem get numberofprocessors") Thanks, Tomas On 08/17/2018 05:11 PM, Srinivasan, Arunkumar wrote: > Dear R-devel list, > > R's detectCores() function internally calls "ncpus" function to get the total > number of logical processors. However, this doesnot seem to take NUMA into > account on Windows machines. > > On a machine having 48 processors (24 cores) in total and windows server 2012 > installed, if NUMA is enabled and has 2 nodes (node 0 and node 1 each having > 24 CPUs), then R's detectCores() only detects 24 instead of the total 48. If > NUMA is disabled, detectCores() returns 48. > > Similarly, on a machine with 88 cores (176 processors) and windows server > 2012, detectCores() with NUMA disabled only returns the maximum value of 64. > If NUMA is enabled with 4 nodes (44 processors each), then detectCores() will > only return 44. This is particularly limiting since we cannot get to use all > processors by enabling/disabling NUMA in this case. > > We think this is because R's ncpus.c file uses > "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION" > (https://msdn.microsoft.com/en-us/library/windows/desktop/ms683194(v=vs.85).aspx) > instead of "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX" > (https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx). > Specifically, quoting from the first link: > > "On systems with more than 64 logical processors, the > GetLogicalProcessorInformation function retrieves logical processor > information about processors in the > https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx > to which the calling thread is currently assigned. Use the > https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx > function to retrieve information about processors in all processor groups on > the system." > > Therefore, it might be possible to get the right count of total processors > even with NUMA enabled by using "GetLogicalProcessorInformationEX". It'd be > nice to know what you think. > > Thank you very much, > Arun. > > -- > Arun Srinivasan > Analyst, Millennium Management LLC > 50 Berkeley Street | London, W1J 8HD > > > ###################################################################### > > The information contained in this communication is con...{{dropped:11}} ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel