My previous message described problems with running out of temporary
files on HP-UX when sorting a 200MB file and a fix for that. But the
original task should not have needed those temporary files since it
was running on a machine with 8GB of physical ram. Why wasn't sort
detecting that correctly? It should have been able to handle the
entire file easily. This message concerns detecting memory on hpux.
It turns out that physical memory was not be determined and compile
time values of 64MB were being used. Unfortunately HP-UX does not
define _SC_PHYS_PAGES. (Although it does define _SC_PAGESIZE.)
Therefore the physmem.c code using sysconf() was unable to determine
the actual amount of ram and used a default value of 64MB instead.
Which worked out to 8MB for sort because the code uses 1/8 of physical
ram for the sort buffer size. A reasonable heuristic.
In order to get the desired information on hpux the pstat_get*()
interface is used instead. Here is a patch against the
textutils-2.0.14 sources for physmem.c which implements this
functionality using pstat_getdynamic(). This is used only if the
sysconf interface does not exist and instead of the hard compiled in
value and only if available.
Note that HAVE_PSTAT_GETDYNAMIC is already tested for in configure
somewhere (jm_MACROS?) and used in getloadavg.c and so nothing else
needed to change. Otherwise would need to be provided with
AC_CHECK_FUNCS(pstat_getdynamic) or some such.
Some explaination of what I think is a subtle point in the provided
code. Note that HP-UX 11.0 and later is 64-bit capable but still
fully supports prior 32-bit code. Which means it is possible to have
more memory than a 32-bit address space allows. Care must be taken to
avoid 32-bit overflow. This also needs to operate properly when
compiled in a 64-bit address space. LONG_MAX / 2 produces the desired
1GB max ram for normal shared 32-bit executables and also does not
limit LP64 executables. (Non-shared executables can get 2GB in the
32-bit address space. And there are other techniques. But that is
not special techniques and not normal for the GNU utilities.)
Providing real memory values enable sort to use reasonable buffer
sizes and to sort entirely in core. In most cases temporary files can
be avoided entirely now where previously 8MB buffers and temporary
files were used. This makes a significant performance increase.
Bob
--- ../textutils-2.0.14.original/lib/physmem.c Mon Dec 18 16:43:51 2000
+++ ./lib/physmem.c Thu Oct 25 14:43:38 2001
@@ -27,6 +27,11 @@
# include <unistd.h>
#endif
+#if defined HAVE_PSTAT_GETDYNAMIC
+# include <sys/pstat.h>
+# include <limits.h>
+#endif
+
/* Return the total amount of physical memory. */
double
physmem_total (void)
@@ -38,6 +43,16 @@
return pages * pagesize;
#endif
+#if defined HAVE_PSTAT_GETDYNAMIC
+ struct pst_static pss;
+ if (pstat_getstatic (&pss, sizeof pss, 1, 0) >= 0)
+ {
+ if (pss.physical_memory > (LONG_MAX / 2) / pss.page_size)
+ return LONG_MAX / 2; /* More ram than address space. */
+ return pss.physical_memory * pss.page_size;
+ }
+#endif
+
/* Guess 64 MB. It's probably an older host, so guess small. */
return 64 * 1024 * 1024;
}
@@ -51,6 +66,18 @@
double pagesize = sysconf (_SC_PAGESIZE);
if (0 <= pages && 0 <= pagesize)
return pages * pagesize;
+#endif
+
+#if defined HAVE_PSTAT_GETDYNAMIC
+ struct pst_static pss;
+ struct pst_dynamic psd;
+ if (pstat_getstatic (&pss, sizeof pss, 1, 0) >= 0
+ && pstat_getdynamic (&psd, sizeof psd, 1, 0) >= 0)
+ {
+ if (psd.psd_free > (LONG_MAX / 2) / pss.page_size)
+ return LONG_MAX / 2; /* More ram than address space. */
+ return psd.psd_free * pss.page_size;
+ }
#endif
/* Guess 25% of physical memory. */
_______________________________________________
Bug-textutils mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/bug-textutils
