bearophile wrote:
While allocating lot of memory for a little memory-hungry program, I have found
results that I don't understand. So I have written the following test programs.
Maybe someone can give me some information on the matter.
I am using a default install of a 32 bit Win XP with 2 GB RAM (so for example I
can't allocate 3 GB of RAM). (I presume answers to my questions are
Windows-related).
From C (MinGW 4.2.1) this is about the largest memory block I can allocate
(even it swaps and requires 7+ seconds to run), 1_920_000_000 bytes:
#include "stdio.h"
#include "stdlib.h"
#define N 480000000
int main() {
unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned int));
unsigned int i;
if (a != NULL)
for (i = 0; i < N; ++i)
a[i] = i;
else
printf("null!");
return 0;
}
But from D this is about the largest memory block I can allocate with
std.c.stdlib.malloc, 1_644_000_000 bytes, do you know why the difference?
//import std.gc: malloc;
import std.c.stdlib: malloc;
void main() {
const uint N = 411_000_000;
uint* a = cast(uint*)malloc(N * uint.sizeof);
if (a !is null)
for (uint i; i < N; ++i)
a[i] = i;
else
printf("null!");
}
(If I use std.gc.malloc the situation is different yet, and generally worse).
-----------------------
So I have tried to use a sequence of smaller memory blocks, this is the C code
(every block is about 1 MB):
#include "stdio.h"
#include "stdlib.h"
#define N 250000
int main(int argc, char** argv) {
unsigned int i, j;
unsigned int m = argc == 2 ? atoi(argv[1]) : 100;
for (j = 0; j < m; ++j) {
unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned int));
if (a != NULL) {
for (i = 0; i < N; ++i)
a[i] = i;
} else {
printf("null! %d\n", j);
break;
}
}
return 0;
}
And the D code:
//import std.gc: malloc;
import std.c.stdlib: malloc;
import std.conv: toUint;
void main(string[] args) {
const uint N = 250_000;
uint m = toUint(args[1]);
for (uint j; j < m; ++j) {
uint* a = cast(uint*)malloc(N * uint.sizeof);
if (a !is null) {
for (uint i; i < N; ++i)
a[i] = i;
} else {
printf("null! %d\n", j);
break;
}
}
}
With such code I can allocate 1_708_000_000 bytes from D and up to
2_038_000_000 bytes from C (but near the last 100-200 MB of RAM the C code
swaps a lot).
So can't I can't use all my RAM from my D code? And do you know why?
Bye,
bearophile
bearophile wrote:
> While allocating lot of memory for a little memory-hungry program, I
have found results that I don't understand. So I have written the
following test programs. Maybe someone can give me some information on
the matter.
> I am using a default install of a 32 bit Win XP with 2 GB RAM (so for
example I can't allocate 3 GB of RAM). (I presume answers to my
questions are Windows-related).
>
> From C (MinGW 4.2.1) this is about the largest memory block I can
allocate (even it swaps and requires 7+ seconds to run), 1_920_000_000
bytes:
>
> #include "stdio.h"
> #include "stdlib.h"
> #define N 480000000
> int main() {
> unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned int));
> unsigned int i;
> if (a != NULL)
> for (i = 0; i < N; ++i)
> a[i] = i;
> else
> printf("null!");
> return 0;
> }
>
>
> But from D this is about the largest memory block I can allocate with
std.c.stdlib.malloc, 1_644_000_000 bytes, do you know why the difference?
>
> //import std.gc: malloc;
> import std.c.stdlib: malloc;
> void main() {
> const uint N = 411_000_000;
> uint* a = cast(uint*)malloc(N * uint.sizeof);
> if (a !is null)
> for (uint i; i < N; ++i)
> a[i] = i;
> else
> printf("null!");
> }
>
> (If I use std.gc.malloc the situation is different yet, and generally
worse).
>
> -----------------------
>
> So I have tried to use a sequence of smaller memory blocks, this is
the C code (every block is about 1 MB):
>
> #include "stdio.h"
> #include "stdlib.h"
>
> #define N 250000
>
> int main(int argc, char** argv) {
> unsigned int i, j;
> unsigned int m = argc == 2 ? atoi(argv[1]) : 100;
>
> for (j = 0; j < m; ++j) {
> unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned
int));
>
> if (a != NULL) {
> for (i = 0; i < N; ++i)
> a[i] = i;
> } else {
> printf("null! %d\n", j);
> break;
> }
> }
>
> return 0;
> }
>
>
> And the D code:
>
> //import std.gc: malloc;
> import std.c.stdlib: malloc;
> import std.conv: toUint;
>
> void main(string[] args) {
> const uint N = 250_000;
> uint m = toUint(args[1]);
>
> for (uint j; j < m; ++j) {
> uint* a = cast(uint*)malloc(N * uint.sizeof);
>
> if (a !is null) {
> for (uint i; i < N; ++i)
> a[i] = i;
> } else {
> printf("null! %d\n", j);
> break;
> }
> }
> }
>
> With such code I can allocate 1_708_000_000 bytes from D and up to
2_038_000_000 bytes from C (but near the last 100-200 MB of RAM the C
code swaps a lot).
> So can't I can't use all my RAM from my D code? And do you know why?
>
> Bye,
> bearophile
Different allocation schemes have different strengths and weaknesses.
Some are fast, some fragment less, some have less overhead, some allow
larger sized blocks. Often these things arn't mutual so there are
always tradoffs. For example, to improve speed an allocator may
allocate into particular buckets which might restrict the maximum size
of one allocation.
I wonder how Ned-Malloc or Hord perform with your tests?
-Joel
