Re: slow realloc: alternate method?

2006-06-17 Thread Graham Toal
Yup, I used this in (function splitfields) where the delimiter was chosen with getopt: http://etudiant.epitech.net/~veins/sort/sort.c Oh yes, sort... that reminds me... http://www.gtoal.com/wordgames/sort/sort.[ch] - see the above for the epitome of managing store yourself... It's

slow realloc: alternate method?

2006-06-16 Thread Jacob Yocom-Piatt
i've got some C code that is reading from a 800 MB CSV file and allocates memory for an array to store the data in. the method used is to read the CSV file line-by-line and realloc additional space with each line read. having timed this and found the realloc speed to be low when the array is

Re: slow realloc: alternate method?

2006-06-16 Thread Dimitry Andric
Jacob Yocom-Piatt wrote: an alternative to this procedure would be to scan through the CSV file to determine how many array entries i would need, realloc it all at once, then go back through the CSV file again to read the data into the array. Try starting with a reasonable number of lines,

Re: slow realloc: alternate method?

2006-06-16 Thread Otto Moerbeek
On Fri, 16 Jun 2006, Jacob Yocom-Piatt wrote: i've got some C code that is reading from a 800 MB CSV file and allocates memory for an array to store the data in. the method used is to read the CSV file line-by-line and realloc additional space with each line read. having timed this and

Re: slow realloc: alternate method?

2006-06-16 Thread veins
On Fri, Jun 16, 2006 at 10:14:07PM +0200, Otto Moerbeek wrote: On Fri, 16 Jun 2006, Jacob Yocom-Piatt wrote: i've got some C code that is reading from a 800 MB CSV file and allocates memory for an array to store the data in. the method used is to read the CSV file line-by-line and

Re: slow realloc: alternate method?

2006-06-16 Thread veins
On Fri, Jun 16, 2006 at 10:40:29PM +0200, [EMAIL PROTECTED] wrote: On Fri, Jun 16, 2006 at 10:14:07PM +0200, Otto Moerbeek wrote: On Fri, 16 Jun 2006, Jacob Yocom-Piatt wrote: i've got some C code that is reading from a 800 MB CSV file and allocates memory for an array to store the

Re: slow realloc: alternate method?

2006-06-16 Thread Otto Moerbeek
On Fri, 16 Jun 2006, [EMAIL PROTECTED] wrote: So basically, why not mmap() the file, go through the map counting \n while replacing them by \0 until you reach end of map. allocate an array the size of the counter and have each array entry point to where it should in the memory map ? I

Re: slow realloc: alternate method?

2006-06-16 Thread veins
On Fri, Jun 16, 2006 at 11:17:39PM +0200, Otto Moerbeek wrote: On Fri, 16 Jun 2006, [EMAIL PROTECTED] wrote: So basically, why not mmap() the file, go through the map counting \n while replacing them by \0 until you reach end of map. allocate an array the size of the counter and have each

Re: slow realloc: alternate method?

2006-06-16 Thread Jacob Yocom-Piatt
Original message Date: Fri, 16 Jun 2006 22:14:07 +0200 (CEST) From: Otto Moerbeek [EMAIL PROTECTED] Subject: Re: slow realloc: alternate method? To: Jacob Yocom-Piatt [EMAIL PROTECTED] Cc: misc@openbsd.org So make your increment larger and start with a larger size. Maybe you can

Re: slow realloc: alternate method?

2006-06-16 Thread Matthew R. Dempsky
On Fri, Jun 16, 2006 at 10:55:05AM -0500, Jacob Yocom-Piatt wrote: the current code uses realloc in the manner suggested by the manpage: newsize = size + 1; time(t1); // start timing realloc if ((newap = (int *)realloc(ap, newsize*sizeof(int))) == NULL) {