Re: mmap mapped segment length
Wes Peters wrote: [...] Thanks to everyone who responded, and for the politeness of the responses in the face of the stupidity of the original question. People are usually more polite when they are, err, having fun. ;- -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org - Come on. - Where are we going? - To get what you came for. - What's that? - Me. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: mmap mapped segment length
Matthew Dillon wrote: :Now I've got to go figure out what *I've* screwed up. I fstat the file before :mapping it and pass S.st_size as the map length. Is there any reason why :mmap would return non-NULL but map less than the requested size? : :Scratching my head, Note that mmap() returns (void *)-1 when an error occurs, *not* NULL. This is because it is legal to mmap at address 0. Uh, yeah, the program source actually checks against MAP_FAILED. I do usually read the man pages when I write code. ;^) I found the problem, it was caused by unexpected input. Doh! The program is now working, and is on the 44th (of 78) 60 megabyte input files. Good thing the REAL system is a dual PII Xeon 450, huh? Maybe I should add a -j jobs option so I can make sure we use that other processor. As I/O bound as this task is, I'm not sure it would help. Thanks to everyone who responded, and for the politeness of the responses in the face of the stupidity of the original question. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC http://softweyr.com/ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mmap mapped segment length
Matthew Dillon wrote: :Now I've got to go figure out what *I've* screwed up. I fstat the file before :mapping it and pass S.st_size as the map length. Is there any reason why :mmap would return non-NULL but map less than the requested size? : :Scratching my head, Note that mmap() returns (void *)-1 when an error occurs, *not* NULL. This is because it is legal to mmap at address 0. Uh, yeah, the program source actually checks against MAP_FAILED. I do usually read the man pages when I write code. ;^) I found the problem, it was caused by unexpected input. Doh! The program is now working, and is on the 44th (of 78) 60 megabyte input files. Good thing the REAL system is a dual PII Xeon 450, huh? Maybe I should add a -j jobs option so I can make sure we use that other processor. As I/O bound as this task is, I'm not sure it would help. Thanks to everyone who responded, and for the politeness of the responses in the face of the stupidity of the original question. -- Where am I, and what am I doing in this handbasket? Wes Peters Softweyr LLC http://softweyr.com/ w...@softweyr.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: mmap mapped segment length
On Aug 21, 2:10am, Wes Peters wrote: } Subject: mmap mapped segment length } I discovered to my dismay today that the length field in the mmap call is } a size_t, not an off_t. I was attempting to process a large (~50 MByte) file } and found I was only processing the first 4 MBytes of it. 50 MB should comfortably fit in a size_t. } Is this intentional, or just an artifact of the implementation? Is there any } reason NOT to change this to an off_t? The type of size_t is supposed to be large enough to express the length of any object that will fit in the virtual address space of a process. Since a size_t is 32 bits on an i386 and pointers are also 32 bits, there wouldn't be any advantage to changing mmap() to use a 64 bit wide length parameter, since you wouldn't be able to access all of such a large object. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mmap mapped segment length
Don Lewis said: On Aug 21, 2:10am, Wes Peters wrote: } Subject: mmap mapped segment length } I discovered to my dismay today that the length field in the mmap call is } a size_t, not an off_t. I was attempting to process a large (~50 MByte) file } and found I was only processing the first 4 MBytes of it. 50 MB should comfortably fit in a size_t. It seems that 4MB and 4GB are being confused :-). John To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mmap mapped segment length
On Sat, 21 Aug 1999 02:10:47 -0600 Wes Peters [EMAIL PROTECTED] wrote: I discovered to my dismay today that the length field in the mmap call is a size_t, not an off_t. I was attempting to process a large (~50 MByte) file and found I was only processing the first 4 MBytes of it. ...first of all, I assume you mean GByte, not MByte. The type of the "len" argument is specified by multiple standards. You could change size_t to a 64-bit quantity, but then you have: (1) A serious ABI incompatibility issue with the rest of the x86 world. (2) You'd have to go to 64-bit arithmetic everywhere in the VM code which could seriously impact performance. Is this intentional, or just an artifact of the implementation? Is there any reason NOT to change this to an off_t? -- Jason R. Thorpe [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mmap mapped segment length
Don Lewis wrote: On Aug 21, 2:10am, Wes Peters wrote: } Subject: mmap mapped segment length } I discovered to my dismay today that the length field in the mmap call is } a size_t, not an off_t. I was attempting to process a large (~50 MByte) file } and found I was only processing the first 4 MBytes of it. 50 MB should comfortably fit in a size_t. Oh, duh. Notice the time on my message? Yeah, that's what I get for jumping into a mailing list when I've been up for way too long. } Is this intentional, or just an artifact of the implementation? Is there any } reason NOT to change this to an off_t? The type of size_t is supposed to be large enough to express the length of any object that will fit in the virtual address space of a process. Since a size_t is 32 bits on an i386 and pointers are also 32 bits, there wouldn't be any advantage to changing mmap() to use a 64 bit wide length parameter, since you wouldn't be able to access all of such a large object. Obviously. I'm not sure which memory space my HEAD was in last night, the moment I read your message I realized size_t spans 4 GB. Duh. Sorry to embarass myself in such a public manner. Now I've got to go figure out what *I've* screwed up. I fstat the file before mapping it and pass S.st_size as the map length. Is there any reason why mmap would return non-NULL but map less than the requested size? Scratching my head, -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC http://softweyr.com/ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mmap mapped segment length
:Wes Peters scribbled this message on Aug 21: : Now I've got to go figure out what *I've* screwed up. I fstat the file before : mapping it and pass S.st_size as the map length. Is there any reason why : mmap would return non-NULL but map less than the requested size? : :no, there is NO reason why it wouldn't map the entire 50meg file... :I have used ffsrecov w/ a 1.8gig file where I fstat it and map the entire :file w/o a problem... : :also, mmap doesn't return NULL, it returns MAP_FAILED... : :the code for ffsrecov might give you some hints to make sure you aren't :doing anything wrong... : :-- : John-Mark Gurney Voice: +1 541 684 8449 : Cu Networking P.O. Box 5693, 97405 I would compile the program -Wall -Wstrict-prototypes. If you get warnings maybe the problem is that you've missed some important #include files. Many system calls take off_t arguments, including mmap, and without the correct prototype the compiler will not generate a good argument stack. mmap'ing a 50MB file definitely works. I run several databases which use mmap() that are several hundred megabytes in size. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mmap mapped segment length
:Now I've got to go figure out what *I've* screwed up. I fstat the file before :mapping it and pass S.st_size as the map length. Is there any reason why :mmap would return non-NULL but map less than the requested size? : :Scratching my head, Note that mmap() returns (void *)-1 when an error occurs, *not* NULL. This is because it is legal to mmap at address 0. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mmap mapped segment length
On Aug 21, 2:10am, Wes Peters wrote: } Subject: mmap mapped segment length } I discovered to my dismay today that the length field in the mmap call is } a size_t, not an off_t. I was attempting to process a large (~50 MByte) file } and found I was only processing the first 4 MBytes of it. 50 MB should comfortably fit in a size_t. } Is this intentional, or just an artifact of the implementation? Is there any } reason NOT to change this to an off_t? The type of size_t is supposed to be large enough to express the length of any object that will fit in the virtual address space of a process. Since a size_t is 32 bits on an i386 and pointers are also 32 bits, there wouldn't be any advantage to changing mmap() to use a 64 bit wide length parameter, since you wouldn't be able to access all of such a large object. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: mmap mapped segment length
Don Lewis said: On Aug 21, 2:10am, Wes Peters wrote: } Subject: mmap mapped segment length } I discovered to my dismay today that the length field in the mmap call is } a size_t, not an off_t. I was attempting to process a large (~50 MByte) file } and found I was only processing the first 4 MBytes of it. 50 MB should comfortably fit in a size_t. It seems that 4MB and 4GB are being confused :-). John To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: mmap mapped segment length
On Sat, 21 Aug 1999 02:10:47 -0600 Wes Peters w...@softweyr.com wrote: I discovered to my dismay today that the length field in the mmap call is a size_t, not an off_t. I was attempting to process a large (~50 MByte) file and found I was only processing the first 4 MBytes of it. ...first of all, I assume you mean GByte, not MByte. The type of the len argument is specified by multiple standards. You could change size_t to a 64-bit quantity, but then you have: (1) A serious ABI incompatibility issue with the rest of the x86 world. (2) You'd have to go to 64-bit arithmetic everywhere in the VM code which could seriously impact performance. Is this intentional, or just an artifact of the implementation? Is there any reason NOT to change this to an off_t? -- Jason R. Thorpe thor...@nas.nasa.gov To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: mmap mapped segment length
Wes Peters scribbled this message on Aug 21: I discovered to my dismay today that the length field in the mmap call is a size_t, not an off_t. I was attempting to process a large (~50 MByte) file and found I was only processing the first 4 MBytes of it. as w/ others I'm assuming the file is 50gigs and you can only use map the first 4gigs... Is this intentional, or just an artifact of the implementation? Is there any reason NOT to change this to an off_t? Granted, I'm no VM hacker, but I'm willing to take a swing at it if there is any interest in putting it into the system. Otherwise, I have a workaround that works just fine for this application. I'm sure that you will find that if you try to map the first 4gigs of the file that it will not fit into memory, and the mapping will fail.. do what you are suppose to do, map 2gigs or so of it (I've mapped 1.8gigs w/o problems before on x86), work with it, then map the next 2gigs and so on... just use the offset to specify where in the file you want to map... -- John-Mark Gurney Voice: +1 541 684 8449 Cu Networking P.O. Box 5693, 97405 The soul contains in itself the event that shall presently befall it. The event is only the actualizing of its thought. -- Ralph Waldo Emerson To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: mmap mapped segment length
Don Lewis wrote: On Aug 21, 2:10am, Wes Peters wrote: } Subject: mmap mapped segment length } I discovered to my dismay today that the length field in the mmap call is } a size_t, not an off_t. I was attempting to process a large (~50 MByte) file } and found I was only processing the first 4 MBytes of it. 50 MB should comfortably fit in a size_t. Oh, duh. Notice the time on my message? Yeah, that's what I get for jumping into a mailing list when I've been up for way too long. } Is this intentional, or just an artifact of the implementation? Is there any } reason NOT to change this to an off_t? The type of size_t is supposed to be large enough to express the length of any object that will fit in the virtual address space of a process. Since a size_t is 32 bits on an i386 and pointers are also 32 bits, there wouldn't be any advantage to changing mmap() to use a 64 bit wide length parameter, since you wouldn't be able to access all of such a large object. Obviously. I'm not sure which memory space my HEAD was in last night, the moment I read your message I realized size_t spans 4 GB. Duh. Sorry to embarass myself in such a public manner. Now I've got to go figure out what *I've* screwed up. I fstat the file before mapping it and pass S.st_size as the map length. Is there any reason why mmap would return non-NULL but map less than the requested size? Scratching my head, -- Where am I, and what am I doing in this handbasket? Wes Peters Softweyr LLC http://softweyr.com/ w...@softweyr.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: mmap mapped segment length
Wes Peters scribbled this message on Aug 21: Now I've got to go figure out what *I've* screwed up. I fstat the file before mapping it and pass S.st_size as the map length. Is there any reason why mmap would return non-NULL but map less than the requested size? no, there is NO reason why it wouldn't map the entire 50meg file... I have used ffsrecov w/ a 1.8gig file where I fstat it and map the entire file w/o a problem... also, mmap doesn't return NULL, it returns MAP_FAILED... the code for ffsrecov might give you some hints to make sure you aren't doing anything wrong... -- John-Mark Gurney Voice: +1 541 684 8449 Cu Networking P.O. Box 5693, 97405 The soul contains in itself the event that shall presently befall it. The event is only the actualizing of its thought. -- Ralph Waldo Emerson To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: mmap mapped segment length
:Wes Peters scribbled this message on Aug 21: : Now I've got to go figure out what *I've* screwed up. I fstat the file before : mapping it and pass S.st_size as the map length. Is there any reason why : mmap would return non-NULL but map less than the requested size? : :no, there is NO reason why it wouldn't map the entire 50meg file... :I have used ffsrecov w/ a 1.8gig file where I fstat it and map the entire :file w/o a problem... : :also, mmap doesn't return NULL, it returns MAP_FAILED... : :the code for ffsrecov might give you some hints to make sure you aren't :doing anything wrong... : :-- : John-Mark Gurney Voice: +1 541 684 8449 : Cu Networking P.O. Box 5693, 97405 I would compile the program -Wall -Wstrict-prototypes. If you get warnings maybe the problem is that you've missed some important #include files. Many system calls take off_t arguments, including mmap, and without the correct prototype the compiler will not generate a good argument stack. mmap'ing a 50MB file definitely works. I run several databases which use mmap() that are several hundred megabytes in size. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: mmap mapped segment length
:Now I've got to go figure out what *I've* screwed up. I fstat the file before :mapping it and pass S.st_size as the map length. Is there any reason why :mmap would return non-NULL but map less than the requested size? : :Scratching my head, Note that mmap() returns (void *)-1 when an error occurs, *not* NULL. This is because it is legal to mmap at address 0. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
