Re: [PATCH] mmap CRC checking 1.2cvs
Jean-Marc Lasgouttes wrote:
>Ben, it seems to me that you sent a patch, but I lost it. Could you
>re-send?
>
Here it is.
I'd appreciate it if people on strange platforms could test it out and
report any problems to me, thanks. I think this one should resolve the
compile problems we had on Solaris and compaq cxx, and satisfy the
regulations of the style police...
Ben.
--- lyx-devel-orig/src/support/ChangeLogThu Dec 6 12:28:42 2001
+++ lyx-devel/src/support/ChangeLog Wed Dec 12 09:53:19 2001
@@ -1,3 +1,7 @@
+2001-12-12 Ben Stanley <[EMAIL PROTECTED]>
+
+ * lyxsum.C: portability fix for mmap patch
+
2001-12-05 Lars Gullik Bjønnes <[EMAIL PROTECTED]>
* filetools.C:
--- lyx-devel-orig/src/support/lyxsum.C Thu Dec 6 12:28:42 2001
+++ lyx-devel/src/support/lyxsum.C Wed Dec 12 09:52:15 2001
@@ -23,6 +23,10 @@
#warning lyx::sum() using mmap (lightning fast)
#endif
+// Make sure we get modern version of mmap and friends with void*,
+// not `compatibility' version with caddr_t.
+#define _POSIX_C_SOURCE 199506L
+
#include
#include
#include
@@ -40,7 +44,8 @@
void * mm = mmap(0, info.st_size, PROT_READ,
MAP_PRIVATE, fd, 0);
- if (mm == MAP_FAILED) {
+ // Some platforms have the wrong type for MAP_FAILED (compaq cxx).
+ if (mm == reinterpret_cast(MAP_FAILED)) {
close(fd);
return 0;
}
Re: [PATCH] mmap CRC checking 1.2cvs
> "Ben" == Ben Stanley <[EMAIL PROTECTED]> writes: Ben> Thanks for the reference - I think it shows the _POSIX_C_SOURCE Ben> macro is the right thing to do to get the correct prototype. Ben, it seems to me that you sent a patch, but I lost it. Could you re-send? JMarc
Re: [PATCH] mmap CRC checking 1.2cvs
On Tue, Dec 11, 2001 at 04:23:00PM +0100, Jean-Marc Lasgouttes wrote: > Is it possible to do a reintepret_cast which would work always? C-style casts are sometimes the bigger stick... > Something like > if (reinterpret_cast(mm) == reinterpret_cast(MAP_FAILED)) > or is C++ stupid enough to reject this when one type is already void*? No, you can reinterpret everything as itself. And it should even give a sensible result ;-) Andre' -- André Pönitz .. [EMAIL PROTECTED]
Re: [PATCH] mmap CRC checking 1.2cvs
> "Lars" == Lars Gullik Bjønnes <[EMAIL PROTECTED]> writes: Lars> Jean-Marc Lasgouttes <[EMAIL PROTECTED]> writes: | Ben> So Lars> what kind of patch are you looking for? Some autoconf magic to | Lars> Ben> somehow detect a bad type for MAP_FAILED and to deal with Lars> it | Ben> regardless of platform? >> Lars> | Is it possible to do a reintepret_cast which would work Lars> always? | Something like | if (reinterpret_cast(mm) == Lars> reinterpret_cast(MAP_FAILED)) | or is C++ stupid enough Lars> to reject this when one type is already void*? Lars> Or we can perhaps just define our own MAP_FAILED. Lars> #undef MAP_FAILED #define MAP_FAILED (void*)(-1) Are we sure -1 is always the right value? JMarc
Re: [PATCH] mmap CRC checking 1.2cvs
> "Ben" == Ben Stanley <[EMAIL PROTECTED]> writes: Ben> Please refresh my memory... on this platform, MAP_FAILED is Ben> defined to be (-1L)? Yes. I meant to make that clear, but forgot. Ben> With the _POSIX_C_SOURCE define, we should get void* from mmap Ben> and others... so it is trying to compare void* with long. So the Ben> only solution left is a reinterpret_cast to get around the bad Ben> header on that platform. Indeed. Ben> I think that the _POSIX_C_SOURCE is the right thing to do - the Ben> MAP_FAILED thing is a separate problem. I am not sure about that, but I believe you :) However, until we are sure we have no complaint from HP-UX, AIX SCO or whatever people, I'd rather not put it in 1.1.6. Ben> So what kind of patch are you looking for? Some autoconf magic to Ben> somehow detect a bad type for MAP_FAILED and to deal with it Ben> regardless of platform? Is it possible to do a reintepret_cast which would work always? Something like if (reinterpret_cast(mm) == reinterpret_cast(MAP_FAILED)) or is C++ stupid enough to reject this when one type is already void*? We could also use good old C casts (since this is a C matter, after all), if Lars agrees. It seems that now that the problem is somewhat understood, a brutal cast (with a comment explaining why it is needed) may be enough and better than convoluted configure things. JMarc Ben> Thanks for the reference - I think it shows the _POSIX_C_SOURCE Ben> macro is the right thing to do to get the correct prototype. We already define _ALL_C_SOURCE in config.h in some cases. Is it also useful here? JMArc
Re: [PATCH] mmap CRC checking 1.2cvs
Jean-Marc Lasgouttes wrote:
>>"Ben" == Ben Stanley <[EMAIL PROTECTED]> writes:
>>
>
>Ben> Jean-Marc, Can you please check if the #define _POSIX_C_SOURCE
>Ben> 199506L
>
>Ben> modification would fix this problem on compaq?
>
>It does not work.
>
Please refresh my memory... on this platform, MAP_FAILED is defined to
be (-1L)?
With the _POSIX_C_SOURCE define, we should get void* from mmap and
others... so it is trying to compare void* with long.
So the only solution left is a reinterpret_cast to get around the bad
header on that platform.
I think that the _POSIX_C_SOURCE is the right thing to do - the
MAP_FAILED thing is a separate problem.
So what kind of patch are you looking for? Some autoconf magic to
somehow detect a bad type for MAP_FAILED and to deal with it regardless
of platform?
ie try to compile from configure
#define _POSIX_C_SOURCE 199506L
#include
int main()
{
void*p;
return p == MAP_FAILED;
}
If that works then #define HAVE_OK_MAP_FAILED
and then at the top of lyxsum.C
#ifdef HAVE_OK_MAP_FAILED
#define CAST_MAP(x) x
#else
#define CAST_MAP(x) reinterpret_cast(x)
#endif
and in the code
if (mm == CAST_MAP(MAP_FAILED)) {
// fail code...
}
Seems a lot easier to just do everywhere (because it will work everywhere)
if (mm == reinterpret_cast(MAP_FAILED)) {
// fail code...
}
Thanks for the reference - I think it shows the _POSIX_C_SOURCE macro is
the right thing to do to get the correct prototype.
Ben.
Re: [PATCH] mmap CRC checking 1.2cvs
> "Ben" == Ben Stanley <[EMAIL PROTECTED]> writes: Ben> Jean-Marc, Can you please check if the #define _POSIX_C_SOURCE Ben> 199506L Ben> modification would fix this problem on compaq? It does not work. For reference, here is the relevant part of sys/mman.h: #if ( defined(_POSIX_C_SOURCE) && (_POSIX_C_SOURCE >= 199309L) ) || defined(_XOP EN_SOURCE_EXTENDED) extern void *mmap __((void *, size_t, int, int, int, off_t)); extern int munmap __((void *, size_t)); #ifdef _XOPEN_SOURCE_EXTENDED extern int mprotect __((void *, size_t, int)); #else extern int mprotect __((const void *, size_t, int)); #endif extern int msync __((void *, size_t, int)); #if defined(_POSIX_C_SOURCE) && (_POSIX_C_SOURCE >= 199309L) extern int shm_open __((const char *, int, mode_t)); extern int shm_unlink __((const char *)); #endif /* _POSIX_C_SOURCE */ #else /* not (_POSIX_C_SOURCE || _XOPEN_SOURCE_EXTENDED) */ extern caddr_t mmap __((caddr_t addr, size_t len, int prot, int flags, int file des, off_t off)); extern int munmap __((caddr_t addr, size_t len)); extern int mprotect __((caddr_t addr, size_t len, int prot)); extern int msync __((caddr_t addr, size_t len, int flags)); #endif /* _POSIX_C_SOURCE || _XOPEN_SOURCE_EXTENDED */ Ben> I'd also appreciate seeing the standards(5) man page from that Ben> platform. Attached. JMarc PS: this probably means I will release 1.1.6fix4 without this patch. It is a bit too unportable for my taste. standards.txt.gz Description: Binary data
Re: [PATCH] mmap CRC checking 1.2cvs
Jean-Marc,
Can you please check if the
#define _POSIX_C_SOURCE 199506L
modification would fix this problem on compaq?
I'd also appreciate seeing the standards(5) man page from that platform.
Ben.
Jean-Marc Lasgouttes wrote:
>Compilation of 1.2.0cvs with compaq cxx dies with:
>
>cxx: Error: ../../../lyx-devel/src/support/lyxsum.C, line 43: operand types
> are incompatible ("void *" and "long")
>if (mm == MAP_FAILED) {
>---^
>
Re: [PATCH] mmap CRC checking 1.2cvs
On Fri, Dec 07, 2001 at 09:29:38AM +1100, Ben Stanley wrote: > The Solaris headers can be told to give us a POSIX compliant definition > of munmap by doing > #define _POSIX_C_SOURCE 199506L > before > #include > > This seems to be fairly standard - see [...] > > So, I think that the _POSIX_C_SOURCE define may fix both problems, but I > would like to see authorative documentation. > > I think it is probably safe to define this macro unconditionally for > this file because we only need mmap here, and on Linux it has no effect. This seems fine to me. I have added just the _POSIX_C_SOURCE definition to the lyxsum.C file and it compiles fine on Solaris. -- Kayvan A. Sylvan | Proud husband of | Father to my kids: Sylvan Associates, Inc. | Laura Isabella Sylvan | Katherine Yelena (8/8/89) http://sylvan.com/~kayvan | "crown of her husband" | Robin Gregory (2/28/92)
Re: [PATCH] mmap CRC checking 1.2cvs
Lars Gullik Bjønnes wrote: >"Kayvan A. Sylvan" <[EMAIL PROTECTED]> writes: > >| On Thu, Dec 06, 2001 at 08:39:47AM +0100, Lars Gullik Bjønnes wrote: > >>>Ben Stanley <[EMAIL PROTECTED]> writes: >>> >>>| Please let me know if the attached patch fixes this problem. >>> >>>| I also tried to fix the Solaris compile problem - Kayvan, could you >>>| please test it? >>> >>>I am pretty sure that this is not how we are going to solve this >>>problem. >>> >| I don't like adding the POSIX level hack either. > Hmmm... I think that this is the right approach, but what I sent in was a hack. See below. > >| How about this? (it compiles for me on Solaris). > >No, won't do. > No, I don't like that either. That won't compile elsewhere. >Either disable mmap for those platforms of find a better solution. > Hmmm... let's try for the better solution. It seems that we have 2 distinct problems: 1) Solaris headers by default provide an old version of munmap with the wrong type in there. 2) compaq provides a definition of MAP_FAILED with a similarly wrong type. The Solaris headers can be told to give us a POSIX compliant definition of munmap by doing #define _POSIX_C_SOURCE 199506L before #include This seems to be fairly standard - see http://vicente.org/xclib/include/xc/standards.h.html http://www.cs.arizona.edu/computer.help/policy/DIGITAL_unix/AA-QTLMA-TE_html/relnotes5.html Also mentioned at http://mail.gnu.org/pipermail/autoconf/2000-August/006020.html, but this suggests problems with defining this unconditionally on Solaris... This solution would mean that we only have to support POSIX mmap, instead of several non-standard variants. This removes #ifdef and reinterpret_cast clutter from the code, which I consider to be a great advantage. It would be nice to have access to some man pages for the platforms in question. The compaq headers may have a similar feature. The definition that Jean-Marc looked up is also a compatibility defintion. The compq man page suggested looking at standards(5) - Jean-Marc, could you please take a look at this, and perhaps send it to me? Specifically, does it mention _POSIX_C_SOURCE? So, I think that the _POSIX_C_SOURCE define may fix both problems, but I would like to see authorative documentation. I think it is probably safe to define this macro unconditionally for this file because we only need mmap here, and on Linux it has no effect. Ben.
Re: [PATCH] mmap CRC checking 1.2cvs
> "Kayvan" == Kayvan A Sylvan <[EMAIL PROTECTED]> writes: Kayvan> On Thu, Dec 06, 2001 at 08:39:47AM +0100, Lars Gullik Bjønnes Kayvan> wrote: >> Ben Stanley <[EMAIL PROTECTED]> writes: >> >> | Please let me know if the attached patch fixes this problem. >> > >> | I also tried to fix the Solaris compile problem - Kayvan, could >> you | please test it? >> >> I am pretty sure that this is not how we are going to solve this >> problem. Kayvan> I don't like adding the POSIX level hack either. I cannot test it right now, but I suspect that on cxx the caddr_t cast will fail (since the right type seems to be void*). JMarc
Re: [PATCH] mmap CRC checking 1.2cvs
On Thu, Dec 06, 2001 at 08:39:47AM +0100, Lars Gullik Bjønnes wrote:
> Ben Stanley <[EMAIL PROTECTED]> writes:
>
> | Please let me know if the attached patch fixes this problem.
> >
> | I also tried to fix the Solaris compile problem - Kayvan, could you
> | please test it?
>
> I am pretty sure that this is not how we are going to solve this
> problem.
I don't like adding the POSIX level hack either.
How about this? (it compiles for me on Solaris).
Index: lyxsum.C
===
RCS file: /cvs/lyx/lyx-devel/src/support/lyxsum.C,v
retrieving revision 1.20
diff -u -r1.20 lyxsum.C
--- lyxsum.C2001/12/03 01:11:05 1.20
+++ lyxsum.C2001/12/06 16:52:22
@@ -40,7 +40,7 @@
void * mm = mmap(0, info.st_size, PROT_READ,
MAP_PRIVATE, fd, 0);
- if (mm == MAP_FAILED) {
+ if (mm == reinterpret_cast(MAP_FAILED)) {
close(fd);
return 0;
}
@@ -52,7 +52,7 @@
crc.process_block(beg, end);
unsigned long result = crc.checksum();
- munmap(mm, info.st_size);
+ munmap((caddr_t) mm, info.st_size);
close(fd);
return result;
Re: [PATCH] mmap CRC checking 1.2cvs
> "Jean-Marc" == Jean-Marc Lasgouttes <[EMAIL PROTECTED]> writes: Jean-Marc> RETURN VALUES Jean-Marc> Upon successful completion, the mmap() function returns Jean-Marc> the address at which the mapping was placed. Otherwise, Jean-Marc> mmap() returns (caddr_t)-1 and sets errno to indicate the Jean-Marc> error. Actually in sysmman.h, I find /usr/include/sys/mman.h:#define MAP_FAILED (-1L) /* unsuccessful return from mmap() */ So the reinterpret_cast is probably the right solution here. UNfortunately, I cannot apply the patch just to try it out, because I am over a telnet connection. JMarc PS: sorry for posting the whole man page. This was not the intent.
Re: [PATCH] mmap CRC checking 1.2cvs
> "Lars" == Lars Gullik Bjønnes <[EMAIL PROTECTED]> writes: Lars> I am pretty sure that this is not how we are going to solve this Lars> problem. Lars> What I need to know is how this is _supposed_ to be done on cxx, Lars> and then we can find the correct fix. Well, mmap() returns a void * #include void *mmap ( void *addr, size_t len, int prot, int flags, int filedes, off_t off ); When there is an error, here is what it does: [Digital] The following definition of the mmap() function does not conform to current standards and is supported only for backward compatibility (see standards(5)): caddr_t mmap ( caddr_t addr, size_t len, int prot, int flags, int filedes, off_t off ); STANDARDS Interfaces documented on this reference page conform to industry standards as follows: mmap(): XPG4-UNIX Refer to the standards(5) reference page for more information about indus- try standards and associated tags. Standards: standards(5) PARAMETERS addr Specifies the starting address of the new region (truncated to a page boundary). len Specifies the length in bytes of the new region (rounded up to a page boundary). prot Specifies access permissions as either PROT_NONE or the result of a logical OR operation on any combination of PROT_READ, PROT_WRITE, and PROT_EXEC. flags Specifies attributes of the mapped region as the results of a bitwise-inclusive OR operation on any combination of MAP_FILE, MAP_ANONYMOUS, MAP_VARIABLE, MAP_FIXED, MAP_SHARED, MAP_PRIVATE, MAP_INHERIT, or MAP_UNALIGNED . filedes Specifies the file to be mapped to the new mapped file region returned by open(). off Specifies the offset into the file that gets mapped at address addr. DESCRIPTION The mmap() function creates a new mapped file region, a new private region, or a new shared memory region. The addr and len parameters specify the requested starting address and length in bytes for the new region. The address is a multiple of the page size returned by sysconf(_SC_PAGE_SIZE). If the len parameter is not a multiple of the page size returned by sysconf(_SC_PAGE_SIZE), then the result of any reference to an address between the end of the region and the end of the page containing the end of the region is undefined. The flags parameter specifies attributes of the mapped region. Values of the flags parameter are constructed by a bitwise-inclusive OR operation on the flags from the following list of symbolic names defined in the sys/mman.h file: MAP_FILE Create a mapped file region. MAP_ANONYMOUS Create an unnamed memory region. MAP_VARIABLE Place region at the computed address. MAP_FIXEDPlace region at fixed address. MAP_SHARED Share changes. MAP_PRIVATE Changes are private. MAP_INHERIT Region not unmapped by exec(2). MAP_UNALIGNED Do not verify that the file offset is page aligned. The MAP_FILE and MAP_ANONYMOUS flags control whether the region to be mapped is a mapped file region or an anonymous shared memory region. One of these flags must be selected. If MAP_FILE is set in the flags parameter: o+ A new mapped file region is created, mapping the file associated with the filedes parameter. o+ The off parameter specifies the file byte offset at which the mapping starts. This offset must be a multiple of the page size returned by sysconf(_SC_PAGE_SIZE). o+ If the end of the mapped file region is beyond the end of the file, the result of any reference to an address in the mapped file region corresponding to an offset beyond the end of the file is unspecified. If MAP_ANONYMOUS is set in the flags parameter: o+ A new memory region is created and initialized to all zeros. This memory region can be shared only with descendents of the current pro- cess. o+ If the filedes parameter is not -1, the mmap() function fails. The new region is placed at the requested address if the requested address is not null and it is possible to place the region at this address. When the requested address is null or the region cannot be placed at the requested address, the MAP_VARIABLE and MAP_FIXED flags control the place- ment of the region. One of these flags must be selected. If MAP_VARIABLE is set in the flags parameter: o+ If the requested address is null or if it is not possible for the sys- tem to place the region at the requested address, the region is placed at an address selected by the system. If MAP_FIXED is set in the flags parameter: o+ If the requested address is not null, the mmap() function succeeds even
Re: [PATCH] mmap CRC checking 1.2cvs
Please let me know if the attached patch fixes this problem.
I also tried to fix the Solaris compile problem - Kayvan, could you
please test it?
I still need to get the proper DepTable patch out. The patch that is
currently in 1.2 is b0rken.
Ben.
Jean-Marc Lasgouttes wrote:
>>"Ben" == Ben Stanley <[EMAIL PROTECTED]> writes:
>>
>
>Ben> This is a version of the mmap patch which applies to cvs HEAD.
>
>Compilation of 1.2.0cvs with compaq cxx dies with:
>
>cxx: Error: ../../../lyx-devel/src/support/lyxsum.C, line 43: operand types
> are incompatible ("void *" and "long")
>if (mm == MAP_FAILED) {
>---^
>
>I guess some cast is needed.
>
>This probably mean that this patch is too risky to go into 1.1.6fix4,
>unfortunately.
>
>JMarc
>
--- lyx-devel-orig/src/support/ChangeLogThu Dec 6 12:28:42 2001
+++ lyx-devel/src/support/ChangeLog Thu Dec 6 12:35:48 2001
@@ -1,3 +1,7 @@
+2001-12-06 Ben Stanley <[EMAIL PROTECTED]>
+
+ * lyxsum.C: small compaq cxx and Solaris fixes for mmap.
+
2001-12-05 Lars Gullik Bjønnes <[EMAIL PROTECTED]>
* filetools.C:
--- lyx-devel-orig/src/support/lyxsum.C Thu Dec 6 12:28:42 2001
+++ lyx-devel/src/support/lyxsum.C Thu Dec 6 12:35:39 2001
@@ -27,6 +27,7 @@
#include
#include
#include
+#define _POSIX_C_SOURCE 3
#include
unsigned long lyx::sum(string const & file)
@@ -40,7 +41,7 @@
void * mm = mmap(0, info.st_size, PROT_READ,
MAP_PRIVATE, fd, 0);
- if (mm == MAP_FAILED) {
+ if (mm == reinterpret_cast(MAP_FAILED)) {
close(fd);
return 0;
}
Re: [PATCH] mmap CRC checking 1.2cvs
> "Ben" == Ben Stanley <[EMAIL PROTECTED]> writes:
Ben> This is a version of the mmap patch which applies to cvs HEAD.
Compilation of 1.2.0cvs with compaq cxx dies with:
cxx: Error: ../../../lyx-devel/src/support/lyxsum.C, line 43: operand types
are incompatible ("void *" and "long")
if (mm == MAP_FAILED) {
---^
I guess some cast is needed.
This probably mean that this patch is too risky to go into 1.1.6fix4,
unfortunately.
JMarc
Re: [PATCH] mmap CRC checking 1.2cvs
Lars Gullik Bjønnes wrote:
>I think you should submit the complete patch again.
>
Complete patch attached.
process_block does lead to a significant improvement.
Warm cache
ben [11:26:45]
/share/install/linux/extras/lyx/lyx-HEAD/lyx-devel/src/support/tests $
time crccheck.do_crc /share/mp3/bassic/EI.mp3
CRC: 3089166751
real0m0.448s
user0m0.450s
sys 0m0.000s
ben [11:26:49]
/share/install/linux/extras/lyx/lyx-HEAD/lyx-devel/src/support/tests $
time crccheck.process_block /share/mp3/bassic/EI.mp3
CRC: 3089166751
real0m0.150s
user0m0.130s
sys 0m0.020s
ben [11:39:13] /share/install/linux/extras/lyx/lyx-HEAD $ ls -al
/share/mp3/bassic/EI.mp3
-rw-r--r--1 ben users 5661289 Nov 5 08:45
/share/mp3/bassic/EI.mp3
--- lyx-devel/src/support/lyxsum.C.orig.cvs Mon Dec 3 09:20:46 2001
+++ lyx-devel/src/support/lyxsum.C Mon Dec 3 11:29:17 2001
@@ -10,8 +10,6 @@
#include
-#include
-#include
#include
#include
@@ -31,23 +29,88 @@
} // namespace
-
-// And this would be the file interface.
-unsigned long lyx::sum(string const & file)
+// Various implementations of lyx::sum(), depending on what methods
+// are available. Order is faster to slowest.
+#if defined(HAVE_MMAP) && defined(HAVE_MUNMAP)
+ #ifdef WITH_WARNINGS
+ #warning lyx::sum() using mmap (lightning fast)
+ #endif
+
+ #include
+ #include
+ #include
+ #include
+ #include
+
+ unsigned long lyx::sum(string const & file)
+ {
+ int fd = open(file.c_str(), O_RDONLY);
+ if( !fd ) return 0;
+
+ struct stat info;
+ fstat(fd, &info);
+
+ void * mm = mmap(0, info.st_size, PROT_READ,
+MAP_PRIVATE, fd, 0);
+ if (mm == MAP_FAILED) {
+ close(fd);
+ return 0;
+ }
+
+ char *beg = static_cast(mm);
+ char *end = beg + info.st_size;
+
+ boost::crc_32_type crc;
+ crc.process_block(beg,end);
+ unsigned long result = crc.checksum();
+
+ munmap( mm, info.st_size );
+ close(fd);
+
+ return result;
+ }
+#else // No mmap
+ #include
+ #include
+
+ #if HAVE_DECL_ISTREAMBUF_ITERATOR
+ #ifdef WITH_WARNINGS
+ #warning lyx::sum() using istreambuf_iterator (fast)
+ #endif
+ unsigned long lyx::sum(string const & file)
+ {
+ std::ifstream ifs(file.c_str());
+ if (!ifs) return 0;
+
+ std::istreambuf_iterator beg(ifs);
+ std::istreambuf_iterator end;
+
+ return do_crc(beg,end);
+ }
+ #else
+ #ifdef WITH_WARNINGS
+ #warning lyx::sum() using istream_iterator (slow as a snail)
+ #endif
+ unsigned long lyx::sum(string const & file)
+ {
+ std::ifstream ifs(file.c_str());
+ if (!ifs) return 0;
+
+ ifs.unsetf(std::ios::skipws);
+ std::istream_iterator beg(ifs);
+ std::istream_iterator end;
+
+ return do_crc(beg,end);
+ }
+ #endif
+#endif // mmap
+
+#if 0
+#include
+int main(int /*argc*/, char * argv[])
{
- std::ifstream ifs(file.c_str());
- if (!ifs) return 0;
-
-#ifdef HAVE_DECL_ISTREAMBUF_ITERATOR
- // This is a lot faster...
- std::istreambuf_iterator beg(ifs);
- std::istreambuf_iterator end;
-#else
- // than this.
- ifs.unsetf(std::ios::skipws);
- std::istream_iterator beg(ifs);
- std::istream_iterator end;
-#endif
+ std::string const fil(argv[1]);
- return do_crc(beg, end);
+ std::cout << "CRC: " << lyx::sum(fil) << std::endl;
}
+#endif
--- lyx-devel/src/support/ChangeLog.origMon Dec 3 11:32:13 2001
+++ lyx-devel/src/support/ChangeLog Mon Dec 3 11:36:32 2001
@@ -1,3 +1,8 @@
+2001-12-03 Ben Stanley <[EMAIL PROTECTED]>
+
+ * lyxsum.C: Added mmap version of CRC and made it selected
+ by default where available. Used process_block for crc for speedup.
+
2001-12-01 John Levon <[EMAIL PROTECTED]>
* filetools.C: more robust failure for DirList()
Re: [PATCH] mmap CRC checking 1.2cvs
Lars Gullik Bjønnes wrote: >Ben Stanley <[EMAIL PROTECTED]> writes: > >| +char *beg = static_cast(mm); >| +char *end = beg + info.st_size; >| + >| +unsigned long result = do_crc(beg,end); > >Did you check the speed difference between this and crc.process_block? > No, I haven't - yet. I didn't investigate boost/crc.hpp. I'll investigate crc.process_block, but it looks like the right way to go for mmap-ed streams. BTW, I just did cvs up and noticed that patch wasn't applied. I thought it had gone in. Please let me know if I should re-submit the whole patch with the process_block stuff in it or just the process_block mod. Ben.
[PATCH] mmap CRC checking 1.2cvs
This is a version of the mmap patch which applies to cvs HEAD.
--- lyx-devel-orig/src/support/lyxsum.C Fri Jun 1 22:10:06 2001
+++ lyx-devel/src/support/lyxsum.C Thu Nov 29 02:31:01 2001
@@ -10,8 +10,6 @@
#include
-#include
-#include
#include
#include
@@ -31,23 +29,85 @@
} // namespace
-
-// And this would be the file interface.
-unsigned long lyx::sum(string const & file)
+// Various implementations of lyx::sum(), depending on what methods
+// are available. Order is faster to slowest.
+#if defined(HAVE_MMAP) && defined(HAVE_MUNMAP)
+ #ifdef WITH_WARNINGS
+ #warning lyx::sum() using mmap (lightning fast but untested)
+ #endif
+
+ #include
+ #include
+ #include
+ #include
+ #include
+
+ unsigned long lyx::sum(string const & file)
+ {
+ int fd = open(file.c_str(), O_RDONLY);
+ if( !fd ) return 0;
+
+ struct stat info;
+ fstat(fd, &info);
+
+ void * mm = mmap(0, info.st_size, PROT_READ,
+MAP_PRIVATE, fd, 0);
+ if (mm == MAP_FAILED) {
+ close(fd);
+ return 0;
+ }
+ char *beg = static_cast(mm);
+ char *end = beg + info.st_size;
+
+ unsigned long result = do_crc(beg,end);
+
+ munmap( mm, info.st_size );
+ close(fd);
+
+ return result;
+ }
+#else // No mmap
+ #include
+ #include
+
+ #if HAVE_DECL_ISTREAMBUF_ITERATOR
+ #ifdef WITH_WARNINGS
+ #warning lyx::sum() using istreambuf_iterator (fast)
+ #endif
+ unsigned long lyx::sum(string const & file)
+ {
+ std::ifstream ifs(file.c_str());
+ if (!ifs) return 0;
+
+ std::istreambuf_iterator beg(ifs);
+ std::istreambuf_iterator end;
+
+ return do_crc(beg,end);
+ }
+ #else
+ #ifdef WITH_WARNINGS
+ #warning lyx::sum() using istream_iterator (slow as a snail)
+ #endif
+ unsigned long lyx::sum(string const & file)
+ {
+ std::ifstream ifs(file.c_str());
+ if (!ifs) return 0;
+
+ ifs.unsetf(std::ios::skipws);
+ std::istream_iterator beg(ifs);
+ std::istream_iterator end;
+
+ return do_crc(beg,end);
+ }
+ #endif
+#endif // mmap
+
+#if 0
+#include
+int main(int /*argc*/, char * argv[])
{
- std::ifstream ifs(file.c_str());
- if (!ifs) return 0;
-
-#ifdef HAVE_DECL_ISTREAMBUF_ITERATOR
- // This is a lot faster...
- std::istreambuf_iterator beg(ifs);
- std::istreambuf_iterator end;
-#else
- // than this.
- ifs.unsetf(std::ios::skipws);
- std::istream_iterator beg(ifs);
- std::istream_iterator end;
-#endif
+ std::string const fil(argv[1]);
- return do_crc(beg, end);
+ std::cout << "CRC: " << lyx::sum(fil) << std::endl;
}
+#endif
--- lyx-devel-orig/src/support/ChangeLogThu Nov 29 03:01:18 2001
+++ lyx-devel/src/support/ChangeLog Thu Nov 29 05:38:31 2001
@@ -1,3 +1,8 @@
+2001-11-29 Ben Stanley <[EMAIL PROTECTED]>
+
+ * lyxsum.C: Added mmap version of CRC and made it selected
+ by default where available.
+
2001-11-28 André Pönitz <[EMAIL PROTECTED]>
* Makefile.am: put types.h in
