Re: Ninja build failures on sgi and mips64el
Thanks for the patch; it's in now. On Mon, Jul 01, 2013 at 02:48:45PM -0700, Matthew Dempsky wrote: Jasper (or someone) will have to. My OpenBSD workstation is down until at least tomorrow. Glad to hear that fix worked though. On Mon, Jul 1, 2013 at 1:40 PM, David Coppa dco...@gmail.com wrote: On Mon, Jul 1, 2013 at 10:37 PM, Jasper Lievisse Adriaanse jas...@openbsd.org wrote: This fixes the build on mips64. Thanks a lot, Mathew. Please go ahead and commit it, you or Jasper ;) Ciao, David On Mon, Jul 01, 2013 at 10:36:09AM -0700, Matthew Dempsky wrote: There's an unaligned word access in ninja's murmurhash implementation: https://github.com/martine/ninja/blob/master/src/hash_map.h We need to change unsigned int k = *(unsigned int *)data; to unsigned int k; memcpy(k, data, sizeof k); and it should fix the unaligned accesses on strict arches like sparc64 and mips. Sorry, I'm not able to fix this myself right now, but I can help upstream the fix tomorrow. (I'm of course ok with any ports fix for this.) On Mon, Jul 1, 2013 at 10:24 AM, James Turner jtur...@openbsd.org wrote: Mathew, I just wanted to let you know we have discovered some build failures on sgi and mips64el during the Build ninja using itself phase. It looks like something is segfaulting. Below is the build output although it isn't very helpful. I can test diffs on mips64el and plan on trying to compile with debug symbols to see if I can get more info out of the core file that is left. Building ninja using itself... *** Error 246 in devel/ninja (Makefile:32 'do-build': @cd /usr/obj/ports/ninja-1.3.4/ninja-1.3.4 /usr/bin/env -i CXX=c++CC=cc PYTHONUS...) *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2634'/usr/obj/ports/ninja-1.3.4/.build_done') *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2367 'build') === Exiting devel/ninja with an error /bin/sh: exit 1: not found *** Error 127 in /usr/ports (infrastructure/mk/bsd.port.subdir.mk:147'build') Error: /usr/ports/packages/mips64el/all/ninja-1.3.4.tgz does not exist -- James Turner -- Regards, Jasper Lievisse Adriaanse, Engineering team M:tier -- Cheers, Jasper Stay Hungry. Stay Foolish
Re: Ninja build failures on sgi and mips64el
Following patch should be vaguely saner, though it may modify the hash
a bit.
(note that memcpy destroys any semblance of meaning of using murmurhash2
in any case, so murmurhash2 is complete bullshit in this case)
--- src/hash_map.h.orig Tue Jun 4 20:47:31 2013
+++ src/hash_map.h Tue Jul 2 11:17:20 2013
@@ -25,6 +25,24 @@ unsigned int MurmurHash2(const void* key, size_t len)
const int r = 24;
unsigned int h = seed ^ len;
const unsigned char * data = (const unsigned char *)key;
+ const unsigned char * b = data;
+ switch(reinterpret_castsize_t(data) 3) {
+ case 1:
+ h ^= b[2] 16;
+ data++;
+ len--;
+ case 2:
+ h ^= b[1] 8;
+ data++;
+ len--;
+ case 3:
+ h ^= b[0];
+ h *= m;
+ data++;
+ len--;
+ break;
+ }
+
while (len = 4) {
unsigned int k = *(unsigned int *)data;
k *= m;
Re: Ninja build failures on sgi and mips64el
Evan Martin, if you're not already aware of that, murmurhash fails
spectacularly on anything with strict alignment.
Idea stolen from siphash
btw, it looks like using siphash instead of murmurhash might be a good
idea. At least THOSE guys know how to write portable code, contrary to
the google dude who wrote murmur hash:
https://131002.net/siphash/
Just grepping for murmurhash, I see it's also used for 64 bit data,
so use the same modification there (there's a chance that data is 32 bit
aligned, but I'm not THAT sure about that)
Index: Makefile
===
RCS file: /cvs/ports/devel/ninja/Makefile,v
retrieving revision 1.4
diff -u -p -r1.4 Makefile
--- Makefile2 Jul 2013 06:37:24 - 1.4
+++ Makefile2 Jul 2013 14:33:38 -
@@ -3,7 +3,7 @@
COMMENT = small build system with a focus on speed
V =1.3.4
DISTNAME = ninja-${V}
-REVISION = 0
+REVISION = 1
CATEGORIES = devel
HOMEPAGE = http://martine.github.io/ninja/
MAINTAINER = Matthew Dempsky matt...@dempsky.org
Index: patches/patch-src_build_log_cc
===
RCS file: patches/patch-src_build_log_cc
diff -N patches/patch-src_build_log_cc
--- /dev/null 1 Jan 1970 00:00:00 -
+++ patches/patch-src_build_log_cc 2 Jul 2013 14:33:38 -
@@ -0,0 +1,50 @@
+$OpenBSD$
+--- src/build_log.cc.orig Tue Jun 4 20:47:31 2013
src/build_log.cc Tue Jul 2 18:30:01 2013
+@@ -54,26 +54,34 @@ uint64_t MurmurHash64A(const void* key, size_t len) {
+ const uint64_t m = BIG_CONSTANT(0xc6a4a7935bd1e995);
+ const int r = 47;
+ uint64_t h = seed ^ (len * m);
+- const uint64_t * data = (const uint64_t *)key;
+- const uint64_t * end = data + (len/8);
+- while (data != end) {
++ const unsigned char* data = (const unsigned char *)key;
++ while (len 8) {
+ uint64_t k = *data++;
++ uint64_t(data[0]) |
++ (uint64_t(data[1]) 8) |
++ (uint64_t(data[2]) 16) |
++ (uint64_t(data[3]) 24) |
++ (uint64_t(data[4]) 32) |
++ (uint64_t(data[5]) 40) |
++ (uint64_t(data[6]) 48) |
++ (uint64_t(data[7]) 56);
+ k *= m;
+ k ^= k r;
+ k *= m;
+ h ^= k;
++data += 8;
++len -= 8;
+ h *= m;
+ }
+- const unsigned char* data2 = (const unsigned char*)data;
+- switch (len 7)
++ switch (len)
+ {
+- case 7: h ^= uint64_t(data2[6]) 48;
+- case 6: h ^= uint64_t(data2[5]) 40;
+- case 5: h ^= uint64_t(data2[4]) 32;
+- case 4: h ^= uint64_t(data2[3]) 24;
+- case 3: h ^= uint64_t(data2[2]) 16;
+- case 2: h ^= uint64_t(data2[1]) 8;
+- case 1: h ^= uint64_t(data2[0]);
++ case 7: h ^= uint64_t(data[6]) 48;
++ case 6: h ^= uint64_t(data[5]) 40;
++ case 5: h ^= uint64_t(data[4]) 32;
++ case 4: h ^= uint64_t(data[3]) 24;
++ case 3: h ^= uint64_t(data[2]) 16;
++ case 2: h ^= uint64_t(data[1]) 8;
++ case 1: h ^= uint64_t(data[0]);
+ h *= m;
+ };
+ h ^= h r;
Index: patches/patch-src_hash_map_h
===
RCS file: /cvs/ports/devel/ninja/patches/patch-src_hash_map_h,v
retrieving revision 1.1
diff -u -p -r1.1 patch-src_hash_map_h
--- patches/patch-src_hash_map_h2 Jul 2013 06:37:24 - 1.1
+++ patches/patch-src_hash_map_h2 Jul 2013 14:33:38 -
@@ -2,15 +2,18 @@ $OpenBSD: patch-src_hash_map_h,v 1.1 201
Work-around unaligned accesses on strict arches such as sparc64/mips64(el).
src/hash_map.h.origMon Jul 1 22:42:26 2013
-+++ src/hash_map.h Mon Jul 1 22:42:54 2013
-@@ -26,7 +26,8 @@ unsigned int MurmurHash2(const void* key, size_t len)
+--- src/hash_map.h.origTue Jun 4 20:47:31 2013
src/hash_map.h Tue Jul 2 18:22:43 2013
+@@ -26,7 +26,11 @@ unsigned int MurmurHash2(const void* key, size_t len)
unsigned int h = seed ^ len;
const unsigned char * data = (const unsigned char *)key;
while (len = 4) {
-unsigned int k = *(unsigned int *)data;
-+unsigned int k;
-+memcpy(k, data, sizeof k);
++unsigned int k =
++ (unsigned int)(data[0]) |
++ ((unsigned int)(data[1]) 8) |
++ ((unsigned int)(data[2]) 16) |
++ ((unsigned int)(data[3]) 24);
k *= m;
k ^= k r;
k *= m;
Re: Ninja build failures on sgi and mips64el
There's an unaligned word access in ninja's murmurhash implementation: https://github.com/martine/ninja/blob/master/src/hash_map.h We need to change unsigned int k = *(unsigned int *)data; to unsigned int k; memcpy(k, data, sizeof k); and it should fix the unaligned accesses on strict arches like sparc64 and mips. Sorry, I'm not able to fix this myself right now, but I can help upstream the fix tomorrow. (I'm of course ok with any ports fix for this.) On Mon, Jul 1, 2013 at 10:24 AM, James Turner jtur...@openbsd.org wrote: Mathew, I just wanted to let you know we have discovered some build failures on sgi and mips64el during the Build ninja using itself phase. It looks like something is segfaulting. Below is the build output although it isn't very helpful. I can test diffs on mips64el and plan on trying to compile with debug symbols to see if I can get more info out of the core file that is left. Building ninja using itself... *** Error 246 in devel/ninja (Makefile:32 'do-build': @cd /usr/obj/ports/ninja-1.3.4/ninja-1.3.4 /usr/bin/env -i CXX=c++CC=cc PYTHONUS...) *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2634'/usr/obj/ports/ninja-1.3.4/.build_done') *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2367 'build') === Exiting devel/ninja with an error /bin/sh: exit 1: not found *** Error 127 in /usr/ports (infrastructure/mk/bsd.port.subdir.mk:147'build') Error: /usr/ports/packages/mips64el/all/ninja-1.3.4.tgz does not exist -- James Turner
Re: Ninja build failures on sgi and mips64el
This fixes the build on mips64. On Mon, Jul 01, 2013 at 10:36:09AM -0700, Matthew Dempsky wrote: There's an unaligned word access in ninja's murmurhash implementation: https://github.com/martine/ninja/blob/master/src/hash_map.h We need to change unsigned int k = *(unsigned int *)data; to unsigned int k; memcpy(k, data, sizeof k); and it should fix the unaligned accesses on strict arches like sparc64 and mips. Sorry, I'm not able to fix this myself right now, but I can help upstream the fix tomorrow. (I'm of course ok with any ports fix for this.) On Mon, Jul 1, 2013 at 10:24 AM, James Turner jtur...@openbsd.org wrote: Mathew, I just wanted to let you know we have discovered some build failures on sgi and mips64el during the Build ninja using itself phase. It looks like something is segfaulting. Below is the build output although it isn't very helpful. I can test diffs on mips64el and plan on trying to compile with debug symbols to see if I can get more info out of the core file that is left. Building ninja using itself... *** Error 246 in devel/ninja (Makefile:32 'do-build': @cd /usr/obj/ports/ninja-1.3.4/ninja-1.3.4 /usr/bin/env -i CXX=c++CC=cc PYTHONUS...) *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2634'/usr/obj/ports/ninja-1.3.4/.build_done') *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2367 'build') === Exiting devel/ninja with an error /bin/sh: exit 1: not found *** Error 127 in /usr/ports (infrastructure/mk/bsd.port.subdir.mk:147'build') Error: /usr/ports/packages/mips64el/all/ninja-1.3.4.tgz does not exist -- James Turner -- Regards, Jasper Lievisse Adriaanse, Engineering team M:tier
Re: Ninja build failures on sgi and mips64el
On Mon, Jul 1, 2013 at 10:37 PM, Jasper Lievisse Adriaanse jas...@openbsd.org wrote: This fixes the build on mips64. Thanks a lot, Mathew. Please go ahead and commit it, you or Jasper ;) Ciao, David On Mon, Jul 01, 2013 at 10:36:09AM -0700, Matthew Dempsky wrote: There's an unaligned word access in ninja's murmurhash implementation: https://github.com/martine/ninja/blob/master/src/hash_map.h We need to change unsigned int k = *(unsigned int *)data; to unsigned int k; memcpy(k, data, sizeof k); and it should fix the unaligned accesses on strict arches like sparc64 and mips. Sorry, I'm not able to fix this myself right now, but I can help upstream the fix tomorrow. (I'm of course ok with any ports fix for this.) On Mon, Jul 1, 2013 at 10:24 AM, James Turner jtur...@openbsd.org wrote: Mathew, I just wanted to let you know we have discovered some build failures on sgi and mips64el during the Build ninja using itself phase. It looks like something is segfaulting. Below is the build output although it isn't very helpful. I can test diffs on mips64el and plan on trying to compile with debug symbols to see if I can get more info out of the core file that is left. Building ninja using itself... *** Error 246 in devel/ninja (Makefile:32 'do-build': @cd /usr/obj/ports/ninja-1.3.4/ninja-1.3.4 /usr/bin/env -i CXX=c++CC=cc PYTHONUS...) *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2634'/usr/obj/ports/ninja-1.3.4/.build_done') *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2367 'build') === Exiting devel/ninja with an error /bin/sh: exit 1: not found *** Error 127 in /usr/ports (infrastructure/mk/bsd.port.subdir.mk:147'build') Error: /usr/ports/packages/mips64el/all/ninja-1.3.4.tgz does not exist -- James Turner -- Regards, Jasper Lievisse Adriaanse, Engineering team M:tier
Re: Ninja build failures on sgi and mips64el
Jasper (or someone) will have to. My OpenBSD workstation is down until at least tomorrow. Glad to hear that fix worked though. On Mon, Jul 1, 2013 at 1:40 PM, David Coppa dco...@gmail.com wrote: On Mon, Jul 1, 2013 at 10:37 PM, Jasper Lievisse Adriaanse jas...@openbsd.org wrote: This fixes the build on mips64. Thanks a lot, Mathew. Please go ahead and commit it, you or Jasper ;) Ciao, David On Mon, Jul 01, 2013 at 10:36:09AM -0700, Matthew Dempsky wrote: There's an unaligned word access in ninja's murmurhash implementation: https://github.com/martine/ninja/blob/master/src/hash_map.h We need to change unsigned int k = *(unsigned int *)data; to unsigned int k; memcpy(k, data, sizeof k); and it should fix the unaligned accesses on strict arches like sparc64 and mips. Sorry, I'm not able to fix this myself right now, but I can help upstream the fix tomorrow. (I'm of course ok with any ports fix for this.) On Mon, Jul 1, 2013 at 10:24 AM, James Turner jtur...@openbsd.org wrote: Mathew, I just wanted to let you know we have discovered some build failures on sgi and mips64el during the Build ninja using itself phase. It looks like something is segfaulting. Below is the build output although it isn't very helpful. I can test diffs on mips64el and plan on trying to compile with debug symbols to see if I can get more info out of the core file that is left. Building ninja using itself... *** Error 246 in devel/ninja (Makefile:32 'do-build': @cd /usr/obj/ports/ninja-1.3.4/ninja-1.3.4 /usr/bin/env -i CXX=c++CC=cc PYTHONUS...) *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2634'/usr/obj/ports/ninja-1.3.4/.build_done') *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2367 'build') === Exiting devel/ninja with an error /bin/sh: exit 1: not found *** Error 127 in /usr/ports (infrastructure/mk/bsd.port.subdir.mk:147'build') Error: /usr/ports/packages/mips64el/all/ninja-1.3.4.tgz does not exist -- James Turner -- Regards, Jasper Lievisse Adriaanse, Engineering team M:tier
Re: Ninja build failures on sgi and mips64el
On Mon, Jul 01, 2013 at 10:37:57PM +0200, Jasper Lievisse Adriaanse wrote: This fixes the build on mips64. mips64el as well. On Mon, Jul 01, 2013 at 10:36:09AM -0700, Matthew Dempsky wrote: There's an unaligned word access in ninja's murmurhash implementation: https://github.com/martine/ninja/blob/master/src/hash_map.h We need to change unsigned int k = *(unsigned int *)data; to unsigned int k; memcpy(k, data, sizeof k); and it should fix the unaligned accesses on strict arches like sparc64 and mips. Sorry, I'm not able to fix this myself right now, but I can help upstream the fix tomorrow. (I'm of course ok with any ports fix for this.) On Mon, Jul 1, 2013 at 10:24 AM, James Turner jtur...@openbsd.org wrote: Mathew, I just wanted to let you know we have discovered some build failures on sgi and mips64el during the Build ninja using itself phase. It looks like something is segfaulting. Below is the build output although it isn't very helpful. I can test diffs on mips64el and plan on trying to compile with debug symbols to see if I can get more info out of the core file that is left. Building ninja using itself... *** Error 246 in devel/ninja (Makefile:32 'do-build': @cd /usr/obj/ports/ninja-1.3.4/ninja-1.3.4 /usr/bin/env -i CXX=c++CC=cc PYTHONUS...) *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2634'/usr/obj/ports/ninja-1.3.4/.build_done') *** Error 1 in devel/ninja (/usr/ports/infrastructure/mk/bsd.port.mk:2367 'build') === Exiting devel/ninja with an error /bin/sh: exit 1: not found *** Error 127 in /usr/ports (infrastructure/mk/bsd.port.subdir.mk:147'build') Error: /usr/ports/packages/mips64el/all/ninja-1.3.4.tgz does not exist -- James Turner -- Regards, Jasper Lievisse Adriaanse, Engineering team M:tier -- James Turner
