Re: Issue with 'Unknown Error: -512'
On Mon, 18 Jul 2011, Andriy Gapon wrote: In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. Any tips? Otherwise I'll be looking into this today myself. Just a hint that you could try using DTrace syscall and fbt providers to see where in kernel (if in kernel) that -512 return value originates. Jon Anderson spotted that here during some Capsicum work -- initially we were concerned it was a local patch, but it sounds like it might be less local. I think he saw it on calls to open(2) as well, and I couldn't help but wonder (given its recent arrival) if it was an outcome of the change to break falloc into two parts, leading to some or another problematic handling of file descriptor numbers. I.e., it's not so much that -512 is being returned, as a number that's a bad file descriptor. (Although now having seen 512 twice on two different machines, that particular explanation seems less credible). Perhaps this is indeed unrelated to Capsicum, and triggered by a clang bug or something else. I've CC'd Jon, maybe he has gained further insight since we chatted. Robert ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
On 7/25/2011 4:47 AM, Robert Watson wrote: On Mon, 18 Jul 2011, Andriy Gapon wrote: In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. Any tips? Otherwise I'll be looking into this today myself. Just a hint that you could try using DTrace syscall and fbt providers to see where in kernel (if in kernel) that -512 return value originates. Jon Anderson spotted that here during some Capsicum work -- initially we were concerned it was a local patch, but it sounds like it might be less local. I think he saw it on calls to open(2) as well, and I couldn't help but wonder (given its recent arrival) if it was an outcome of the change to break falloc into two parts, leading to some or another problematic handling of file descriptor numbers. I.e., it's not so much that -512 is being returned, as a number that's a bad file descriptor. (Although now having seen 512 twice on two different machines, that particular explanation seems less credible). Perhaps this is indeed unrelated to Capsicum, and triggered by a clang bug or something else. I've CC'd Jon, maybe he has gained further insight since we chatted. Robert I've been building head every single day to check for the disappearance of this bug, and it seems to be gone as of 224302 (maybe before, but 224302 is what I built). I wrote a program to do tons of open()s and truncate()s to try to get the error, where initially I'd get an error every other attempt, then it was every 100 or so iterations it would fail, now I've tested 500,000 opens and truncates and there have been no issues. -Brandon Falk ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
On 7/19/2011 10:31 AM, Brandon Falk wrote: On 7/19/2011 5:14 AM, Dimitry Andric wrote: On 2011-07-18 16:53, Brandon Falk wrote: In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). I have never seen this, neither with clang-compiled nor gcc-compiled systems. Can you please verify if it still occurs if you build your system with gcc? If you don't want (or can) rebuild the whole system, you can try to rebuild just your kernel with gcc, and see if the problems disappear. Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. I have been running clang-compiled systems for a long time now, and have never seen this. It works fine here, I cannot reproduce any of your examples. Do you build with any special settings in make.conf or src.conf, particularly modified CFLAGS or COPTFLAGS? What is your architecture, i386 or amd64? Any other non-standard configuration or environment settings? Dimitry, I will try to build with gcc sometime today. I might as well try out a different filesystem too. I have no special settings besides CPUTYPE?=native, but there was someone with a similar error who did not specify a cpu type when building, so I know that isn't a problem. Arch is amd64, no other nonstandard configuration. -Brandon Falk ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org I just did a clang build with CPUTYPE?=native and the unknown error issue has seemed to calm down a large amount. I still occasionally get it, but given multiple attempts at say compiling software, it will move on and all will be well. It is for sure not happening every other O_TRUNC, but it is still happening. Depending on system configuration, this still could cause some major issues for people. I'll keep my eye out and see what changes and try to figure out how to duplicate the issue on the new rev. Rev: 224221 Built with clang and CPUTYPE?=native -Brandon Falk ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
on 19/07/2011 03:19 Brandon Falk said the following: On 7/18/2011 10:18 AM, Andriy Gapon wrote: on 18/07/2011 17:53 Brandon Falk said the following: Hello, In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. Any tips? Otherwise I'll be looking into this today myself. Just a hint that you could try using DTrace syscall and fbt providers to see where in kernel (if in kernel) that -512 return value originates. Update: I've traced more and more into the kernel, and currently I have found the following trace: ufs_setattr() (in sys/ufs/ufs/ufs_vnops.c:507) (In truncate a bunch of stuff is done, but it succeeds until VOP_SETATTR) vn_truncate() (in sys/kern/vfs_vnops.c:636) kern_openat() (in sys/kern/vfs_syscalls.c:1043) kern_open() (in sys/kern/vfs_syscalls.c:1035) open() (in sys/kern/vfs_syscalls.c:1006) ufs_setattr() returns with -1 (EPERM) But not -512, though. I'll continue to try to find the exact problem. A quick workaround currently would probably be to use a different filesystem other than ufs, but then again, I have no clue if other filesystems have the same issue. Hopefully that trace will help anyone who wants to help out. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
On 2011-07-18 16:53, Brandon Falk wrote: In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). I have never seen this, neither with clang-compiled nor gcc-compiled systems. Can you please verify if it still occurs if you build your system with gcc? If you don't want (or can) rebuild the whole system, you can try to rebuild just your kernel with gcc, and see if the problems disappear. Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. I have been running clang-compiled systems for a long time now, and have never seen this. It works fine here, I cannot reproduce any of your examples. Do you build with any special settings in make.conf or src.conf, particularly modified CFLAGS or COPTFLAGS? What is your architecture, i386 or amd64? Any other non-standard configuration or environment settings? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
On Mon, Jul 18, 2011 at 5:19 PM, Brandon Falk falk...@gamozo.org wrote: On 7/18/2011 10:18 AM, Andriy Gapon wrote: on 18/07/2011 17:53 Brandon Falk said the following: Hello, In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. Any tips? Otherwise I'll be looking into this today myself. Just a hint that you could try using DTrace syscall and fbt providers to see where in kernel (if in kernel) that -512 return value originates. Update: I've traced more and more into the kernel, and currently I have found the following trace: ufs_setattr() (in sys/ufs/ufs/ufs_vnops.c:507) (In truncate a bunch of stuff is done, but it succeeds until VOP_SETATTR) vn_truncate() (in sys/kern/vfs_vnops.c:636) kern_openat() (in sys/kern/vfs_syscalls.c:1043) kern_open() (in sys/kern/vfs_syscalls.c:1035) open() (in sys/kern/vfs_syscalls.c:1006) ufs_setattr() returns with -1 (EPERM) I'll continue to try to find the exact problem. A quick workaround currently would probably be to use a different filesystem other than ufs, but then again, I have no clue if other filesystems have the same issue. Hopefully that trace will help anyone who wants to help out. What UFS options do you have defined in your kernel? Thanks, -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
On Tue, Jul 19, 2011 at 4:58 AM, Garrett Cooper yaneg...@gmail.com wrote: On Mon, Jul 18, 2011 at 5:19 PM, Brandon Falk falk...@gamozo.org wrote: On 7/18/2011 10:18 AM, Andriy Gapon wrote: on 18/07/2011 17:53 Brandon Falk said the following: Hello, In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. Any tips? Otherwise I'll be looking into this today myself. Just a hint that you could try using DTrace syscall and fbt providers to see where in kernel (if in kernel) that -512 return value originates. Update: I've traced more and more into the kernel, and currently I have found the following trace: ufs_setattr() (in sys/ufs/ufs/ufs_vnops.c:507) (In truncate a bunch of stuff is done, but it succeeds until VOP_SETATTR) vn_truncate() (in sys/kern/vfs_vnops.c:636) kern_openat() (in sys/kern/vfs_syscalls.c:1043) kern_open() (in sys/kern/vfs_syscalls.c:1035) open() (in sys/kern/vfs_syscalls.c:1006) ufs_setattr() returns with -1 (EPERM) I'll continue to try to find the exact problem. A quick workaround currently would probably be to use a different filesystem other than ufs, but then again, I have no clue if other filesystems have the same issue. Hopefully that trace will help anyone who wants to help out. What UFS options do you have defined in your kernel? Also, what does mount say and have you tried running as root? Thanks, -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
On 7/19/2011 7:31 AM, Garrett Cooper wrote: On Tue, Jul 19, 2011 at 4:58 AM, Garrett Cooperyaneg...@gmail.com wrote: On Mon, Jul 18, 2011 at 5:19 PM, Brandon Falkfalk...@gamozo.org wrote: On 7/18/2011 10:18 AM, Andriy Gapon wrote: on 18/07/2011 17:53 Brandon Falk said the following: Hello, In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi'test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi'test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. Any tips? Otherwise I'll be looking into this today myself. Just a hint that you could try using DTrace syscall and fbt providers to see where in kernel (if in kernel) that -512 return value originates. Update: I've traced more and more into the kernel, and currently I have found the following trace: ufs_setattr() (in sys/ufs/ufs/ufs_vnops.c:507) (In truncate a bunch of stuff is done, but it succeeds until VOP_SETATTR) vn_truncate() (in sys/kern/vfs_vnops.c:636) kern_openat() (in sys/kern/vfs_syscalls.c:1043) kern_open() (in sys/kern/vfs_syscalls.c:1035) open() (in sys/kern/vfs_syscalls.c:1006) ufs_setattr() returns with -1 (EPERM) I'll continue to try to find the exact problem. A quick workaround currently would probably be to use a different filesystem other than ufs, but then again, I have no clue if other filesystems have the same issue. Hopefully that trace will help anyone who wants to help out. What UFS options do you have defined in your kernel? Also, what does mount say and have you tried running as root? Thanks, -Garrett Garrett, I am running as root. No issues with the mounting. As for options, which do you want? -Brandon Falk ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
On 7/19/2011 5:14 AM, Dimitry Andric wrote: On 2011-07-18 16:53, Brandon Falk wrote: In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). I have never seen this, neither with clang-compiled nor gcc-compiled systems. Can you please verify if it still occurs if you build your system with gcc? If you don't want (or can) rebuild the whole system, you can try to rebuild just your kernel with gcc, and see if the problems disappear. Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. I have been running clang-compiled systems for a long time now, and have never seen this. It works fine here, I cannot reproduce any of your examples. Do you build with any special settings in make.conf or src.conf, particularly modified CFLAGS or COPTFLAGS? What is your architecture, i386 or amd64? Any other non-standard configuration or environment settings? Dimitry, I will try to build with gcc sometime today. I might as well try out a different filesystem too. I have no special settings besides CPUTYPE?=native, but there was someone with a similar error who did not specify a cpu type when building, so I know that isn't a problem. Arch is amd64, no other nonstandard configuration. -Brandon Falk ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
on 18/07/2011 17:53 Brandon Falk said the following: Hello, In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. Any tips? Otherwise I'll be looking into this today myself. Just a hint that you could try using DTrace syscall and fbt providers to see where in kernel (if in kernel) that -512 return value originates. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Issue with 'Unknown Error: -512'
On 7/18/2011 10:18 AM, Andriy Gapon wrote: on 18/07/2011 17:53 Brandon Falk said the following: Hello, In recent branches (confirmed with 224119) builds compiled with clang happen to throw 'Unknown error: -512' in a lot of places, making the system unusable. (Untested on gcc compiled systems). Originally I thought the problem was with specific programs, then I narrowed it down to file I/O, and now I've narrowed it down to open() with O_TRUNC. Without O_TRUNC there seems to be no issues whatsoever. With O_TRUNC on open() it fails with that 'Unknown error: -512' every other time you run the program. Common issues, portsnap is affected, making it impossible to fetch/extract ports. As well as redirecting output in shells eg `echo 'hi' test` fails every other try. You have the same issue with text editors like `edit` where it fails every other save. There are no issues with `echo 'hi' test` as there is no O_TRUNC, it only seems to be an O_TRUNC error. Any tips? Otherwise I'll be looking into this today myself. Just a hint that you could try using DTrace syscall and fbt providers to see where in kernel (if in kernel) that -512 return value originates. Update: I've traced more and more into the kernel, and currently I have found the following trace: ufs_setattr() (in sys/ufs/ufs/ufs_vnops.c:507) (In truncate a bunch of stuff is done, but it succeeds until VOP_SETATTR) vn_truncate() (in sys/kern/vfs_vnops.c:636) kern_openat() (in sys/kern/vfs_syscalls.c:1043) kern_open() (in sys/kern/vfs_syscalls.c:1035) open() (in sys/kern/vfs_syscalls.c:1006) ufs_setattr() returns with -1 (EPERM) I'll continue to try to find the exact problem. A quick workaround currently would probably be to use a different filesystem other than ufs, but then again, I have no clue if other filesystems have the same issue. Hopefully that trace will help anyone who wants to help out. -Brandon Falk ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
