Re: Another showstopper in 2.2.5
On Wed, 21 Aug 2002 [EMAIL PROTECTED] wrote: On Wed, Aug 21, 2002 at 07:40:04PM +0200, Fredrik Ohrn wrote: OK, I've tried it out and it didn't help. I removed my BlockSignals stuff and added ZERO_STRUCT(act) instead, now the smbd dies on SIGPIPE just as before. Ok, we need to know when SIGPIPE is being removed from the blocked signal set on your system. Can you add some code in the main loop that prints out the current signal mask using this code: sigset_t blocked_mask = siggetmask(); DEBUG(0,SIGPIPE is %s\n, sigismember(blocked_mask, SIGPIPE) ? blocked : nonblocked )); So we can see when the SIGPIPE is being removed from your masked set. OK, I put this at the top of the while loop in smbd_process: sigset_t dummy_mask, blocked_mask; sigemptyset (dummy_mask); sigemptyset (blocked_mask); sigprocmask (SIG_BLOCK, dummy_mask, blocked_mask); DEBUG (0, (SIGPIPE is %s\n, sigismember (blocked_mask, SIGPIPE) ? blocked : nonblocked )); The signal gets unblocked right after I connect to the share. Below is the log (at level 4), it got a bit large so I added some ** marks at the SIGPIPE is blocked/nonblocked messages. I'll go through the code tonight and add more debug messages to try to pinpoint the exact place the signal is unmasked. /Fredrik ... ... [2002/08/22 08:29:49, 3, pid=8064] smbd/server.c:main(753) loaded services [2002/08/22 08:29:49, 3, pid=8064] smbd/server.c:main(768) Becoming a daemon. [2002/08/22 08:29:49, 3, pid=8065] lib/util_sock.c:open_socket_in(815) bind succeeded on port 139 [2002/08/22 08:29:49, 2, pid=8065] smbd/server.c:open_sockets(215) waiting for a connection [2002/08/22 08:30:02, 3, pid=8078] smbd/oplock.c:init_oplocks(1211) open_oplock_ipc: opening loopback UDP socket. [2002/08/22 08:30:02, 3, pid=8078] lib/util_sock.c:open_socket_in(815) bind succeeded on port 0 [2002/08/22 08:30:02, 3, pid=8078] smbd/oplock_linux.c:linux_init_kernel_oplocks(299) Linux kernel oplocks enabled [2002/08/22 08:30:02, 3, pid=8078] smbd/oplock.c:init_oplocks(1241) open_oplock ipc: pid = 8078, global_oplock_port = 4957 [2002/08/22 08:30:02, 4, pid=8078] lib/time.c:get_serverzone(114) Serverzone is -7200 **[2002/08/22 08:30:02, 0, pid=8078] smbd/process.c:smbd_process(1264) ** SIGPIPE is blocked [2002/08/22 08:30:02, 3, pid=8078] smbd/process.c:process_smb(877) Transaction 0 of length 72 [2002/08/22 08:30:02, 2, pid=8078] smbd/reply.c:reply_special(91) netbios connect: name1=FELINE name2=CANINE [2002/08/22 08:30:02, 2, pid=8078] smbd/reply.c:reply_special(110) netbios connect: local=feline remote=canine **[2002/08/22 08:30:02, 0, pid=8078] smbd/process.c:smbd_process(1264) ** SIGPIPE is blocked [2002/08/22 08:30:02, 3, pid=8078] smbd/process.c:process_smb(877) Transaction 1 of length 137 [2002/08/22 08:30:02, 3, pid=8078] smbd/process.c:switch_message(684) switch message SMBnegprot (pid 8078) [2002/08/22 08:30:02, 3, pid=8078] smbd/sec_ctx.c:set_sec_ctx(313) setting sec ctx (0, 0) - sec_ctx_stack_ndx = 0 [2002/08/22 08:30:02, 3, pid=8078] smbd/negprot.c:reply_negprot(342) Requested protocol [PC NETWORK PROGRAM 1.0] [2002/08/22 08:30:02, 3, pid=8078] smbd/negprot.c:reply_negprot(342) Requested protocol [LANMAN1.0] [2002/08/22 08:30:02, 3, pid=8078] smbd/negprot.c:reply_negprot(342) Requested protocol [Windows for Workgroups 3.1a] [2002/08/22 08:30:02, 3, pid=8078] smbd/negprot.c:reply_negprot(342) Requested protocol [LM1.2X002] [2002/08/22 08:30:02, 3, pid=8078] smbd/negprot.c:reply_negprot(342) Requested protocol [LANMAN2.1] [2002/08/22 08:30:02, 3, pid=8078] smbd/negprot.c:reply_negprot(342) Requested protocol [NT LM 0.12] [2002/08/22 08:30:02, 3, pid=8078] libsmb/namequery.c:resolve_lmhosts(768) resolve_lmhosts: Attempting lmhosts lookup for name PDC0x20 [2002/08/22 08:30:02, 4, pid=8078] libsmb/namequery.c:startlmhosts(474) startlmhosts: Can't open lmhosts file /usr/local/samba/lib/lmhosts. Error was No such file or directory [2002/08/22 08:30:02, 3, pid=8078] libsmb/namequery.c:resolve_hosts(808) resolve_hosts: Attempting host lookup for name PDC0x20 [2002/08/22 08:30:02, 3, pid=8078] lib/util_sock.c:open_socket_out(847) Connecting to 129.16.214.4 at port 445 [2002/08/22 08:30:03, 2, pid=8078] lib/util_sock.c:open_socket_out(875) error connecting to 129.16.214.4:445 (Connection refused) [2002/08/22 08:30:03, 3, pid=8078] lib/util_sock.c:open_socket_out(847) Connecting to 129.16.214.4 at port 139 [2002/08/22 08:30:03, 3, pid=8078] smbd/password.c:server_cryptkey(1045) connected to password server PDC [2002/08/22 08:30:03, 3, pid=8078] smbd/password.c:server_cryptkey(1062) got session [2002/08/22 08:30:03, 3, pid=8078] smbd/password.c:server_cryptkey(1077) password server OK [2002/08/22 08:30:03, 3, pid=8078] smbd/negprot.c:reply_nt1(179) using password server validation [2002/08/22 08:30:03, 3, pid=8078] smbd/negprot.c:reply_negprot(426) Selected protocol NT LM 0.12
Re: Another showstopper in 2.2.5
On Thu, Aug 22, 2002 at 09:04:39AM +0200, Fredrik Ohrn wrote: OK, I put this at the top of the while loop in smbd_process: sigset_t dummy_mask, blocked_mask; sigemptyset (dummy_mask); sigemptyset (blocked_mask); sigprocmask (SIG_BLOCK, dummy_mask, blocked_mask); DEBUG (0, (SIGPIPE is %s\n, sigismember (blocked_mask, SIGPIPE) ? blocked : nonblocked )); The signal gets unblocked right after I connect to the share. Below is the log (at level 4), it got a bit large so I added some ** marks at the SIGPIPE is blocked/nonblocked messages. I'll go through the code tonight and add more debug messages to try to pinpoint the exact place the signal is unmasked. Great - thanks. Maybe a debug level 10 may help pinpoint this ? I'm not seeing this on my system, neither is tridge or others here at the CIFS conference. Jeremy.
Re: Another showstopper in 2.2.5
On Tue, 20 Aug 2002 [EMAIL PROTECTED] wrote: smbd should never get a SIGPIPE. Look at this fragment of code from smbd/server.c : fault_setup((void (*)(void *))exit_server); CatchSignal(SIGTERM , SIGNAL_CAST sig_term); CatchSignal(SIGHUP,SIGNAL_CAST sig_hup); /* we are never interested in SIGPIPE */ BlockSignals(True,SIGPIPE); We ignore SIGPIPE at server startup, it was added about 10 years ago :-). Can you give more details about getting SIGPIPE in smbd ? What platform, etc. I see this on 2 boxes, one running Mandrake 8.2 and the other RedHat 7.3 It happens on both the included 'distro' kernels and on a plain 2.4.19 kernel. The glibc is glibc-2.2.4-25mdk.rpm on mandrake and glibc-2.2.5-36.rpm on RedHat. /Fredrik -- It is easy to be blinded to the essential uselessness of computers by the sense of accomplishment you get from getting them to work at all. - Douglas Adams Fredrik Öhrn Chalmers University of Technology [EMAIL PROTECTED] Sweden
Re: Another showstopper in 2.2.5
On Wed, 21 Aug 2002, Richard Bollinger wrote: I looked around to see where all of the signal handling code is / should be... lib/signal.c has most of the good stuff... BUT there are calls to sigaction also in smbd/notify_kernel.c and smbd/oplock_linux.c which look suspicious. I can't reproduce the problem myself, but it looks as though there should be a call to ZERO_STRUCT(act) inserted in both kernel_notify_init() and in linux_init_kernel_oplocks(), because the structure element sa_mask is left undefined in both cases... thus making it possible for _any_ signal to occur inside signal handlers being installed in those places. What do you guys think? Rich Bollinger, Elliott Company OK, I've tried it out and it didn't help. I removed my BlockSignals stuff and added ZERO_STRUCT(act) instead, now the smbd dies on SIGPIPE just as before. /Fredrik -- It is easy to be blinded to the essential uselessness of computers by the sense of accomplishment you get from getting them to work at all. - Douglas Adams Fredrik Öhrn Chalmers University of Technology [EMAIL PROTECTED] Sweden
Re: Another showstopper in 2.2.5
On Wed, Aug 21, 2002 at 07:40:04PM +0200, Fredrik Ohrn wrote: OK, I've tried it out and it didn't help. I removed my BlockSignals stuff and added ZERO_STRUCT(act) instead, now the smbd dies on SIGPIPE just as before. Ok, we need to know when SIGPIPE is being removed from the blocked signal set on your system. Can you add some code in the main loop that prints out the current signal mask using this code: sigset_t blocked_mask = siggetmask(); DEBUG(0,SIGPIPE is %s\n, sigismember(blocked_mask, SIGPIPE) ? blocked : nonblocked )); So we can see when the SIGPIPE is being removed from your masked set. Thanks, Jeremy.
Re: Another showstopper in 2.2.5
Hmmm... reading the man page for sigaction()... the semantics are a bit unclear regarding sa_mask: sa_mask gives a mask of signals which should be blocked during execution of the signal handler. In addition, the signal which triggered the handler will be blocked, unless the SA_NODEFER or SA_NOMASK flags are used. So... for example, if the mask indicated only that we wanted to block SIGQUIT, for example, could the signal handler being registered be hit by SIGPIPE while it is running? That being the case, would we need to specify a mask which includes at least everything we've decided to IGNORE? - Original Message - From: [EMAIL PROTECTED] To: Fredrik Ohrn [EMAIL PROTECTED] Cc: Richard Bollinger [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Wednesday, August 21, 2002 9:58 AM Subject: Re: Another showstopper in 2.2.5 On Wed, Aug 21, 2002 at 07:40:04PM +0200, Fredrik Ohrn wrote: OK, I've tried it out and it didn't help. I removed my BlockSignals stuff and added ZERO_STRUCT(act) instead, now the smbd dies on SIGPIPE just as before. Ok, we need to know when SIGPIPE is being removed from the blocked signal set on your system. Can you add some code in the main loop that prints out the current signal mask using this code: sigset_t blocked_mask = siggetmask(); DEBUG(0,SIGPIPE is %s\n, sigismember(blocked_mask, SIGPIPE) ? blocked : nonblocked )); So we can see when the SIGPIPE is being removed from your masked set. Thanks, Jeremy.
Re: Another showstopper in 2.2.5
On Wed, Aug 21, 2002 at 04:17:01PM -0400, Richard Bollinger wrote: Hmmm... reading the man page for sigaction()... the semantics are a bit unclear regarding sa_mask: sa_mask gives a mask of signals which should be blocked during execution of the signal handler. In addition, the signal which triggered the handler will be blocked, unless the SA_NODEFER or SA_NOMASK flags are used. So... for example, if the mask indicated only that we wanted to block SIGQUIT, for example, could the signal handler being registered be hit by SIGPIPE while it is running? That being the case, would we need to specify a mask which includes at least everything we've decided to IGNORE? Ok (I'm at the CIFS plugfest) - I just checked the POSIX spec and the mask in the sigaction struct should be *added* to the signal mask when the signal is taken. Tridge just checked on his Linux 2.4.x box and SIGPIPE is definately masked (checked in proc on the running smbd process). It could be a bug in Linux and not Solaris though - can you try adding the line sigfillset(act.sa_mask); just before the sigaction call in the CatchSignal() call. Jeremy.
Re: Another showstopper in 2.2.5
On Tue, 20 Aug 2002, David Collier-Brown wrote: Fredrik Ohrn wrote: Sooner or later the smbd dies on a SIGPIPE when trying to send a keepalive. The SIGPIPE isn't catched, thus it leaves stale sharemodes. Thank you, Oh Kind Person! (I was wondering how this could possibly happen, you see) Here's more good news. After getting a hint on signalhandling I put a BlockSignals call into send_keepalive. --- lib/util_sock.c.org Tue Aug 20 12:31:58 2002 +++ lib/util_sock.c Tue Aug 20 12:33:15 2002 @@ -401,6 +401,7 @@ buf[0] = 0x85; buf[1] = buf[2] = buf[3] = 0; + BlockSignals(True,SIGPIPE); return(write_socket_data(client,(char *)buf,4) == 4); } This protects the smbd from flaming death, instead I get these (harmless) messages in the log: [2002/08/20 18:21:37, 0, pid=3392] lib/util_sock.c:write_socket_data(501) write_socket_data: write failure. Error = Broken pipe [2002/08/20 18:21:37, 0, pid=3392] smbd/process.c:timeout_processing(1147) password server keepalive failed. Apparently it's the password-server, not the client that drops the ball. We are using samba 2.2.5 with a ldap backend as our domaincontroller and the fileservers use 'security = server'. Anyway, I don't know if send_keepalive is the correct place to block signals or if it should be done higher up in the chain, I'll leave that for you samba gurus to decide. I have pounded at my quick-fixed server for an hour now and my problems with broken Word documents etc. seems to be gone, nor have I seen any bad sideeffects (yet). /Fredrik, happy as a clam. -- It is easy to be blinded to the essential uselessness of computers by the sense of accomplishment you get from getting them to work at all. - Douglas Adams Fredrik Öhrn Chalmers University of Technology [EMAIL PROTECTED] Sweden
Re: Another showstopper in 2.2.5
On Tue, 20 Aug 2002, Tom Jansen wrote: Sooner or later the smbd dies on a SIGPIPE when trying to send a keepalive. The SIGPIPE isn't catched, thus it leaves stale sharemodes. After getting a hint on signalhandling I put a BlockSignals call into send_keepalive. --- lib/util_sock.c.org Tue Aug 20 12:31:58 2002 +++ lib/util_sock.c Tue Aug 20 12:33:15 2002 @@ -401,6 +401,7 @@ buf[0] = 0x85; buf[1] = buf[2] = buf[3] = 0; + BlockSignals(True,SIGPIPE); return(write_socket_data(client,(char *)buf,4) == 4); } What about unblocking ? Well, since there's no signal handler for SIGPIPE you might just end up with death by SIGPIPE somewhere else in the code instead. I guess the proper thing to do is to globally block SIGPIPE at startup and assume that all calls to read()/write() properly handles the case of failure with EPIPE. /Fredrik -- It is easy to be blinded to the essential uselessness of computers by the sense of accomplishment you get from getting them to work at all. - Douglas Adams Fredrik Öhrn Chalmers University of Technology [EMAIL PROTECTED] Sweden
Re: Another showstopper in 2.2.5
On Tue, Aug 20, 2002 at 10:06:30PM +0200, Fredrik Ohrn wrote: On Tue, 20 Aug 2002, Tom Jansen wrote: Sooner or later the smbd dies on a SIGPIPE when trying to send a keepalive. The SIGPIPE isn't catched, thus it leaves stale sharemodes. After getting a hint on signalhandling I put a BlockSignals call into send_keepalive. --- lib/util_sock.c.org Tue Aug 20 12:31:58 2002 +++ lib/util_sock.c Tue Aug 20 12:33:15 2002 -401,6 +401,7 buf[0] = 0x85; buf[1] = buf[2] = buf[3] = 0; + BlockSignals(True,SIGPIPE); return(write_socket_data(client,(char *)buf,4) == 4); } What about unblocking ? Well, since there's no signal handler for SIGPIPE you might just end up with death by SIGPIPE somewhere else in the code instead. I guess the proper thing to do is to globally block SIGPIPE at startup and assume that all calls to read()/write() properly handles the case of failure with EPIPE. smbd should never get a SIGPIPE. Look at this fragment of code from smbd/server.c : fault_setup((void (*)(void *))exit_server); CatchSignal(SIGTERM , SIGNAL_CAST sig_term); CatchSignal(SIGHUP,SIGNAL_CAST sig_hup); /* we are never interested in SIGPIPE */ BlockSignals(True,SIGPIPE); We ignore SIGPIPE at server startup, it was added about 10 years ago :-). Can you give more details about getting SIGPIPE in smbd ? What platform, etc. Jremy.
Re: Another showstopper in 2.2.5
On Tue, Aug 20, 2002 at 12:24:51PM -0400, David Collier-Brown wrote: Fredrik Ohrn wrote: Sooner or later the smbd dies on a SIGPIPE when trying to send a keepalive. The SIGPIPE isn't catched, thus it leaves stale sharemodes. Thank you, Oh Kind Person! (I was wondering how this could possibly happen, you see) Program received signal SIGPIPE, Broken pipe. 0x4015e332 in send () from /lib/libc.so.6 (gdb) bt #0 0x4015e332 in send () from /lib/libc.so.6 #1 0x0812f81d in sys_send () #2 0x0813e90b in write_socket_data () #3 0x0813e70b in send_keepalive () #4 0x080a2348 in timeout_processing () #5 0x080a264e in smbd_process () #6 0x08069959 in main () #7 0x4008a280 in __libc_start_main () from /lib/libc.so.6 (gdb) cont Continuing. Program terminated with signal SIGPIPE, Broken pipe. The program no longer exists. (gdb) Things to note is that the client is alive and have had no (obvious) reason to drop the connection or otherwise stop responding. I have tried this over and over and it's allways in send_keepalive that the SIGPIPE of death happens. The problem is this shouldn't happen - we block all SIGPIPE signals in smbd startup. If we're getting them there is a problem somewhere. Remember, running under gdb may reset the process signal mask - when not running under gdb we will never see SIGPIPE. Jeremy.
Another showstopper in 2.2.5
Ok, here's another showstopper I have started to get alot over the last week. Average user complains that that he can't open Word/Excel/PowerPoint document foo because it is already opened by another (unknown) user. But he had no problems working with the document some time ago. This happens a lot with Win98 clients and sometimes with WinXP but it's not as common. After some digging I have figured out that the samba process serving the document the first time is dead BUT according to smbstatus samba still thinks the process is alive and the file locked... A small example: [root@olivia ~]# /usr/local/samba/bin/smbstatus | grep lottab lottab lottab stdvgl6887 studv-5-118 (129.16.xxx.xxx) Mon Aug 12 08:31:16 2002 6490 DENY_WRITE 0x1 RDONLY NONE /olivia/home3/stdvgl/lottab/netscape/rpclsvc_se.dll Mon Aug 12 08:16:33 2002 6490 DENY_NONE 0x3 RDWR NONE /olivia/home3/stdvgl/lottab/eudora/EudPriv/Ads/Eudora.idx Mon Aug 12 08:16:38 2002 ... lots more nixed ... Lets look for pid 6490 [root@olivia ~]# ps -ef | grep 6490 root 7240 7150 0 08:40 pts/400:00:00 grep 6490 Not found! Lets look for the username instead: [root@olivia ~]# ps -ef | grep lottab lottab6887 18504 0 08:31 ?00:00:00 /usr/local/samba/bin/smbd -D Look mom! A new smbd process is now serving this user. There is no trace what so ever in the logfile on why the smbd died. Tonight I'll see if I can figure out a way to force the problem and use the panic action to get more clues. Does anyone have a good recepie for 'panic action' that just dumps a backtrace into a logfile? Regards, Fredrik -- It is easy to be blinded to the essential uselessness of computers by the sense of accomplishment you get from getting them to work at all. - Douglas Adams Fredrik Öhrn Chalmers University of Technology [EMAIL PROTECTED] Sweden
Re: Another showstopper in 2.2.5
Fredrik Ohrn wrote: Look mom! A new smbd process is now serving this user. There is no trace what so ever in the logfile on why the smbd died. Hmmn: could you set keepalive = 30, and see if samba cleans up and releases the locks held? I'm almost about to reccomend keepalive = 3600 as a default... It's an option to clean up after clients which exploded, blue-screened or just started to flake. Alas, it tends to hide PC problems by working around them. --dave -- David Collier-Brown, | Always do right. This will gratify Performance Engineering | some people and astonish the rest. Americas Customer Engineering, | -- Mark Twain (905) 415-2849 | [EMAIL PROTECTED]
Re: Another showstopper in 2.2.5
On Mon, 12 Aug 2002, David Collier-Brown wrote: Fredrik Ohrn wrote: Look mom! A new smbd process is now serving this user. There is no trace what so ever in the logfile on why the smbd died. Hmmn: could you set keepalive = 30, and see if samba cleans up and releases the locks held? I'm almost about to reccomend keepalive = 3600 as a default... It's an option to clean up after clients which exploded, blue-screened or just started to flake. Alas, it tends to hide PC problems by working around them. OK, I'll try 30 instead of the default 300. But I don't expect it to help, in this case it seems that it's the smbd process that blows up, not the client. /Fredrik -- It is easy to be blinded to the essential uselessness of computers by the sense of accomplishment you get from getting them to work at all. - Douglas Adams Fredrik Öhrn Chalmers University of Technology [EMAIL PROTECTED] Sweden
Re: Another showstopper in 2.2.5
Fredrik Ohrn wrote: OK, I'll try 30 instead of the default 300. But I don't expect it to help, in this case it seems that it's the smbd process that blows up, not the client. Ok, I was afraid of that! Blown SMBDs are Bad Things (:-)) --dave -- David Collier-Brown, | Always do right. This will gratify Performance Engineering | some people and astonish the rest. Americas Customer Engineering, | -- Mark Twain (905) 415-2849 | [EMAIL PROTECTED]
Re: Another showstopper in 2.2.5
On Mon, 2002-08-12 at 10:54, David Collier-Brown wrote: Fredrik Ohrn wrote: OK, I'll try 30 instead of the default 300. But I don't expect it to help, in this case it seems that it's the smbd process that blows up, not the client. Ok, I was afraid of that! Blown SMBDs are Bad Things (:-)) I have run into this same problem from time to time with 2.2.2. I was hoping that the upgrade to 2.2.5 that I did yesterday would fix it... On the surface, it appears as though a panic action program that does lock scrubbing would be useful. I envision a standalone program that takes a filename and pid as an argument. If it encouters a lock belonging to the given pid, it removes the lock from the database. Perhaps if the program is given only one argument (the lock file) it traverses the tdb checking to see if each pid mentioned is alive. What would this break? Mike
