Re: [Samba] Sudden authentication failures, hex dumps in log.samba
On 14.5.2013 19:49, Pekka L.J. Jalkanen wrote: On 14.5.2013 19:31, Andrew Bartlett wrote: On Tue, 2013-05-14 at 11:04 +0300, Pekka L.J. Jalkanen wrote: On 14.5.2013 8:04, Andrew Bartlett wrote: The issue is the same for all of these accounts. We simply have a password encoded in a format that we do not correctly parse. The 00 20 stuff is literally some unicode space (ie the spacebar, yes!) padding that is in this structure. Huh?! Now I'm surprised, both about that there is such a parsing problem and that the problem is _that_ trivial. Shouldn't this mean that I can most likely work the problem away by simply changing the passwords of these users? Now that would be great news indeed! Yes, if I'm understanding it correctly. OK, I'll ask some of them to change their password, and then see what will happen. Thank you! Confirmed (so far only on one user): after password change logon works normally, and there are no errors in log.samba. (The account migration between domains was done earlier this year, but the accounts were temporarily marked as having never expiring passwords so that no password changes would be imposed in the process. Perhaps the whole issue wouldn't exist if this hadn't been done...) So it is now quite clear that there is something fishy in passwords migrated by MS's Password Export Server (the one shipped with ADMT v3.0). Unless this is fixed it will be important that at least one password change should be forced for any users previously migrated from other domains, or any present or future Samba DCs in the target domain won't understand them. (It's good to remember that even pure Samba DC environments could end up using ADMT if in need, because running it only requires presence of a Windows DC for a short period of time, so eval version should do. Also, one-way trust should suffice, with the target domain as the trusted one, so having Samba DC's on target shouldn't cause any problems. ADMT is also often simply quite necessary tool when organisation's structure changes for some reason.) Pekka L.J. Jalkanen -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] Sudden authentication failures, hex dumps in log.samba
On 14.5.2013 8:04, Andrew Bartlett wrote: On Mon, 2013-05-13 at 14:24 +0300, Pekka L.J. Jalkanen wrote: Any ideas how to resolve this problem? No comments, it seems. I can see that even if this is a bug in Samba it would be really hard to reproduce. But it's really frustrating too, because if the authentication isn't reliable I sort of have to keep the Windows DC around. So if somebody would have an enlightened suggestion what to do, I'd be grateful. The only idea I'm having myself would be to recreate the machine accounts of the computers in question, but that'd be just a shot in the dark, and if the problem lies within the user accounts instead, that wouldn't help. G'Day, I'm sorry I haven't been able to get back to you. Please don't. I've had all too many questions for you already. Thank you for your patience with me! The issue is the same for all of these accounts. We simply have a password encoded in a format that we do not correctly parse. The 00 20 stuff is literally some unicode space (ie the spacebar, yes!) padding that is in this structure. Huh?! Now I'm surprised, both about that there is such a parsing problem and that the problem is _that_ trivial. Shouldn't this mean that I can most likely work the problem away by simply changing the passwords of these users? Now that would be great news indeed! I need to get both and encrypted copy of the data and some time to work over it, so we can correct this issue in our IDL. You already have a complete copy of our Samba DC's DB due to that exportkeytab issue. I can send you nonsanitised logs separately so that you can see the relevant account names. Is that enough, or do you need me to try to make an actual packet capture of this problem? Pekka L.J. Jalkanen -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] Sudden authentication failures, hex dumps in log.samba
On Tue, 2013-05-14 at 11:04 +0300, Pekka L.J. Jalkanen wrote: On 14.5.2013 8:04, Andrew Bartlett wrote: On Mon, 2013-05-13 at 14:24 +0300, Pekka L.J. Jalkanen wrote: Any ideas how to resolve this problem? No comments, it seems. I can see that even if this is a bug in Samba it would be really hard to reproduce. But it's really frustrating too, because if the authentication isn't reliable I sort of have to keep the Windows DC around. So if somebody would have an enlightened suggestion what to do, I'd be grateful. The only idea I'm having myself would be to recreate the machine accounts of the computers in question, but that'd be just a shot in the dark, and if the problem lies within the user accounts instead, that wouldn't help. G'Day, I'm sorry I haven't been able to get back to you. Please don't. I've had all too many questions for you already. Thank you for your patience with me! The issue is the same for all of these accounts. We simply have a password encoded in a format that we do not correctly parse. The 00 20 stuff is literally some unicode space (ie the spacebar, yes!) padding that is in this structure. Huh?! Now I'm surprised, both about that there is such a parsing problem and that the problem is _that_ trivial. Shouldn't this mean that I can most likely work the problem away by simply changing the passwords of these users? Now that would be great news indeed! Yes, if I'm understanding it correctly. I need to get both and encrypted copy of the data and some time to work over it, so we can correct this issue in our IDL. You already have a complete copy of our Samba DC's DB due to that exportkeytab issue. I can send you nonsanitised logs separately so that you can see the relevant account names. Is that enough, or do you need me to try to make an actual packet capture of this problem? The exportkeytab issue is the same issue. You are just seeing the same failure to read the password for a particular account in multiple ways. Andrew Bartlett -- Andrew Bartletthttp://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] Sudden authentication failures, hex dumps in log.samba
On 14.5.2013 19:31, Andrew Bartlett wrote: On Tue, 2013-05-14 at 11:04 +0300, Pekka L.J. Jalkanen wrote: On 14.5.2013 8:04, Andrew Bartlett wrote: The issue is the same for all of these accounts. We simply have a password encoded in a format that we do not correctly parse. The 00 20 stuff is literally some unicode space (ie the spacebar, yes!) padding that is in this structure. Huh?! Now I'm surprised, both about that there is such a parsing problem and that the problem is _that_ trivial. Shouldn't this mean that I can most likely work the problem away by simply changing the passwords of these users? Now that would be great news indeed! Yes, if I'm understanding it correctly. OK, I'll ask some of them to change their password, and then see what will happen. Thank you! (The account migration between domains was done earlier this year, but the accounts were temporarily marked as having never expiring passwords so that no password changes would be imposed in the process. Perhaps the whole issue wouldn't exist if this hadn't been done...) I need to get both and encrypted copy of the data and some time to work over it, so we can correct this issue in our IDL. You already have a complete copy of our Samba DC's DB due to that exportkeytab issue. I can send you nonsanitised logs separately so that you can see the relevant account names. Is that enough, or do you need me to try to make an actual packet capture of this problem? The exportkeytab issue is the same issue. You are just seeing the same failure to read the password for a particular account in multiple ways. Oh. Well, now I'm understanding it all much better (or I at least hope that I do now). Thank you for explaining this! And thanks for replying--that reminded me that I forgot to send that log file to you. Will do so promptly. Pekka L.J. Jalkanen -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] Sudden authentication failures, hex dumps in log.samba
On 10.5.2013 16:32, Pekka L.J. Jalkanen wrote: On 10.5.2013 14:04, Pekka L.J. Jalkanen wrote: Question: how much more verbosity for log.samba would be needed to further investigate this problem? I'd rather not log everything with -d10 for extended periods of time, because I really can't know how long it will take for the problem to reappear. I've now increased logging from the default level to -d3. -d3 logging pays off: [2013/05/10 14:31:06, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: Client no longer in database: someu...@mydomain.site [2013/05/10 14:31:06, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: Failed building TGS-REP to ipv4:10.10.59.151:4736 [2013/05/10 14:31:06, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: TGS-REQ someu...@mydomain.site from ipv4:10.10.59.151:4737 for cifs/w2k3r2dc.mydomain.s...@mydomain.site [renewable, forwardable] [2013/05/10 14:31:06, 1] ../librpc/ndr/ndr.c:412(ndr_pull_error) ndr_pull_error(11): Pull bytes 2 (../librpc/ndr/ndr_basic.c:103) Client is Windows XP. I've yet to see this problem on newer clients... this and the other one that previously failed are the last two XP clients here that still remain in heavy production use. Somewhat similar error occurred with a Windows 7 machine. But note that for some reason only the short domain dame was used in reference to the realm: [2013/05/13 08:04:53, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: AS-REQ otheruser@MYDOMAIN from ipv4:10.10.59.148:58027 for krbtgt/MYDOMAIN@MYDOMAIN [2013/05/13 08:04:53, 1] ../librpc/ndr/ndr.c:412(ndr_pull_error) ndr_pull_error(11): Pull bytes 2 (../librpc/ndr/ndr_basic.c:103) [2013/05/13 08:04:53, 0] ../lib/util/util.c:457(dump_data) [] 00 00 00 00 62 00 00 00 00 00 00 00 20 00 20 00 b... . . [0010] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0020] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0030] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0040] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0050] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0060] 20 00 20 00 20 00 20 00 20 00 20 00 50 00 00 . . . . . .P.. [2013/05/13 08:04:53, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: UNKNOWN -- otheruser@MYDOMAIN: no such entry found in hdb What is also common with this client and the other that previously failed is that they both have once been migrated from a different domain (that no longer exists) using MS ADMT. This also applies to the users' accounts that were used. Don't know if that really matters, but just for the record. Also the Windows 7 client was once migrated this way. Both the second case and the third case were also different from the first one in the way that the users had no problems logging on. However, even though they said to me that they had had no authentication problems, I still think that they haven't just noticed, as I found the following from the event log of the second client: - Event Type: Warning Event Source: LSASRV Event Category: SPNEGO (Negotiator) Event ID: 40960 Date: 10.5.2013 Time: 13:52:42 User: N/A Computer: XPWKSTN2 Description: The Security System detected an attempted downgrade attack for server LDAP/samba4dc.mydomain.site. The failure code from authentication protocol Kerberos was Insufficient system resources exist to complete the API. (0xc09a). Event Type: Warning Event Source: LSASRV Event Category: SPNEGO (Negotiator) Event ID: 40961 Date: 10.5.2013 Time: 13:52:42 User: N/A Computer: XPWKSTN2 Description: The Security System could not establish a secured connection with the server LDAP/samba4dc.mydomain.site. No authentication protocol was available. Event Type: Warning Event Source: LSASRV Event Category: SPNEGO (Negotiator) Event ID: 40960 Date: 10.5.2013 Time: 14:31:05 User: N/A Computer: XPWKSTN2 Description: The Security System detected an attempted downgrade attack for server cifs/w2k3r2dc.mydomain.site. The failure code from authentication protocol Kerberos was Insufficient system resources exist to complete the API. (0xc09a). Event Type: Warning Event Source: LSASRV Event Category: SPNEGO (Negotiator) Event ID: 40961 Date: 10.5.2013 Time: 14:31:06 User: N/A Computer: XPWKSTN2 Description: The Security System could not establish a secured connection with the server cifs/w2k3r2dc.mydomain.site. No authentication protocol was available. - All this is really odd, though, as these machines have been part of the domain
Re: [Samba] Sudden authentication failures, hex dumps in log.samba
On Mon, 2013-05-13 at 14:24 +0300, Pekka L.J. Jalkanen wrote: Any ideas how to resolve this problem? No comments, it seems. I can see that even if this is a bug in Samba it would be really hard to reproduce. But it's really frustrating too, because if the authentication isn't reliable I sort of have to keep the Windows DC around. So if somebody would have an enlightened suggestion what to do, I'd be grateful. The only idea I'm having myself would be to recreate the machine accounts of the computers in question, but that'd be just a shot in the dark, and if the problem lies within the user accounts instead, that wouldn't help. G'Day, I'm sorry I haven't been able to get back to you. The issue is the same for all of these accounts. We simply have a password encoded in a format that we do not correctly parse. The 00 20 stuff is literally some unicode space (ie the spacebar, yes!) padding that is in this structure. I need to get both and encrypted copy of the data and some time to work over it, so we can correct this issue in our IDL. Andrew Bartlett -- Andrew Bartletthttp://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] Sudden authentication failures, hex dumps in log.samba
On 10.5.2013 14:04, Pekka L.J. Jalkanen wrote: Question: how much more verbosity for log.samba would be needed to further investigate this problem? I'd rather not log everything with -d10 for extended periods of time, because I really can't know how long it will take for the problem to reappear. I've now increased logging from the default level to -d3. -d3 logging pays off: [2013/05/10 14:31:05, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: TGS-REQ someu...@mydomain.site from ipv4:10.10.59.151:4736 for cifs/w2k3r2dc.mydomain.s...@mydomain.site [renewable, forwardable] [2013/05/10 14:31:06, 1] ../librpc/ndr/ndr.c:412(ndr_pull_error) ndr_pull_error(11): Pull bytes 2 (../librpc/ndr/ndr_basic.c:103) [2013/05/10 14:31:06, 0] ../lib/util/util.c:457(dump_data) [] 00 00 00 00 62 00 00 00 00 00 00 00 20 00 20 00 b... . . [0010] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0020] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0030] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0040] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0050] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0060] 20 00 20 00 20 00 20 00 20 00 20 00 50 00 00 . . . . . .P.. [2013/05/10 14:31:06, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: Client no longer in database: someu...@mydomain.site [2013/05/10 14:31:06, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: Failed building TGS-REP to ipv4:10.10.59.151:4736 [2013/05/10 14:31:06, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: TGS-REQ someu...@mydomain.site from ipv4:10.10.59.151:4737 for cifs/w2k3r2dc.mydomain.s...@mydomain.site [renewable, forwardable] [2013/05/10 14:31:06, 1] ../librpc/ndr/ndr.c:412(ndr_pull_error) ndr_pull_error(11): Pull bytes 2 (../librpc/ndr/ndr_basic.c:103) [2013/05/10 14:31:06, 0] ../lib/util/util.c:457(dump_data) [] 00 00 00 00 62 00 00 00 00 00 00 00 20 00 20 00 b... . . [0010] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0020] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0030] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0040] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0050] 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00. . . . . . . . [0060] 20 00 20 00 20 00 20 00 20 00 20 00 50 00 00 . . . . . .P.. [2013/05/10 14:31:06, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: Client no longer in database: someu...@mydomain.site [2013/05/10 14:31:06, 3] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper) Kerberos: Failed building TGS-REP to ipv4:10.10.59.151:4737 [2013/05/10 14:31:20, 3] ../source4/dsdb/repl/drepl_service.c:202(_drepl_schedule_replication) Client is Windows XP. I've yet to see this problem on newer clients... this and the other one that previously failed are the last two XP clients here that still remain in heavy production use. What is also common with this client and the other that previously failed is that they both have once been migrated from a different domain (that no longer exists) using MS ADMT. This also applies to the users' accounts that were used. Don't know if that really matters, but just for the record. Any ideas how to resolve this problem? Pekka L.J. Jalkanen -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
