Re: [c-nsp] FWSM logging problem
To answer all questions about versions e.d. We are running 3.1(4), not the latest I know, but people here are 'allergic' to network downtime and with semester exams coming up, I won't be able to upgrade before February. I removed the log option from the rule which should have given me 106023 messages in my logs but they don't show up ; the ACE is being hit however : access-list Internet-out line 24 extended deny ip any host x.x.x.x (hitcnt=13) 0x6e051e8c As far as I can tell, there is no queue problem : Logging Queue length limit : 1024 msg(s), 30947037 msg(s) discarded. Current 502 msg on queue, 512 msgs most on queue I raised the limit to 1024 yesterday and the number of discards stayed the same since then. There doesn't seem to be a caching problem either : fwcdep/fwcdep1# sh access-list | incl cache access-list cached ACL log flows: total 5, denied 3 (deny-flow-max 4096) I'll have to live with this until I can upgrade. Wim -Original Message- From: Andrew Yourtchenko [mailto:ayour...@cisco.com] Sent: woensdag 16 december 2009 19:35 To: Holemans Wim Cc: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] FWSM logging problem On Wed, 16 Dec 2009, Holemans Wim wrote: It seems our FWSM doesn't log all denied ACLs. I blocked an IP address on our FWSM and wanted to see whomever on campus is trying to access this address (Botnet CC). I added the following line in the ACL (even raised priority), you can see that the rules triggers when I tried to telnet the address : access-list Internet-out line 24 extended deny ip any host X1.X2.X3.X4 log critical interval 30 (hitcnt=9) 0x6e051e8c There is however no corresponding syslog message on our syslog server or in the buffered logs on the FWSM. Any chances you'd have %FWSM-1-106101: Number of cached deny-flows for ACL log has reached limit somewhere ? Check on show access-list output: FWSM(config)# sh access-list | inc flows access-list cached ACL log flows: total 1, denied 1 (deny-flow-max 1) Here I've configured 1 flow. Once you reach the flow limit, the further logs are suppressed (AFAIK, with the logic being, that since the whole idea behind the log is to decrease the amount of logging messages, if we get a lot of hits, we are probably already under stress, so would not want to stress further by downgrading the logs to sending them per-packet). If you have a lot of ACEs that are marked with log keyword, this might be what you see. Decreasing the interval should help to keep the # of logs under max. These are our logging settings : already raised queue size, some messages moved to another log level so they don't get send to our syslog server. ACL log messages are normally of ID 106100 level debugging, I can find several of them on the syslog server but not for the specifiec ACE. For the specific ACE, you can remove the log keyword. Bit counter-intuitive as this might seem, it would not stop the logging for the denied sessions - just the messages will be different (firewall-style): %FWSM-4-106023: Deny icmp src outside:X.1.1.1 dst inside:Y.1.1.1 (type 8, code 0) by access-group foo [0x17a38302, 0x0] instead of: %FWSM-6-106100: access-list foo denied icmp outside/X.1.1.1(0) - inside/Y.1.1.3(8) hit-cnt 1 (first hit) [0xe6aea397, 0x0] That 106023 will be sent one-message-per-hit. So I think it should precisely fit what you are looking for. cheers, andrew ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FWSM logging problem
What code are you on? These types of items have been going on for a while in various iterations of code. There's been so many it's hard for me to keep them straight LOL! But, if you post your code I'll try and look up my notes. In the end, you'll have to call TAC and they will tell you to upgrade to xyz. Try to get a bugid and make sure the recommended upgrade fixes your problem. I've had a couple logging issues that had no id and TAC just said upgrade. As a side note, have you had the issue of traffic blowing by an ACE? :) tv - Original Message - From: Holemans Wim wim.holem...@ua.ac.be To: cisco-nsp@puck.nether.net Sent: Wednesday, December 16, 2009 9:44 AM Subject: [c-nsp] FWSM logging problem It seems our FWSM doesn't log all denied ACLs. I blocked an IP address on our FWSM and wanted to see whomever on campus is trying to access this address (Botnet CC). I added the following line in the ACL (even raised priority), you can see that the rules triggers when I tried to telnet the address : access-list Internet-out line 24 extended deny ip any host X1.X2.X3.X4 log critical interval 30 (hitcnt=9) 0x6e051e8c There is however no corresponding syslog message on our syslog server or in the buffered logs on the FWSM. These are our logging settings : already raised queue size, some messages moved to another log level so they don't get send to our syslog server. ACL log messages are normally of ID 106100 level debugging, I can find several of them on the syslog server but not for the specifiec ACE. logging enable logging timestamp logging emblem logging console debugging logging monitor debugging logging buffered debugging logging trap informational logging asdm informational logging queue 1024 logging host DA-rt x.x.x.x logging message 305010 level debugging logging message 305009 level debugging logging message 302015 level debugging logging message 302014 level debugging logging message 302013 level debugging logging message 302016 level debugging logging message 302021 level debugging Anyone has a clue on how to get all syslog messages for the ACE's that have a log part ? Wim Holemans Netwerkdienst Universiteit Antwerpen ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FWSM logging problem
Tony, As a side note, have you had the issue of traffic blowing by an ACE? :) What you referring to here? I run both the FWSM and ACE module. We have had a plethora of problems with the ACE. The best is it just stops responding and passing traffic and it doesn't failover when that happens. Nick -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Tony Varriale Sent: Wednesday, December 16, 2009 12:31 PM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] FWSM logging problem What code are you on? These types of items have been going on for a while in various iterations of code. There's been so many it's hard for me to keep them straight LOL! But, if you post your code I'll try and look up my notes. In the end, you'll have to call TAC and they will tell you to upgrade to xyz. Try to get a bugid and make sure the recommended upgrade fixes your problem. I've had a couple logging issues that had no id and TAC just said upgrade. As a side note, have you had the issue of traffic blowing by an ACE? :) tv - Original Message - From: Holemans Wim wim.holem...@ua.ac.be To: cisco-nsp@puck.nether.net Sent: Wednesday, December 16, 2009 9:44 AM Subject: [c-nsp] FWSM logging problem It seems our FWSM doesn't log all denied ACLs. I blocked an IP address on our FWSM and wanted to see whomever on campus is trying to access this address (Botnet CC). I added the following line in the ACL (even raised priority), you can see that the rules triggers when I tried to telnet the address : access-list Internet-out line 24 extended deny ip any host X1.X2.X3.X4 log critical interval 30 (hitcnt=9) 0x6e051e8c There is however no corresponding syslog message on our syslog server or in the buffered logs on the FWSM. These are our logging settings : already raised queue size, some messages moved to another log level so they don't get send to our syslog server. ACL log messages are normally of ID 106100 level debugging, I can find several of them on the syslog server but not for the specifiec ACE. logging enable logging timestamp logging emblem logging console debugging logging monitor debugging logging buffered debugging logging trap informational logging asdm informational logging queue 1024 logging host DA-rt x.x.x.x logging message 305010 level debugging logging message 305009 level debugging logging message 302015 level debugging logging message 302014 level debugging logging message 302013 level debugging logging message 302016 level debugging logging message 302021 level debugging Anyone has a clue on how to get all syslog messages for the ACE's that have a log part ? Wim Holemans Netwerkdienst Universiteit Antwerpen ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FWSM logging problem
What does the output of 'show logging queue' look like? Are msgs being actively discarded? How large of a queue depth is too large -- 2048, 4096, 8192? -- Eric Cables On Wed, Dec 16, 2009 at 10:03 AM, nm...@guesswho.com wrote: Tony, As a side note, have you had the issue of traffic blowing by an ACE? :) What you referring to here? I run both the FWSM and ACE module. We have had a plethora of problems with the ACE. The best is it just stops responding and passing traffic and it doesn't failover when that happens. Nick -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto: cisco-nsp-boun...@puck.nether.net] On Behalf Of Tony Varriale Sent: Wednesday, December 16, 2009 12:31 PM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] FWSM logging problem What code are you on? These types of items have been going on for a while in various iterations of code. There's been so many it's hard for me to keep them straight LOL! But, if you post your code I'll try and look up my notes. In the end, you'll have to call TAC and they will tell you to upgrade to xyz. Try to get a bugid and make sure the recommended upgrade fixes your problem. I've had a couple logging issues that had no id and TAC just said upgrade. As a side note, have you had the issue of traffic blowing by an ACE? :) tv - Original Message - From: Holemans Wim wim.holem...@ua.ac.be To: cisco-nsp@puck.nether.net Sent: Wednesday, December 16, 2009 9:44 AM Subject: [c-nsp] FWSM logging problem It seems our FWSM doesn't log all denied ACLs. I blocked an IP address on our FWSM and wanted to see whomever on campus is trying to access this address (Botnet CC). I added the following line in the ACL (even raised priority), you can see that the rules triggers when I tried to telnet the address : access-list Internet-out line 24 extended deny ip any host X1.X2.X3.X4 log critical interval 30 (hitcnt=9) 0x6e051e8c There is however no corresponding syslog message on our syslog server or in the buffered logs on the FWSM. These are our logging settings : already raised queue size, some messages moved to another log level so they don't get send to our syslog server. ACL log messages are normally of ID 106100 level debugging, I can find several of them on the syslog server but not for the specifiec ACE. logging enable logging timestamp logging emblem logging console debugging logging monitor debugging logging buffered debugging logging trap informational logging asdm informational logging queue 1024 logging host DA-rt x.x.x.x logging message 305010 level debugging logging message 305009 level debugging logging message 302015 level debugging logging message 302014 level debugging logging message 302013 level debugging logging message 302016 level debugging logging message 302021 level debugging Anyone has a clue on how to get all syslog messages for the ACE's that have a log part ? Wim Holemans Netwerkdienst Universiteit Antwerpen ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FWSM logging problem
Sorry...Access Control Entry in an ACL on FWSM. What code are you running on 6500 and ACE that you are having these issues? I seen that on the appliances in some early 2.x. tv - Original Message - From: nm...@guesswho.com To: tvarri...@comcast.net; cisco-nsp@puck.nether.net Sent: Wednesday, December 16, 2009 12:03 PM Subject: RE: [c-nsp] FWSM logging problem Tony, As a side note, have you had the issue of traffic blowing by an ACE? :) What you referring to here? I run both the FWSM and ACE module. We have had a plethora of problems with the ACE. The best is it just stops responding and passing traffic and it doesn't failover when that happens. Nick -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Tony Varriale Sent: Wednesday, December 16, 2009 12:31 PM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] FWSM logging problem What code are you on? These types of items have been going on for a while in various iterations of code. There's been so many it's hard for me to keep them straight LOL! But, if you post your code I'll try and look up my notes. In the end, you'll have to call TAC and they will tell you to upgrade to xyz. Try to get a bugid and make sure the recommended upgrade fixes your problem. I've had a couple logging issues that had no id and TAC just said upgrade. As a side note, have you had the issue of traffic blowing by an ACE? :) tv - Original Message - From: Holemans Wim wim.holem...@ua.ac.be To: cisco-nsp@puck.nether.net Sent: Wednesday, December 16, 2009 9:44 AM Subject: [c-nsp] FWSM logging problem It seems our FWSM doesn't log all denied ACLs. I blocked an IP address on our FWSM and wanted to see whomever on campus is trying to access this address (Botnet CC). I added the following line in the ACL (even raised priority), you can see that the rules triggers when I tried to telnet the address : access-list Internet-out line 24 extended deny ip any host X1.X2.X3.X4 log critical interval 30 (hitcnt=9) 0x6e051e8c There is however no corresponding syslog message on our syslog server or in the buffered logs on the FWSM. These are our logging settings : already raised queue size, some messages moved to another log level so they don't get send to our syslog server. ACL log messages are normally of ID 106100 level debugging, I can find several of them on the syslog server but not for the specifiec ACE. logging enable logging timestamp logging emblem logging console debugging logging monitor debugging logging buffered debugging logging trap informational logging asdm informational logging queue 1024 logging host DA-rt x.x.x.x logging message 305010 level debugging logging message 305009 level debugging logging message 302015 level debugging logging message 302014 level debugging logging message 302013 level debugging logging message 302016 level debugging logging message 302021 level debugging Anyone has a clue on how to get all syslog messages for the ACE's that have a log part ? Wim Holemans Netwerkdienst Universiteit Antwerpen ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FWSM logging problem
On Wed, 16 Dec 2009, Holemans Wim wrote: It seems our FWSM doesn't log all denied ACLs. I blocked an IP address on our FWSM and wanted to see whomever on campus is trying to access this address (Botnet CC). I added the following line in the ACL (even raised priority), you can see that the rules triggers when I tried to telnet the address : access-list Internet-out line 24 extended deny ip any host X1.X2.X3.X4 log critical interval 30 (hitcnt=9) 0x6e051e8c There is however no corresponding syslog message on our syslog server or in the buffered logs on the FWSM. Any chances you'd have %FWSM-1-106101: Number of cached deny-flows for ACL log has reached limit somewhere ? Check on show access-list output: FWSM(config)# sh access-list | inc flows access-list cached ACL log flows: total 1, denied 1 (deny-flow-max 1) Here I've configured 1 flow. Once you reach the flow limit, the further logs are suppressed (AFAIK, with the logic being, that since the whole idea behind the log is to decrease the amount of logging messages, if we get a lot of hits, we are probably already under stress, so would not want to stress further by downgrading the logs to sending them per-packet). If you have a lot of ACEs that are marked with log keyword, this might be what you see. Decreasing the interval should help to keep the # of logs under max. These are our logging settings : already raised queue size, some messages moved to another log level so they don't get send to our syslog server. ACL log messages are normally of ID 106100 level debugging, I can find several of them on the syslog server but not for the specifiec ACE. For the specific ACE, you can remove the log keyword. Bit counter-intuitive as this might seem, it would not stop the logging for the denied sessions - just the messages will be different (firewall-style): %FWSM-4-106023: Deny icmp src outside:X.1.1.1 dst inside:Y.1.1.1 (type 8, code 0) by access-group foo [0x17a38302, 0x0] instead of: %FWSM-6-106100: access-list foo denied icmp outside/X.1.1.1(0) - inside/Y.1.1.3(8) hit-cnt 1 (first hit) [0xe6aea397, 0x0] That 106023 will be sent one-message-per-hit. So I think it should precisely fit what you are looking for. cheers, andrew ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FWSM logging problem
Oops..sorry for the confusion. We are working with TAC and the BU directly with this. They are aware of the issue and acknowledge that it is happening across all code releases A2(1.x/2.x/3.x) Unfortunately when this happens you can't even run any diag commands. I have a plugin from TAC that dumps to the Linux shell of the blade but it looks like whatever process that runs away is dynamic and they don't know what it is yet. They acknowledge we are not the only customer. -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Tony Varriale Sent: Wednesday, December 16, 2009 1:34 PM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] FWSM logging problem Sorry...Access Control Entry in an ACL on FWSM. What code are you running on 6500 and ACE that you are having these issues? I seen that on the appliances in some early 2.x. tv - Original Message - From: nm...@guesswho.com To: tvarri...@comcast.net; cisco-nsp@puck.nether.net Sent: Wednesday, December 16, 2009 12:03 PM Subject: RE: [c-nsp] FWSM logging problem Tony, As a side note, have you had the issue of traffic blowing by an ACE? :) What you referring to here? I run both the FWSM and ACE module. We have had a plethora of problems with the ACE. The best is it just stops responding and passing traffic and it doesn't failover when that happens. Nick -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Tony Varriale Sent: Wednesday, December 16, 2009 12:31 PM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] FWSM logging problem What code are you on? These types of items have been going on for a while in various iterations of code. There's been so many it's hard for me to keep them straight LOL! But, if you post your code I'll try and look up my notes. In the end, you'll have to call TAC and they will tell you to upgrade to xyz. Try to get a bugid and make sure the recommended upgrade fixes your problem. I've had a couple logging issues that had no id and TAC just said upgrade. As a side note, have you had the issue of traffic blowing by an ACE? :) tv - Original Message - From: Holemans Wim wim.holem...@ua.ac.be To: cisco-nsp@puck.nether.net Sent: Wednesday, December 16, 2009 9:44 AM Subject: [c-nsp] FWSM logging problem It seems our FWSM doesn't log all denied ACLs. I blocked an IP address on our FWSM and wanted to see whomever on campus is trying to access this address (Botnet CC). I added the following line in the ACL (even raised priority), you can see that the rules triggers when I tried to telnet the address : access-list Internet-out line 24 extended deny ip any host X1.X2.X3.X4 log critical interval 30 (hitcnt=9) 0x6e051e8c There is however no corresponding syslog message on our syslog server or in the buffered logs on the FWSM. These are our logging settings : already raised queue size, some messages moved to another log level so they don't get send to our syslog server. ACL log messages are normally of ID 106100 level debugging, I can find several of them on the syslog server but not for the specifiec ACE. logging enable logging timestamp logging emblem logging console debugging logging monitor debugging logging buffered debugging logging trap informational logging asdm informational logging queue 1024 logging host DA-rt x.x.x.x logging message 305010 level debugging logging message 305009 level debugging logging message 302015 level debugging logging message 302014 level debugging logging message 302013 level debugging logging message 302016 level debugging logging message 302021 level debugging Anyone has a clue on how to get all syslog messages for the ACE's that have a log part ? Wim Holemans Netwerkdienst Universiteit Antwerpen ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FWSM logging problem
- Original Message - From: Andrew Yourtchenko ayour...@cisco.com To: Tony Varriale tvarri...@comcast.net Cc: cisco-nsp@puck.nether.net Sent: Wednesday, December 16, 2009 12:54 PM Subject: Re: [c-nsp] FWSM logging problem That's indeed the proper thing to do. And please, after making sure - also let the case owner know, that it did fix the problem - it's a step sometimes overseen :-) Yup sure is. :( shoot me the case#s unicast, if you still have them. The one I found in a quick search did mention the bug ids along with the pretty detailed explanations for each, but maybe there were some others where there was less info, that I could not find... I haven't fielded one of these in a little while. Last one was earlier this year. I'll have to look. http://www.cisco.com/warp/public/707/cisco-sa-20070214-fwsm.shtml ? There could be some other scenarios where by tweaking the object group one gets the ACL exploded so much that it does not fit into the network processors anymore - then the previously compiled version is being used - but generally you get a pretty prominent warning about that. Nope...NP was fine. How we found it was the ACE not getting hits. So, we then added an ACE next below the one that was getting passed over and it would get hit. Obviously this actually added to the size :) thanks, andrew No problem. :) tv ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FWSM logging problem
On Wed, 16 Dec 2009, Tony Varriale wrote: gets the ACL exploded so much that it does not fit into the network processors anymore - then the previously compiled version is being used - but generally you get a pretty prominent warning about that. Nope...NP was fine. How we found it was the ACE not getting hits. So, we then added an ACE next below the one that was getting passed over and it would get hit. Obviously this actually added to the size :) No, if you'd hit the size limitation you'd see a prominent warning. So got to be something different. If you get this to happen again, that'd be a case indeed. (And if it's something new that's something that we would need to replicate here in the lab, so the more context details you have around it, that might help this effort - the better). kind regards, andrew ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
