'gmirror clear' error
for provider ad3s1f is ufsid/4a3ae4ec312ebf3b. +GEOM_LABEL: Label ufsid/4a3ae4dfa451f7f0 removed. +GEOM_MIRROR: Device gm0 already configured. +GEOM_LABEL: Label for provider ad3s1g is ufsid/4a3ae4dfa451f7f0. +GEOM_LABEL: Label ufsid/4b70da022dd1047b removed. +GEOM_LABEL: Label ufsid/4b70da02f80cbfe5 removed. +GEOM_LABEL: Label ufsid/4b70da0d55f96d5b removed. +GEOM_LABEL: Label ufsid/4b70da0e19c93161 removed. +GEOM_MIRROR: Device gm0: rebuilding provider ad2. +GEOM_MIRROR: Device gm0: rebuilding provider ad2 finished. How concerned to I have to be? I'm not worried about any meta data on /dev/ad3, but if I have to, is there a way to clear the old meta data out of the mirrored devices without taking it down to single user mode again? Thanks! James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: yikes! MAC address changed ??
Sorry for replying to myself (AND top-posting!) twice in a row, but this is become a huge concern. My first thought is that my provider changed routers or router Ethernet ports, hence the MAC address change. They deny this, plus I find the two MAC addresses: 00:17:e0:4f:b9:c0 to 00:13:e0:4f:b9:c0 too close to each other for comfort. My obvious concern here is that the recent php compromises somehow allowed an attacker to alter the ARP table entry of the default gateway. Specific questions are as follows: 1) If this were done via a perl or php script, presumably executing an 'arp -s' command, would it show up in the log like that? I've never changed an ARP entry (except to delete it using 'arp -d'), so I've only seen log entries like that due to external changes, like somebody changing IPs on the LAN from one Ether to another. 2) Could an Ethernet card defect or re0 driver problem cause anything like this? Other bug? 3) If this was an attacker using a local script, how the hell does he get a php or perl script owned by UID 80 (or worst case, a user), to do this? Thanks again for any insight...appreciate a reply to both list and directly. On Wed, 10 Feb 2010, James Smallacombe wrote: Please disregard this...sleep deprication...the IP in questions (which I should have disfuised anyway) was not my server's IP, but that of the default gateway...the problem was external. On Wed, 10 Feb 2010, James Smallacombe wrote: This freaked me out a bit, so I'm just running it past the list to make sure this is just a hardware issue...I've never seen it before. My dedicated server provider replaced my defective server that had been up for 6 months after it had apparent failures of a NIC and hard drives. It had also recently been the victim of the Zen Cart exploits (I posted about this not long ago). Tonight I lost connectivity to it, got in via KVM/IP and saw this in the syslog: Feb 10 20:42:51 mail kernel: arp: 209.17.170.1 moved from 00:17:e0:4f:b9:c0 to 00:13:e0:4f:b9:c0 on re0 My first reaction was that somebody else on the LAN had used my IP address, which would have explained the connectivity issues. However, the IP couldn't be pinged and I also noticed that only one number in the address had changed...the odds of somebody else having it were long. ifconfig showed the I/F down, no carrier. I rebooted and then it came up with yet a third MAC address, 00:14:d1:3c:1e:31 Not really even close. Still no carrier. Provider swaps out the Realtek NIC for a new one and it's working (for now). Questions that come to mind: could their be a DoS perhaps from a bot or c99shell I didn't find? Even if their was, would it be possible for the www user, with no priveleges to even cause this kind of problem? I had disabled suhosin after customers patched their Zen Carts, because it interfered with it. Or...could this be a bug in the re0 driver? It's just weird. James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: yikes! MAC address changed ??
On Thu, 11 Feb 2010, Matthew Seaman wrote: On 11/02/2010 11:00, James Smallacombe wrote: Sorry for replying to myself (AND top-posting!) twice in a row, but this is become a huge concern. My first thought is that my provider changed routers or router Ethernet ports, hence the MAC address change. They deny this, plus I find the two MAC addresses: 00:17:e0:4f:b9:c0 to 00:13:e0:4f:b9:c0 too close to each other for comfort. My obvious concern here is that the recent php compromises somehow allowed an attacker to alter the ARP table entry of the default gateway. Specific questions are as follows: They're not just close: it's a single bit change between the two MACs 1) If this were done via a perl or php script, presumably executing an 'arp -s' command, would it show up in the log like that? I've never changed an ARP entry (except to delete it using 'arp -d'), so I've only seen log entries like that due to external changes, like somebody changing IPs on the LAN from one Ether to another. You'ld need root level access to change something like that, no matter if it was from the shell or via some scripting language. If an attacker has the capability to do that to you, then it's *game* *over* -- wipe the box and start again. Of course, that's a pretty bizarre thing for an attacker to do. It draws attention to itself by disrupting your network communications and there isn't any obvious advantage to be gained by doing that. [There might be if the MAC was changed to collide with another one on the same network segment but I believe that is not the case here.] I figure root at some point is needed, but wondered if there was another POA I had to worry about. In effect, I already wiped out this server a few days ago...new drives with new / FS from 7.2-RELEASE. However, I did copy over /usr/local and /home file systems from the old server's drive, and parts of /var. Everything in / (including /usr) is fresh. It's not 'arp -s' that is used to change the MAC address on an interface, but ifconfig(8) -- something like this: # ifconfig re0 ether 00:17:e0:4f:b9:c0 See my second post. I screwed up in my first post. It wasn't the MAC address of my NIC that changed, it's the MAC address of the DEFAULT GATEWAY that changed. I believe that would use 'arp', not 'ifconfig', right? 2) Could an Ethernet card defect or re0 driver problem cause anything like this? Other bug? Yes -- this is the most likely cause. Hardware problems. The MAC address is built into the network card using an EEPROM or such like, and those can conceivably go bad. Replace the NIC and see if the problems go away. Ok, longer shot here...could a hardware problem on my box screw up the MAC address of the default gateway? It should be noted that when I did and ifconfig -a during this down time, the Ether showed no carrier. Could messed up ARP tables even do that? I would think that the carrier just needs a cable plugged from the NIC into a switch? 3) If this was an attacker using a local script, how the hell does he get a php or perl script owned by UID 80 (or worst case, a user), to do this? You don't. You need root access to change the MAC on a network interface. Same as for changing the IP number on the interface. Check /etc/rc.conf -- if there aren't ifconfig commands in there to modify the ether or link address, and if the modified MAC survives a system reboot, then it's almost certainly hardware going kaput. Even if the MAC does recover on reboot, it still might be flakey hardware. Still had no carrier after reboot. Only after swapping the NIC. Does a reboot wipe out the ARP tables? James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: yikes! MAC address of default gateway changed ??
Hi: Please reply-all ; I am not subscribed On Thu, 11 Feb 2010, Vince Hoffman wrote: On 11/02/2010 11:00, James Smallacombe wrote: Sorry for replying to myself (AND top-posting!) twice in a row, but this is become a huge concern. My first thought is that my provider changed routers or router Ethernet ports, hence the MAC address change. They deny this, plus I find the two MAC addresses: 00:17:e0:4f:b9:c0 to 00:13:e0:4f:b9:c0 On 11/02/2010 11:00, James Smallacombe wrote: Sorry for replying to myself (AND top-posting!) twice in a row, but this is become a huge concern. My first thought is that my provider changed routers or router Ethernet ports, hence the MAC address change. They deny this, plus I find the two MAC addresses: 00:17:e0:4f:b9:c0 to 00:13:e0:4f:b9:c0 However in your case, while 00:17:E0 is reasonable (a cisco mac address) 00:13:E0 is a little worrying as apparently its a Murata Manufacturing(whoever they are) mac address (see http://www.coffer.com/mac_find/?string=00%3A13%3Ae0%3A4f%3Ab9%3Ac0) Well, that rules out anything by the provider. you can check if its a static entry in your arp tables using arp -a | grep permanent The only permanent entries should be your local IPs (whatever you have configured on your interfaces) unless you have any others you have put in yourself. so for my server i have r...@seaurchin ~]# arp -a | grep permanent seaurchin.the.namesco.net (85.233.xxx.xxx) at 00:11:43:d8:2c:df on em0 permanent [ethernet] ? (10.20.0.3) at 00:11:43:d8:2c:df on em0 permanent [ethernet] Obviously the ARP entry is long gone now and I don't recall if it was permanent or not. It just leaves a couple of questions: If it was caused by a malicious arp command on my server, wouldn't a reboot have gotten rid of it? Would it also result in a NO CARRIER on the interface? Network did not come back until the Ethernet card was swapped. The bottom line is whether it is possible for a NIC failure to cause the kernel to register an ARP change. Thanks again to everyone... James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Mac address changed ??
This freaked me out a bit, so I'm just running it past the list to make sure this is just a hardware issue...I've never seen it before. My dedicated server provider replaced my defective server that had been up for 6 months after it had apparent failures of a NIC and hard drives. It had also recently been the victim of the Zen Cart exploits (I posted about this not long ago). Tonight I lost connectivity to it, got in via KVM/IP and saw this in the syslog: Feb 10 20:42:51 mail kernel: arp: 209.17.170.1 moved from 00:17:e0:4f:b9:c0 to 00:13:e0:4f:b9:c0 on re0 My first reaction was that somebody else on the LAN had used my IP address, which would have explained the connectivity issues. However, the IP couldn't be pinged and I also noticed that only one number in the address had changed...the odds of somebody else having it were long. ifconfig showed the I/F down, no carrier. I rebooted and then it came up with yet a third MAC address, 00:14:d1:3c:1e:31 Not really even close. Still no carrier. Provider swaps out the Realtek NIC for a new one and it's working (for now). Questions that come to mind: could their be a DoS perhaps from a bot or c99shell I didn't find? Even if their was, would it be possible for the www user, with no priveleges to even cause this kind of problem? I had disabled suhosin after customers patched their Zen Carts, because it interfered with it. Or...could this be a bug in the re0 driver? It's just weird. James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Mac address changed ??
Please disregard this...sleep deprication...the IP in questions (which I should have disfuised anyway) was not my server's IP, but that of the default gateway...the problem was external. On Wed, 10 Feb 2010, James Smallacombe wrote: This freaked me out a bit, so I'm just running it past the list to make sure this is just a hardware issue...I've never seen it before. My dedicated server provider replaced my defective server that had been up for 6 months after it had apparent failures of a NIC and hard drives. It had also recently been the victim of the Zen Cart exploits (I posted about this not long ago). Tonight I lost connectivity to it, got in via KVM/IP and saw this in the syslog: Feb 10 20:42:51 mail kernel: arp: 209.17.170.1 moved from 00:17:e0:4f:b9:c0 to 00:13:e0:4f:b9:c0 on re0 My first reaction was that somebody else on the LAN had used my IP address, which would have explained the connectivity issues. However, the IP couldn't be pinged and I also noticed that only one number in the address had changed...the odds of somebody else having it were long. ifconfig showed the I/F down, no carrier. I rebooted and then it came up with yet a third MAC address, 00:14:d1:3c:1e:31 Not really even close. Still no carrier. Provider swaps out the Realtek NIC for a new one and it's working (for now). Questions that come to mind: could their be a DoS perhaps from a bot or c99shell I didn't find? Even if their was, would it be possible for the www user, with no priveleges to even cause this kind of problem? I had disabled suhosin after customers patched their Zen Carts, because it interfered with it. Or...could this be a bug in the re0 driver? It's just weird. James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Server compromised Zen-Cart record company Exploit
Replying to Bogdan Webb's reply recommending sohusin:
This appears to be exactly what I needed, thanks! The stock ports PHP
install already has the suhosin patch, but the extension is a godsend!
Not only does it log everything, but it let's you manage php functions on
a per virtual host basis, not just in php.ini. Fantastic and is working
great. About the only thing I could want more would be to control the
functions under the apache Directory directives (on top of in
VirtualHost).
On Mon, 1 Feb 2010, James Smallacombe wrote:
(please reply-all; I am not sub'd and sorry for the top posting):
I have safe_mode off due to popular demand. So many customer apps demand
that it be kept off. In fact, here is a post from one of the Zen people on
the Zen-cart forum. In light of this exploit, this might be a little ironic:
http://www.zen-cart.com/forum/showthread.php?t=76740
There is one for-sure patch: Turn off safe-mode.
Keep in mind that future versions of PHP will *not* even include a safe-mode
... because it's a weak bandage giving a false sense of security to hosts who
don't otherwise know how to properly secure their servers.
This begs the question: why? ie: why would you want to run your online
business on a server that's got to use safe-mode in order to think they're
securing the server?
I'm not trying to badmouth your server administrator; rather I'm attempting
to strongly make the point that unless safe-mode is being used for a very
specific reason for which there is no other solution (an unlikely situation),
it shouldn't be used. And, if it is being used, you shouldn't run your
business there, because there will be other security issues to which you'll
be vulnerable but never have a clue about it until disaster strikes, because
the big picture of security protection has been poorly implemented.
That said, Zen Cart will install and run even if Safe Mode is active;
however, you run the risk of certain features not working with or without
notice, and the unexpected appearance of warning or fatal errors while
customers are using the site. And then there's the issue of the admin side
needing to do various things that safe-mode doesn't like.
So, I guess, in short ... you can do it, but you do so at your own risk.
Maybe that's more than you wanted to hear ... sorry
From: Bogdan Webb bog...@pgn.ro
try php's safe_mode but it is likely to keep the hackers off, indeed they
can get in and snatch some data but they would be kept out of a shell's
reach... but sometimes safe_mode is not enough... try considering Suhosin
but the addon not the patch... and define the
suhosin.executor.func.blacklist witch will deny use of certain php commands
that allow shell execution... but keep in mind it's impossible to prevent
all breaches... this php patch will only keep the hacker kiddos off but
there's still a good chance it can be broken... stay safe !
ref's:
http://www.hardened-php.net/suhosin.127.html
http://beta.pgn.ro/phps/phpinfo.php
On Sun, 31 Jan 2010, James Smallacombe wrote:
Whoever speculated that my server may have been compromised was on to
something (see bottom). The good news is, it does appear to be contained
to the www unpriveleged user (with no shell). The bad news is, they can
still cause a lot of trouble. I found the compromised customer site and
chmod 0 their cart (had php binaries called core(some number).php that
gave the hacker a nice browser screen to cause all kinds of trouble)
Not sure if this is related to the UDP floods, but if not, it's a heck of a
coincidence. At times, CPU went through the roof for the www user, mostly
running some sort of perl scripts (nothing in the suexec-log). I would
kill apache, but couldn't restart it as it would show port 80 in use. I
would have to manually kill processes like these:
www 70471 1.4 0.1 6056 3824 ?? R 4:21PM 0:44.75 [eth0] (perl)
www 70470 1.2 0.1 6060 3828 ?? R 4:21PM 0:44.50 [bash] (perl)
www 64779 1.0 0.1 6056 3820 ?? R 4:07PM 2:24.34
/sbin/klogd -c 1 -x -x (perl)
www 70472 1.0 0.1 6060 3828 ?? R 4:21PM 0:44.84
I could not find ANY file named klogd on the system, let alone in /sbin.
Clues as to how to dig myself out of this are appreciated
I found this in /tmp/bx1.txt:
--More--(5%)#!/usr/bin/php
?php
#
# --- Zen Cart 1.3.8 Remote Code Execution
# http://www.zen-cart.com/
# Zen Cart Ecommerce - putting the dream of server rooting within reach of
anyone!
# A new version (1.3.8a) is avaible on http://www.zen-cart.com/
#
# BlackH :)
#
error_reporting(E_ALL ^ E_NOTICE);
if($argc 2)
{
echo
=___ Zen Cart 1.3.8 Remote Code Execution Exploit =
| BlackH bl4c...@gmail.com
Re: Server compromised Zen-Cart record company Exploit
(please reply-all; I am not sub'd and sorry for the top posting):
I have safe_mode off due to popular demand. So many customer apps demand
that it be kept off. In fact, here is a post from one of the Zen people
on the Zen-cart forum. In light of this exploit, this might be a little
ironic:
http://www.zen-cart.com/forum/showthread.php?t=76740
There is one for-sure patch: Turn off safe-mode.
Keep in mind that future versions of PHP will *not* even include a
safe-mode ... because it's a weak bandage giving a false sense of security
to hosts who don't otherwise know how to properly secure their servers.
This begs the question: why? ie: why would you want to run your online
business on a server that's got to use safe-mode in order to think they're
securing the server?
I'm not trying to badmouth your server administrator; rather I'm
attempting to strongly make the point that unless safe-mode is being used
for a very specific reason for which there is no other solution (an
unlikely situation), it shouldn't be used. And, if it is being used, you
shouldn't run your business there, because there will be other security
issues to which you'll be vulnerable but never have a clue about it until
disaster strikes, because the big picture of security protection has been
poorly implemented.
That said, Zen Cart will install and run even if Safe Mode is active;
however, you run the risk of certain features not working with or without
notice, and the unexpected appearance of warning or fatal errors while
customers are using the site. And then there's the issue of the admin side
needing to do various things that safe-mode doesn't like.
So, I guess, in short ... you can do it, but you do so at your own risk.
Maybe that's more than you wanted to hear ... sorry
From: Bogdan Webb bog...@pgn.ro
try php's safe_mode but it is likely to keep the hackers off, indeed they
can get in and snatch some data but they would be kept out of a shell's
reach... but sometimes safe_mode is not enough... try considering Suhosin
but the addon not the patch... and define the
suhosin.executor.func.blacklist witch will deny use of certain php
commands
that allow shell execution... but keep in mind it's impossible to prevent
all breaches... this php patch will only keep the hacker kiddos off but
there's still a good chance it can be broken... stay safe !
ref's:
http://www.hardened-php.net/suhosin.127.html
http://beta.pgn.ro/phps/phpinfo.php
On Sun, 31 Jan 2010, James Smallacombe wrote:
Whoever speculated that my server may have been compromised was on to
something (see bottom). The good news is, it does appear to be contained to
the www unpriveleged user (with no shell). The bad news is, they can still
cause a lot of trouble. I found the compromised customer site and chmod 0
their cart (had php binaries called core(some number).php that gave the
hacker a nice browser screen to cause all kinds of trouble)
Not sure if this is related to the UDP floods, but if not, it's a heck of a
coincidence. At times, CPU went through the roof for the www user, mostly
running some sort of perl scripts (nothing in the suexec-log). I would kill
apache, but couldn't restart it as it would show port 80 in use. I would
have to manually kill processes like these:
www 70471 1.4 0.1 6056 3824 ?? R 4:21PM 0:44.75 [eth0] (perl)
www 70470 1.2 0.1 6060 3828 ?? R 4:21PM 0:44.50 [bash] (perl)
www 64779 1.0 0.1 6056 3820 ?? R 4:07PM 2:24.34
/sbin/klogd -c 1 -x -x (perl)
www 70472 1.0 0.1 6060 3828 ?? R 4:21PM 0:44.84
I could not find ANY file named klogd on the system, let alone in /sbin.
Clues as to how to dig myself out of this are appreciated
I found this in /tmp/bx1.txt:
--More--(5%)#!/usr/bin/php
?php
#
# --- Zen Cart 1.3.8 Remote Code Execution
# http://www.zen-cart.com/
# Zen Cart Ecommerce - putting the dream of server rooting within reach of
anyone!
# A new version (1.3.8a) is avaible on http://www.zen-cart.com/
#
# BlackH :)
#
error_reporting(E_ALL ^ E_NOTICE);
if($argc 2)
{
echo
=___ Zen Cart 1.3.8 Remote Code Execution Exploit =
| BlackH bl4c...@gmail.com |
| |
| \$system php $argv[0] url|
| Notes: url ex: http://victim.com/site (no slash) |
| |
;exit(1);
--- snipped --
It is dated from two nights ago, after these issues started, but it's
nonetheless larming. Security Focus is aware of the issue and refers you to
Zen for the fix. Only problem
Server compromised Zen-Cart record company Exploit
Whoever speculated that my server may have been compromised was on to
something (see bottom). The good news is, it does appear to be contained
to the www unpriveleged user (with no shell). The bad news is, they can
still cause a lot of trouble. I found the compromised customer site and
chmod 0 their cart (had php binaries called core(some number).php that
gave the hacker a nice browser screen to cause all kinds of trouble)
Not sure if this is related to the UDP floods, but if not, it's a heck of
a coincidence. At times, CPU went through the roof for the www user,
mostly running some sort of perl scripts (nothing in the suexec-log). I
would kill apache, but couldn't restart it as it would show port 80 in
use. I would have to manually kill processes like these:
www 70471 1.4 0.1 6056 3824 ?? R 4:21PM 0:44.75 [eth0] (perl)
www 70470 1.2 0.1 6060 3828 ?? R 4:21PM 0:44.50 [bash] (perl)
www 64779 1.0 0.1 6056 3820 ?? R 4:07PM 2:24.34
/sbin/klogd -c 1 -x -x (perl)
www 70472 1.0 0.1 6060 3828 ?? R 4:21PM 0:44.84
I could not find ANY file named klogd on the system, let alone in /sbin.
Clues as to how to dig myself out of this are appreciated
I found this in /tmp/bx1.txt:
--More--(5%)#!/usr/bin/php
?php
#
# --- Zen Cart 1.3.8 Remote Code Execution
# http://www.zen-cart.com/
# Zen Cart Ecommerce - putting the dream of server rooting within reach of
anyone!
# A new version (1.3.8a) is avaible on http://www.zen-cart.com/
#
# BlackH :)
#
error_reporting(E_ALL ^ E_NOTICE);
if($argc 2)
{
echo
=___ Zen Cart 1.3.8 Remote Code Execution Exploit =
| BlackH bl4c...@gmail.com |
| |
| \$system php $argv[0] url|
| Notes: url ex: http://victim.com/site (no slash) |
| |
;exit(1);
--- snipped --
It is dated from two nights ago, after these issues started, but it's
nonetheless larming. Security Focus is aware of the issue and refers you
to Zen for the fix. Only problem is, this is an old version of Zen cart,
and the
James Smallacombe PlantageNet, Inc. CEO and Janitor
u...@3.am http://3.am
=
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
UDP flooding / Ethernet issues? WAS Re: named error sending response: not enough free resources
On Thu, Jan 28, 2010 at 12:59 PM, James Smallacombe u...@3.am wrote: To follow up on this: Noticed the issue again this morning, which also was accompanied by latency so high that I could not connect (some pings got through at very high latency). I emailed the provider and they told me that they had my port on their Ether switch set to 10Mbs. They switched it to 100Mbs and only time will tell if that fixes it. Does this sound like it could be the entire cause? I ask because I've maxed out pipes before, but never seen it shut all traffic down this much. One key difference that I forgot to mention is that this server is running TWO instances of named, on two different IPs (for different domains), each running a few hundred zones. Bottom line: Would congestion cause this issue, or would this issue cause congestion? Some updates that may confuse more than inform: I caught this while it was happening yesterday and was able to do a tcpdump. I saw a ton of UDP traffic outbound to one IP that turned out to be a colocated server in Chicago. I put that IP in my ipfw rules and once I blocked any to that IP, it seemed to stop. Since then however, the logs have show the same issue again and there have been a few brief service disruptions. Today's security run output showed this: +(RULE NUMBER) 16054161 131965203420 deny ip from any to (blocked IP) and more alarmingly, this: kernel log messages: +++ /tmp/security.BErFHSS3 2010-01-29 03:09:32.0 -0500 +re0: link state changed to DOWN +re0: link state changed to UP +re0: promiscuous mode enabled +re0: promiscuous mode disabled +re0: promiscuous mode enabled +re0: promiscuous mode disabled +re0: promiscuous mode enabled +re0: promiscuous mode disabled re0 obviously being the Realtek Ethernet driver. The server itself never went down during this time, but the Ethernet did. Is there any DOS type of event that could cause this, or could the root of the problem be an Ethernet hardware or driver issue? Again, it is not clear to me which is the cause and which is the effect. Last bit of info: I just did a: 'tcpdump -n | grep -i udp' and saw a bunch of these, coming up a couple of times per second: 11:31:59.387561 IP (IP REMOVED) (IP REMOVED): NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST Where the source and destination IPs vary, but are NOT one of mine, but DO appear to belong to my colo/dedicated server provider and their customers. Is my server being used to DDOS others? If so, how? TIA, James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named error sending response: not enough free resources
On Wed, 27 Jan 2010, Chuck Swiger wrote: Hi-- On Jan 27, 2010, at 1:15 PM, James Smallacombe wrote: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources Jan 26 21:50:32 host named[667]: client IP REMOVED#59830: error sending response: not enough free resources OK, if the nameserver is published / authoritative, then it would be expected to be fielding requests from the Internet at large. To follow up on this: Noticed the issue again this morning, which also was accompanied by latency so high that I could not connect (some pings got through at very high latency). I emailed the provider and they told me that they had my port on their Ether switch set to 10Mbs. They switched it to 100Mbs and only time will tell if that fixes it. Does this sound like it could be the entire cause? I ask because I've maxed out pipes before, but never seen it shut all traffic down this much. One key difference that I forgot to mention is that this server is running TWO instances of named, on two different IPs (for different domains), each running a few hundred zones. Bottom line: Would congestion cause this issue, or would this issue cause congestion? Thanks again! James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
named error sending response: not enough free resources
NOTE: Please reply off-list as well as I am not subscribed My server (7.2-STABLE) suffered at least two outages Sunday through yesterday after having been up since July (it is a rented dedicated server with my FSBD install). The first time, I was able to log in via remotely, saw a ton of spam apparently abusing a php mail form script (more on that later) filling the /var partition. I purged it, but it still required a reboot as CPU was through the roof. Yesterday morning, I was unable to get into the server at all...pings were very high. I called the provider and got in via KVM over IP. CPU was fine and there wre no full partitions. As I had to catch a flight, I just rebooted it and it was fine. After getting home, I looked in the syslog and see thousands of these: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources Jan 26 21:50:32 host named[667]: client IP REMOVED#59830: error sending response: not enough free resources Some googling on this error found a reference to a possible queue limiting problem in pf/qlimit, but the only firewalling I do is a very basic ipfw setup strictly for bruteblock. I am not even sure if this error caused the outage(s) or was caused by them, let alone a fix or workaround. Appreciate any and all clues, especially if you are familiar with this. TIA! James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named error sending response: not enough free resources
On Wed, 27 Jan 2010, Chuck Swiger wrote: On Jan 27, 2010, at 10:24 AM, James Smallacombe wrote: NOTE: Please reply off-list as well as I am not subscribed OK. In return, please don't cross-post or multi-post the same question to multiple FreeBSD lists. I posted to the -isp list a couple of hours earlier, then looked at the archives and noticed zero traffic on that list for the past couple of weeks, so I then posted here. After getting home, I looked in the syslog and see thousands of these: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources Jan 26 21:50:32 host named[667]: client IP REMOVED#59830: error sending response: not enough free resources Were these client IPs expected to be talking to this machine? It This server is authoritative for a few hundred domains, so I would imagine anybody doing a query on any of them would need to talk to it...unless I misunderstand what you mean by talk. indicates a problem sending UDP traffic; netstat -s output would be Unfortunately, I did not have time for netstats or tcpdumps when this was happening and I've not seen this log entry since yesterday evening. informative. You might find that setting options in named.conf to tune the # of outstanding queries will help: clients-per-query 10; max-clients-per-query 20; Thanks, I will look into those. the man page for named.conf doesn't tell you much and my latest cricket book is 3rd edition (only up to BIND 8), so I guess it's time to break down and get the latest. James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named error sending response: not enough free resources
On Wed, 27 Jan 2010, Chuck Swiger wrote: On Jan 27, 2010, at 1:15 PM, James Smallacombe wrote: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources indicates a problem sending UDP traffic; netstat -s output would be Unfortunately, I did not have time for netstats or tcpdumps when this was happening and I've not seen this log entry since yesterday evening. Unless you rebooted the machine again since the errors were reported, the netstat output would still be relevant. Ok, I saw this at least once since the last reboot, so here are the tcp and udp portions of the netstat -s: tcp: 31422122 packets sent 23133142 data packets (3473553079 bytes) 314215 data packets (132175418 bytes) retransmitted 6579 data packets unnecessarily retransmitted 11 resends initiated by MTU discovery 5408494 ack-only packets (200066 delayed) 0 URG only packets 1237 window probe packets 868892 window update packets 1713629 control packets 28600984 packets received 17029642 acks (for 3351867346 bytes) 1256410 duplicate acks 73760 acks for unsent data 11363962 packets (548204663 bytes) received in-sequence 184682 completely duplicate packets (16657176 bytes) 2327 old duplicate packets 1468 packets with some dup. data (339128 bytes duped) 334018 out-of-order packets (337877573 bytes) 85687 packets (637782 bytes) of data after window 10 window probes 114047 window update packets 160975 packets received after close 1148 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 9123 discarded due to memory problems 413250 connection requests 1504359 connection accepts 6 bad connection attempts 100 listen queue overflows 186225 ignored RSTs in the windows 1912682 connections established (including accepts) 2050764 connections closed (including 1022550 drops) 1058803 connections updated cached RTT on close 1065370 connections updated cached RTT variance on close 252114 connections updated cached ssthresh on close 3769 embryonic connections dropped 11958433 segments updated rtt (of 11574855 attempts) 285733 retransmit timeouts 12079 connections dropped by rexmit timeout 1884 persist timeouts 4 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 385 keepalive timeouts 345 keepalive probes sent 40 connections dropped by keepalive 2663719 correct ACK header predictions 5996181 correct data packet header predictions 1520655 syncache entries added 58477 retransmitted 26560 dupsyn 20622 dropped 1504359 completed 137 bucket overflow 0 cache overflow 6190 reset 10206 stale 100 aborted 0 badack 47 unreach 0 zone failures 1541277 cookies sent 415 cookies received 21638 SACK recovery episodes 37110 segment rexmits in SACK recovery episodes 51620488 byte rexmits in SACK recovery episodes 240368 SACK options (SACK blocks) received 217836 SACK options (SACK blocks) sent 0 SACK scoreboard overflow udp: 9663633 datagrams received 0 with incomplete header 0 with bad data length field 549 with bad checksum 9609 with no checksum 12092 dropped due to no socket 49230 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 9601762 delivered 42443353 datagrams output 0 times multicast source filter matched James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Copying binaries to new server
A couple of months ago, I spent a couple of weeks compiling and configuring the latest FBSD, apache, perl, qmail and the bazzilion modules, patches and addon apps that go with all of it on an existing server, and ironing out all the upgrade issues that entailed. Since then, due to apparent hardware problems with that server, I just put together a new server using new hardware. The old hardware was dual P-III, Adaptec SCSI RAID 1 The new hardware is single Xeon, LSI SAS RAID 1 Both running 6.2-Prerelease. Is there any reason I shouldn't just copy all of /usr and /var from the old server, or do I really need to compile everything anew and sort out any simlinks to other file systems? Please copy me directly, since I am no subscribed TIA! James SmallacombeInternet Access for The Delaware [EMAIL PROTECTED]Valley in PA, NJ and DE PlantageNet Internet Ltd.http://www.pil.net = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Copying binaries to new server
On Tue, 24 Oct 2006, Eric wrote: James Smallacombe wrote: A couple of months ago, I spent a couple of weeks compiling and configuring the latest FBSD, apache, perl, qmail and the bazzilion modules, patches and addon apps that go with all of it on an existing server, and ironing out all the upgrade issues that entailed. Since then, due to apparent hardware problems with that server, I just put together a new server using new hardware. The old hardware was dual P-III, Adaptec SCSI RAID 1 The new hardware is single Xeon, LSI SAS RAID 1 Both running 6.2-Prerelease. Is there any reason I shouldn't just copy all of /usr and /var from the old server, or do I really need to compile everything anew and sort out any simlinks to other file systems? Please copy me directly, since I am no subscribed TIA! why not just a dump/restore of the file systems in question? That's what I'm asking...I just want to make sure that binaries compiled on a dual PIII wouldn't have stability issues running on a single Xeon...I know they're both I386, but am uncertain as to how system-specific binaries can be... James SmallacombeInternet Access for The Delaware [EMAIL PROTECTED]Valley in PA, NJ and DE PlantageNet Internet Ltd.http://www.pil.net = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Copying binaries to new server
On Tue, 24 Oct 2006, Eric wrote: James Smallacombe wrote: A couple of months ago, I spent a couple of weeks compiling and configuring the latest FBSD, apache, perl, qmail and the bazzilion modules, patches and addon apps that go with all of it on an existing server, and ironing out all the upgrade issues that entailed. Since then, due to apparent hardware problems with that server, I just put together a new server using new hardware. The old hardware was dual P-III, Adaptec SCSI RAID 1 The new hardware is single Xeon, LSI SAS RAID 1 Both running 6.2-Prerelease. Is there any reason I shouldn't just copy all of /usr and /var from the old server, or do I really need to compile everything anew and sort out any simlinks to other file systems? Please copy me directly, since I am no subscribed why not just a dump/restore of the file systems in question? Here's another issue I just ran into while trying to do just that, using tar: su-2.05b# tar xpPvfz snip thousands of lines x /usr/lib/libtacplus.so.2 x /usr/lib/libtacplus.so x /usr/lib/libutil.a x /usr/lib/libutil.so x /usr/lib/libypclnt.a x /usr/lib/libypclnt.so.2 x /usr/lib/libypclnt.so x /usr/lib/libalias.a x /usr/lib/libalias.so x /usr/lib/libarchive.a x /usr/lib/libarchive.so.2Bus error: 10 (core dumped) I take it the core dump occured because I was trying to overwrite a lib that was in use by tar, right? Is there a good way around this? TIA, James SmallacombeInternet Access for The Delaware [EMAIL PROTECTED]Valley in PA, NJ and DE PlantageNet Internet Ltd.http://www.pil.net = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Copying binaries to new server
On Tue, 24 Oct 2006, Dan Nelson wrote: In the last episode (Oct 24), James Smallacombe said: Here's another issue I just ran into while trying to do just that, using tar: su-2.05b# tar xpPvfz snip thousands of lines x /usr/lib/libypclnt.so x /usr/lib/libalias.a x /usr/lib/libalias.so x /usr/lib/libarchive.a x /usr/lib/libarchive.so.2Bus error: 10 (core dumped) I take it the core dump occured because I was trying to overwrite a lib that was in use by tar, right? Is there a good way around this? You can try the -U option, which will unlink existing files before creating the new version. I just noticed the problem was in creating the tarball, not extracting itI will try the -W exclude=libarchive.* first... thanks James SmallacombeInternet Access for The Delaware [EMAIL PROTECTED]Valley in PA, NJ and DE PlantageNet Internet Ltd.http://www.pil.net = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Copying binaries to new server
On Tue, 24 Oct 2006, Jerry McAllister wrote: True, but I think the poster was suggesting that dump/restore is a better way than using tar. I'm not as familiar with BSD dump...does it compress well? Also, what's this? su-2.05b# dump -0L -f ns1.usr.dump /usr DUMP: Date of this level 0 dump: Tue Oct 24 15:52:01 2006 DUMP: Date of last level 0 dump: the epoch DUMP: Dumping snapshot of /dev/da0s1d (/usr) to ns1.usr.dump DUMP: mapping (Pass I) [regular files] DUMP: mapping (Pass II) [directories] DUMP: estimated 3077070 tape blocks on 79.03 tape(s). DUMP: dumping (Pass III) [directories] DUMP: Closing ns1.usr.dump DUMP: Change Volumes: Mount volume #2 DUMP: Is the new volume mounted and ready to go?: (yes or no) yes DUMP: Volume 2 begins with blocks from inode 149561 DUMP: Closing ns1.usr.dump DUMP: Change Volumes: Mount volume #3 DUMP: Is the new volume mounted and ready to go?: (yes or no) What volume? The one I'm dumping? If so, why does it keep asking whether it's mounted? What are all these different volume numbers? I just want to dump /usr to one file, compressing and preserving permissions and symlinks as much as possible, so I can restore it to a new server. James SmallacombeInternet Access for The Delaware [EMAIL PROTECTED]Valley in PA, NJ and DE PlantageNet Internet Ltd.http://www.pil.net = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Copying binaries to new server
On Tue, 24 Oct 2006, Eric wrote: James Smallacombe wrote: On Tue, 24 Oct 2006, Jerry McAllister wrote: True, but I think the poster was suggesting that dump/restore is a better way than using tar. I'm not as familiar with BSD dump...does it compress well? Also, what's this? Take a dump dump -0uanLf - /var | bzip2 | dd of=/some/path/dump-var-level0.bz2 dump -0uanLf - / | bzip2 | dd of=/some/path/dump-root-level0.bz2 dump -0uanLf - /usr | bzip2 | dd of=/some/path/dump-usr-level0.bz2 Restore a dump To Restore Interactively Thanks, this seems to be doing the trick. However, I am getting a ton of these upon restoring the contents of /usr on the new server: warning: cannot create symbolic link ./src/sys/i386/compile/NEW_NS1.SMP/modules/usr/src/sys/modules/if_ppp/machine-/usr/src/sys/i386/include: File exists Is it just that the symlinks already exist on the new server and it won't overwrite them the way they overwrite regular files? Thanks again, James SmallacombeInternet Access for The Delaware [EMAIL PROTECTED]Valley in PA, NJ and DE PlantageNet Internet Ltd.http://www.pil.net = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Copying binaries to new server
On Tue, 24 Oct 2006, James Smallacombe wrote: On Tue, 24 Oct 2006, Eric wrote: James Smallacombe wrote: On Tue, 24 Oct 2006, Jerry McAllister wrote: True, but I think the poster was suggesting that dump/restore is a better way than using tar. I'm not as familiar with BSD dump...does it compress well? Also, what's this? Take a dump dump -0uanLf - /var | bzip2 | dd of=/some/path/dump-var-level0.bz2 dump -0uanLf - / | bzip2 | dd of=/some/path/dump-root-level0.bz2 dump -0uanLf - /usr | bzip2 | dd of=/some/path/dump-usr-level0.bz2 Restore a dump To Restore Interactively Thanks, this seems to be doing the trick. However, I am getting a ton of these upon restoring the contents of /usr on the new server: warning: cannot create symbolic link ./src/sys/i386/compile/NEW_NS1.SMP/modules/usr/src/sys/modules/if_ppp/machine-/usr/src/sys/i386/include: File exists Is it just that the symlinks already exist on the new server and it won't overwrite them the way they overwrite regular files? Ugh...my ssh session to the new (remote) server was killed, presumably when restore overwrote something sshd needed. I was able to telnet back in, restart sshd and get back in, but since I was doing the restore interactively, and not in the background, was the restore interrupted? This is the problem with trying to copy a /usr or / file system to a live system... James SmallacombeInternet Access for The Delaware [EMAIL PROTECTED]Valley in PA, NJ and DE PlantageNet Internet Ltd.http://www.pil.net = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
