Re: [CentOS] Centos/Linux Disk Caching, might be OT in some ways
Hi, Split the TEXT/BLOB data out of the primary table into tables of their own indexed to the primary table by it's key column. This is part of what I was planning to do, there are a lot of stuff I am planning to split out into their own tables with reference key. The problem is I'm unsure whether the added overheads of joins would negate the IO benefits hence trying to figure out more about how Centos/Linux does the caching. Think about distributing the parts to different boxes as necessary. You can start with the DBMS which is the logical candidate. Eventually I figured that would probably have to be done but I don't know enough at this point. So I'm taking the approach of optimizing stage by stage starting with things I'm more familiar with and less likely to muck up totally, i.e.from the app/script side first. Then after getting more familiar with the setup, experiment with the hardware based solutions. On the DBMS backend, give it plenty of memory, good storage for the workload and good networking. Again problem is old server so memory is maxed, drives controller is probably not helping. On the Apache/PHP side, look for a good DBMS inter-connect and some PHP caching module and of course enough CPU for the PHP code and network for Apache+DBMS inter-connect. If you wanted to split it up even more you could look into some sort of PHP distributed cache/processing system and have PHP processed behind Apache. Thanks for the heads up, I didn't realize it was possible to separate the PHP processing from Apache itself. However, for the time being, I'm probably still limited to a single server situation so will keep this in mind for future. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Centos/Linux Disk Caching, might be OT in some ways
Hi, I believe the OP said he was running postgresql. Quoted from OPs previous mail hes not sure lol The web application is written in PHP and runs off MySQL and/or Postgresql. Ah, well #1 on his list then is to figure out what he is running! LOL, I know it sounds quite noobish, coming across like I've no idea what DBMS it is running on. The system currently runs on MySQL but part of my update requirement was to decouple the DBMS so that we can make an eventual switch to postgresql. Hence the solution cannot be dependent on some specific MySQL functionality. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Centos/Linux Disk Caching, might be OT in some ways
MySQL's acquisition was one of the factor, the client wants to keep everything on the opensource side as far as possible. On the technical side, all tables are using the InnoDB engine because myISAM doesn't support either. Also previously during development, it was discovered that on some particular application/function, MyISAM caused a heavy load that went away after switching to InnoDB. Also, as part of my idea was to subsequently put the tables on different disks for better improvement. Postgresql supports that while MySQL appears to require all the tables remain on the same filesystem. There were other considerations that was discussed internally previously but without digging up docs, off hand, these are the key factors I can recall that drove the decision to eventually replace MySQL with Postgresql. On 1/27/10, Chan Chung Hang Christopher christopher.c...@bradbury.edu.hk wrote: Ah, well #1 on his list then is to figure out what he is running! LOL, I know it sounds quite noobish, coming across like I've no idea what DBMS it is running on. The system currently runs on MySQL but part of my update requirement was to decouple the DBMS so that we can make an eventual switch to postgresql. Hence the solution cannot be dependent on some specific MySQL functionality. mysql's isam tables have a reputation for surviving just about anything and great builtin replication support... postgresql less so (I suspect due to fake fsync/fsyncdata in the days before barriers) but maybe things have improved a lot nowadays. Why are you switching? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Centos/Linux Disk Caching, might be OT in some ways
Hi, On 1/27/10, Ross Walker rswwal...@gmail.com wrote: But if your doing mysql on top of LVM your basically doing the same, cause LVM (other then current kernels) doesn't support barriers. Still if you have a battery backed write-caching controller that negates the fsync risk, LVM or not, mysql or postgresql. This is a bit of a surpise. Am I understanding correctly that running postgresql or mysql on top of LVM negates any data reliability measures the DBMS might have in the event of an unexpected shutdown? I have several servers configured to run LVM on top of MD1 for the convenience of being able to add more space to a volume in the future. I didn't realize this was a reliability risk. :( ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Centos/Linux Disk Caching, might be OT in some ways
Hi, If you want a fast database forget about file system caching, use Direct I/O and put your memory to better use - application level caching. The web application is written in PHP and runs off MySQL and/or Postgresql. So I don't think I can access the raw disk data directly, nor do I think it would be safe since that bypasses the DBMS's checks. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Centos/Linux Disk Caching, might be OT in some ways
Hi, 20 feilds or columns is really nothing. BUT That's dependant on the type of data being inserted. 20 was an arbitary number :) Ok so break the one table down create 2 or more, then you will have Joins clustered indexes thus slowing you down more possibly. That is greatly dependant on your select, delete, and update scripts. That was the reason the original develop gave for having these massive rows! Admittedly it is easier to read but when each row also contains text/blob fields, they tend to grow rather big. Some users have been complaining the server seems to be getting sluggish so I'm trying to plan ahead and make changes before it becomes a real problem. Possibly very correct, but Nate is very correct on how you are accessing the DB ie direct i/o also. Your fastest access come in optimized SPROCS and Triggers and TSQL. Slam enough memory into the server and load it in memory. It's an old server with all slots populated so adding memory is not an option. I thought of doing an image and porting it into a VM on a newer/faster machine. But then at the rate this client's usage growing, I foresee that as simply delaying the inevitable. If speed is what your after why are you worried about VFS? CentOS does support Raw Disk Access (no filesystem). To be honest, I don't really care about VFS since I didn't know it existed until I started looking up Linux file/disk caching :D So I assumed that was what PHP and DBMS like MySQL/Postgresql would be working through. It made sense since they wouldn't need to worry about what filesystem was really used. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Centos/Linux Disk Caching, might be OT in some ways
I'm trying to optimize some database app running on a CentOS server and wanted to confirm some things about the disk/file caching mechanism. From what I've read, Linux has a Virtual Filesystem layer that sits between the physical file system and everything else. So no matter what FS is used, applications are still addressing the VFS. Due to this, disk caching is done on an inode/block basis. I'm assuming that this is still the case in CentOS or am I badly mistaken? If that is correct, then here is my scenario and hypothesis. Assuming the server has xxx MB of free memory and the database consist of several tables more than xxx MB in size. So no table will fit entirely into memory. And assuming other processes do not interfere with the caching behaviour or available memory etc. Given the inode caching behaviour, if the DBMS only access a bunch of inodes that total less than xxx MB, is it therefore likely to be always using the cache, hence faster? My thought is that if this is the case, then I could likely speed up the application behaviour if I further split the tables into parts that are more frequently accessed, and parts that are unlikely touched. e.g. the table may currently have rows with 20 fields and total 1KB/row, but very often say only 5/20 fields are used in actual processing. Reading x rows from this table may access x inodes which would not fit into the cache/memory. However if now I break the table into two parts with those 5 fields into a smaller table, there would be a speed increase since the reading the same x rows from this table would only access 1/x inodes. Further more, these would more likely fit into the disk/memory cache for even faster access. Or would I simply be duplicating what the DBMS's index files would already be doing and therefore see no improvement? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Are SSD disks worth the cost for server usage?
Hi, - A: one is with 80 GB SSD (and 12 GB memory) http://www.ovh.co.uk/products/eg_ssd.xml - B: the other with 750 GB SATA2 (and 8 GB memory). http://www.ovh.co.uk/products/eg_best_of.xml The Intel SSD are fast but have a history of firmware problems. So I wouldn't suggest using them on a mission critical data. Personally I think asking for more RAM on the SATA server would do more for performance especially since you are going to be running several VM. Just my noobish 2 cents' worth. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Find reason for heavy load
Hi, since initially it seems like the high load may be due to I/O wait Maybe this will help you to identify the IO loading process: http://dag.wieers.com/blog/red-hat-backported-io-accounting-to-rhel5 Thanks for the suggestion, I did install dstat earlier while trying to figure things out on my own. However, I think my kernel being the older version does not support the latest feature the website was pointing out. Given that it's a live server not within physical touch, I'm a little wary of doing kernel updates that might just kill it :D I'll try other methods first and see if they help, if not, I'll probably have to bite the bullet and do it over a weekend where I get more time to repair any inadvertent damage. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Find reason for heavy load
Hi, You should also try out atop instead of just using top. The major advantage is that it gives you more information about the disk and network utilization. Thanks for the tip, I tried it and if the red lines are any indication, it seems that atop thinks my disks (md raid 1) are the problem being busy over 60~70% of the time. However that is sort of expected since most of the expected activity on the server is smtp/pop3. Unfortunately, I did not know about atop previously and don't have a baseline to compare against :( ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Find reason for heavy load
Hi, Dstat could at least tell you if your problem is CPU or I/O. This was the result of running the following command which I obtained from reading up about two weeks ago when I started trying to investigate the abnormal server behaviour. dstat -c --top-cpu -d --top-bio --top-latency usr sys idl wai hiq siq| cpu process | read writ| latency process 4 1 93 2 0 0|mysqld 0.0| 80k 82k|khelper 8 42 46 0 12 0 0|httpd 12| 648k0 |ksoftirqd/0 111 26 37 12 26 0 0|httpd1.5| 520k 11M|ksoftirqd/175 23 49 8 19 0 0|exim 1.0| 652k 16k|ksoftirqd/044 26 44 3 28 0 0|exim 1.0| 652k 1296k|ksoftirqd/044 32 41 4 23 0 0|exim 1.5| 620k 16k|ksoftirqd/050 28 52 3 16 0 0|exim 1.5| 700k0 |ksoftirqd/147 21 41 11 28 0 0|exim 1.0| 556k 11M|ksoftirqd/079 27 46 3 24 0 0|exim 1.5| 684k 16k|ksoftirqd/140 29 45 2 24 0 0|exim 1.0| 672k 944k|ksoftirqd/025 28 33 3 37 0 0|httpd 14| 852k 5992k|ksoftirqd/139 36 39 2 23 0 0|httpd5.0|1024k0 |ksoftirqd/084 Even better, run vmstat 2 10 Look at the first two columns. What column have higher numbers? If r, you're CPU-bound. If b, you're I/O bound. procs ---memory-- ---swap-- -io --system-- -cpu-- r b swpd free buff cache si sobibo in cs us sy id wa st 8 1 3092 131460 100692 83366800402110 4 1 92 2 0 9 1 3092 130708 100700 83501600 578 206 577 1420 32 50 3 15 0 7 1 3092 128324 100716 83614800 546 2866 594 1465 31 44 7 18 0 4 1 3092 126860 100724 83726800 540 256 596 1505 28 43 6 23 0 7 2 3092 125600 100740 83856400 620 234 661 1442 30 41 2 26 0 5 1 3092 124028 100756 83975200 570 2692 635 1430 24 45 6 25 0 6 0 3092 122040 100784 84096400 584 1464 682 1434 27 44 2 28 0 6 1 3092 120588 100792 84223200 602 278 624 1562 32 46 2 20 0 2 3 3092 120556 100840 84306400 440 2908 603 1299 22 35 6 37 0 3 1 3092 119832 100876 84408800 430 1104 605 1348 23 36 1 40 0 According to this, am I correct to conclude that I'm CPU bound and the system is busy doing some unknown processing? Did you check if you have a defect disk or a rebuilding array? That could be the cause. I usually run a cat /proc/mdstat whenever I log into the server to check my MD raid status. So far the array appears ok. There are no disk warning when I run dmesg. smartctl also reports no error logged and passed for both disks, although no self test was ran. Would I be safe to conclude that the disks are OK and not part of the problem? Thanks again to everybody for the suggestions and help so far. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Find reason for heavy load
Hi, Yes, these figures indicate that you are fairly close to being cpu bound. What kind of filtering are you doing? If you have any connection tracking/state related rules set, you will need to be using a fair amount of cpu. Initially, when the load start going up, I had thought the APF filtering rules were the problem since the Indian fellow is still hammering away at the server even now. However, I've since taken the risk of turning off APF and rely on static iptables rules, which adds up to less than one screenful on SSH. I also thought it might had to do with exim/spamassassin but making a few changes to reduce the number of emails that goes to spamd doesn't seem to be helping much. In fact as you can see from the stats, load has gone up even further since. I've been averaging 10+ for the whole working day. At the moment it's between 6 to 10 when it should be at 0.3 from past months of logs. This is despite the fact most of my clients should be out celebrating New Year's Eve. From weeks of logs, the Indian spammer is also a very punctual fellow who should have knock off work about 17 minutes ago. So there shouldn't be any heavy 'known' activities on the server at this point. So I'm quite stumped as to what's chewing up the CPU cycles. I am also starting to worry if the server's been compromised and is now doing something I don't want it to be. I'm probably going to shutdown the mail/httpd services after midnight when the impact is the least and see how the server reacts for a couple of minutes with everything else cut off. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Find reason for heavy load
Hi, I do not know about now but I had to unload the modules in question. Just clearing the rules was not enough to ensure that the netfilter connection tracking modules were not using any cpu at all. Thanks for pointing this out. Being a noob admin as my pseudonym states, I'd assumed stopping apf and restarting iptables was sufficient. I'll have to look up unloading module later. /me shrugs. When I was the mta admin at Outblaze Ltd. (messaging business now owned by IBM and called Lotus Live) spammers always ensured I got called. All they do is just press the big red button (aka start the script/system) and then go and play while I would have to deal with whatever was started. Based on the almost precise timing of around 9:30 to 5:30 India time, I'm inclined to think in my case it wasn't so much a spammer pressing a red button but a compromised machine in an office starting up when the user gets into office and knocks off on time at 5:30 :D I remember only one occasion when the spams were launched but neutralized very soon because they were pushing a website and I found a sample real early and so the anti spam system could just dump the spams and knock out accounts being used to send the crap. Could I ask how do I knock out the accounts sending the crap if they are not within my systems? First, try rmmod'ing the netfilter modules after you have cleared away the state related rules to make sure that you are only using static rules in netfilter...unless you have done that already.. I think I'm only using static rules because after I restart iptables, I would then do a service iptables status to check my rules were in, and that list was very short compared to when APF was active. The good news is, I think I've fixed the big problem after doing my shutdown tests and returned to the original problem. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Find reason for heavy load
I initiated services shutdown as previously planned and once the external services like exim, dovecot, httpd, crond (because it kept restarting these services), the problem child stood out like a sore thumb. There was two exim instances that didn't go away despite service exim stop. Once I killed these two PID, the load average started dropping rapidly. After a minute or so, the server went back to a happy 0.2~0.3 load and disk activity became almost negligible. I think these, orphaned? zombied?, exim instances were related to a mail loop problem I discovered earlier today where one of my client on holiday had a full mailbox and keep bouncing mails from a contact whose site was suspended. Although I terminated that loop, it seemed that exim had gotten those two instances stuck in limbo sucking up processing power and hitting the disk somewhere unknown since they weren't showing up in my exim logs. After observing a while, I brought the services back and once exim got started, my load went back to 2.x ~ 3.x. Unfortunately while I was typing this email, I realize it didn't stop there. I'm up to 4.x ~ 5.x load level by now. So the application that is the cause of the load is definitely exim, more specifically I think it's spam assassin because now that the mail logs entries are slow, I can read the spamd details and mails are taking between 3 to 8 seconds to be checked. Thanks again to everybody who had offer suggestions and advice and do have a Happy New Year :) On 1/1/10, Noob Centos Admin centos.ad...@gmail.com wrote: Hi, I do not know about now but I had to unload the modules in question. Just clearing the rules was not enough to ensure that the netfilter connection tracking modules were not using any cpu at all. Thanks for pointing this out. Being a noob admin as my pseudonym states, I'd assumed stopping apf and restarting iptables was sufficient. I'll have to look up unloading module later. /me shrugs. When I was the mta admin at Outblaze Ltd. (messaging business now owned by IBM and called Lotus Live) spammers always ensured I got called. All they do is just press the big red button (aka start the script/system) and then go and play while I would have to deal with whatever was started. Based on the almost precise timing of around 9:30 to 5:30 India time, I'm inclined to think in my case it wasn't so much a spammer pressing a red button but a compromised machine in an office starting up when the user gets into office and knocks off on time at 5:30 :D I remember only one occasion when the spams were launched but neutralized very soon because they were pushing a website and I found a sample real early and so the anti spam system could just dump the spams and knock out accounts being used to send the crap. Could I ask how do I knock out the accounts sending the crap if they are not within my systems? First, try rmmod'ing the netfilter modules after you have cleared away the state related rules to make sure that you are only using static rules in netfilter...unless you have done that already.. I think I'm only using static rules because after I restart iptables, I would then do a service iptables status to check my rules were in, and that list was very short compared to when APF was active. The good news is, I think I've fixed the big problem after doing my shutdown tests and returned to the original problem. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Find reason for heavy load
Just an concluding update to anybody who might be interested :) My apologies for blaming spamassassin in the earlier email. It was taking so long because of the real problem. Apparently the odd exim processes that was related to the mail loop problem I nipped was still the culprit. I had overlooked the fact that by the time I caught onto the mail loop issue, there were actually hundreds if not thousands of bounced and rebounced messages in the queue already. Attempting to deliver these messages queued before I terminated the mail loop was what those exim processes were trying to do. This would had been ok if not for the other problem. The user apparently went on 2 week vacation since 15th and thought it was a good idea to enlarge his mailbox before doing so. So there was this 2.5GB mailbox choked full of both valid rebounced mails, plus the queue of more rebounced mails. So every time exim attempted to add the queued mails to the user's account, the quota system rejected it. The cpu load was probably due to this never ending ping pong match between exim and the quota. Yeah, I can't help but feel this must be such a noob mistake allowing that to develop without realizing it. Now that I've purged the queue of those bounced messages and other housekeeping for that user, server load has finally gone back to the expected sub 1.0 levels so I can finally go and enjoy my holiday :) On 1/1/10, Noob Centos Admin centos.ad...@gmail.com wrote: I initiated services shutdown as previously planned and once the external services like exim, dovecot, httpd, crond (because it kept restarting these services), the problem child stood out like a sore thumb. There was two exim instances that didn't go away despite service exim stop. Once I killed these two PID, the load average started dropping rapidly. After a minute or so, the server went back to a happy 0.2~0.3 load and disk activity became almost negligible. I think these, orphaned? zombied?, exim instances were related to a mail loop problem I discovered earlier today where one of my client on holiday had a full mailbox and keep bouncing mails from a contact whose site was suspended. Although I terminated that loop, it seemed that exim had gotten those two instances stuck in limbo sucking up processing power and hitting the disk somewhere unknown since they weren't showing up in my exim logs. After observing a while, I brought the services back and once exim got started, my load went back to 2.x ~ 3.x. Unfortunately while I was typing this email, I realize it didn't stop there. I'm up to 4.x ~ 5.x load level by now. So the application that is the cause of the load is definitely exim, more specifically I think it's spam assassin because now that the mail logs entries are slow, I can read the spamd details and mails are taking between 3 to 8 seconds to be checked. Thanks again to everybody who had offer suggestions and advice and do have a Happy New Year :) On 1/1/10, Noob Centos Admin centos.ad...@gmail.com wrote: Hi, I do not know about now but I had to unload the modules in question. Just clearing the rules was not enough to ensure that the netfilter connection tracking modules were not using any cpu at all. Thanks for pointing this out. Being a noob admin as my pseudonym states, I'd assumed stopping apf and restarting iptables was sufficient. I'll have to look up unloading module later. /me shrugs. When I was the mta admin at Outblaze Ltd. (messaging business now owned by IBM and called Lotus Live) spammers always ensured I got called. All they do is just press the big red button (aka start the script/system) and then go and play while I would have to deal with whatever was started. Based on the almost precise timing of around 9:30 to 5:30 India time, I'm inclined to think in my case it wasn't so much a spammer pressing a red button but a compromised machine in an office starting up when the user gets into office and knocks off on time at 5:30 :D I remember only one occasion when the spams were launched but neutralized very soon because they were pushing a website and I found a sample real early and so the anti spam system could just dump the spams and knock out accounts being used to send the crap. Could I ask how do I knock out the accounts sending the crap if they are not within my systems? First, try rmmod'ing the netfilter modules after you have cleared away the state related rules to make sure that you are only using static rules in netfilter...unless you have done that already.. I think I'm only using static rules because after I restart iptables, I would then do a service iptables status to check my rules were in, and that list was very short compared to when APF was active. The good news is, I think I've fixed the big problem after doing my shutdown tests and returned to the original problem. ___ CentOS mailing list CentOS@centos.org
Re: [CentOS] Find reason for heavy load
Hi, last time I saw something like that, it was a bunch of chinese 'bots' hammering on my public services like ssh. another admin had turned pop3 on too, this created a very heavy load yet they didn't show up in top (bunches of pop3 and ssh processes showed up in ps -auxww, however, plug netstat -an Unfortunately the server is meant for web/email purposes so I can't turn off pop3/smtp. Naturally ps shows up a lot of httpd/mysql exim/dovecot processes but a cursory glance doesn't see any suspicious IPs. Similarly, I did a quick look at netstat -an and most of the IP are from local ISP that my clients are using. One thing that occurred to me is, does using iptables to block smtp attempt uses more system resources as opposed to letting the bot flood my smtp logs with pointless attempts? :) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Find reason for heavy load
Hi, Try blocking the IPs on the router and see if that helps. Unfortunately the server's in a DC so the router is not under our control. You can also run iostat and look at the disk usage which also generates load. I did try iostat and its iowait% did coincide with top's report, which is basically in the low 1~2%. However, iostat reports much lower %user and $system compared to top running at the same time so I'm not quite sure if I can rely on its figures. How many cores does your machine have? Load avg is calculated for a single core, so a quad core would reach 100% utilization at a load of 4, but high iowaits can generate an artificially high load avg as well (and why one sees greater than 100% utilization). It's a dual core that's why I was getting concerned since loads above 2.0 would imply the system's processing capacity was apparently maxed. However, load and percentages don't add up. For example, now I'm seeing top - 14:04:30 up 171 days, 7:14, 1 user, load average: 3.33, 3.97, 3.81 Tasks: 246 total, 2 running, 236 sleeping, 0 stopped, 8 zombie Cpu(s): 13.3%us, 16.0%sy, 0.0%ni, 67.5%id, 3.0%wa, 0.0%hi, 0.2%si, 0.0%st iostat Linux 2.6.18-128.1.16.el5xen 12/30/2009 avg-cpu: %user %nice %system %iowait %steal %idle 3.280.201.162.380.01 92.97 I really wish load would be broken down as CPU/memory/disk instead of the ambiguous load avg, and show network read/write utilization in ifconfig. Totally agreed. All the load number is doing is telling me something is using up resources somewhere but not a single clue otherwise! Confusing, frustrating and worrying at the same time :( ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] NIC traffic monitoring, recording and reporting software?
On Fri, Jul 17, 2009 at 12:07 AM, James B. Byrnebyrn...@harte-lyne.ca wrote: I have snmpd and mrtg running and reporting against my Cisco router. What I want to do is to configure snmp so that I can monitor network traffic across the host's own eth0 NIC. Is this even possible for a generic NIC running on a x86_64 or i686 host? Shouldn't be a problem since I was monitoring my server's own NIC traffic and load with MRTG before it stopped working. If I'm not mistaken, it's a matter of configuring snmp to check localhost in addition to your router's IP. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS/SNMP update breaks MRTG?
Hi, well, i note there's a few versions of rrdtool in the various repositories. the stock CentOS 5 version 9from upstream) is 1.2.30, while rpmforge has 1.3.7, also a seperate rrdutils package (I have no idea whats in it) *sigh* The stuff of nightmares, I did have 1.3.7 installed after checking. But searching on this direction finally yielded an important piece of information. Somebody posted back in 2008 on a site to IGNORE the jrrd problem because OpenNMS supposedly comes with some kind of java rrd already installed (which begs the question of why then is the jrrd step mentioned in the install guide). So I went ahead with the install process which then complained that my postgresql was the wrong version, i.e. 8.4 instead of max of 8.3, but at least this time it kindly offered a -Q option to ignore the version restrictions at my own risk. I did. Then it was on to another problem, with OpenNMS dying on startup due to port clash with DHCP. Fortunately again, this was noted as something that happens quite often on Linux systems and a quick fix was to simply comment out the dhcp configuration. After that, it was just the usual matter of opening a port in iptables for the opennms/tomcat and FINALLY something was working. I'm crossing my fingers that ignoring the jrrd, ignoring the versions and ignoring the dhcp monitor isn't going to bite me one of these days. For now, ignorence is bliss :D ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS/SNMP update breaks MRTG?
Hi, java. I don't remember seeing this problem when installing from the opennms yum repository, though. I didn't expect it either, honestly. In most cases, updates/installs does go relatively painlessly if I don't mess up following instructions/guides. In this case, I guess I just tripped up over the unessential jrrd. Are you getting any benefit from mixing all of these non-stock versions on your system? How many different repositories that contain conflicting versions of packages do you use? Normally epel doesn't overwrite stock packages and opennms I've no idea honestly, my primary role isn't server admin and I'm just winging it as I go along to support what I'm supposed to be doing with the server. The PG 8.4 was because we're developing something for our client who's on that server, so I'm standardizing on 8.4 and likely will stick with it for quite a while, rather than going with the 8.3 since there appears to be quite a few changes in 8.4, especially on warm standby features. Apart from what's needed, I usually try to avoid installing things on the public web servers we have. That is normal - typically you'd run opennms on a machine dedicated to monitoring, with perhaps thousands of targets so it wouldn't be running a lot of other services. Well, unfortunately, there's only that pair of machine in that particular location. I really needed the monitoring tool up on it because I've been noticing a higher than normal load since the weekend. My quick hack of a PHP/cat /proc/loadavg script was also alerting me consistently. After a couple of hours on opennms, it became obvious that something was hitting the server. Turns out that the client did not set the appropriate measures on their forum software and bots were having a field day hitting it to break the image recognition and finally got through to spamming. Removing it won't bother opennms. It has an assortment of application probes that it uses in addition to snmp and is intended to work automatically with large numbers of targets - when it discovers a node (or you add it), it probes the application ports to see what is running, then periodically tests again and notifies you when something that was previously running stops working. However, it is very configurable and you can add/remove whatever you want. Yup, it's pretty cool and that web interface really helps. While I am perfectly at home using a text editor, I really don't want to have to wade through and edit tons of text just to do something a few clicks should handle. Thanks again for pointing me to opennms :) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS/SNMP update breaks MRTG?
Hi, A possible work-around is to use a VPN like openvpn to give you what look like normal routes to remote locations even with private addressing. Given the amount of trouble I've had just getting monitoring to work, I don't think I'm even going to try fiddling with openVPN. Besides which, after I went to sleep happily last night, I woke up this morning to find openNMS has decided to mysteriously stop working just like MRTG previously. The service is running, opennms -v status indicates every is a-OK, but the web interface is just not responding. No log entries, not a single clue. Nothing changed, except my mood or maybe the datacenter decided port 8980 is an hacking attempt and decided to close it off. :( I'm so tired of this whole monitoring crap that I'm not even going to bother to fix it. My crude load warning script still runs fine. So until it starts complaining consistently about the load, I think I'm just going to be an irresponsible admin on top of being a noob one and just do work that I'm getting paid for. *sigh* ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS/SNMP update breaks MRTG?
Thanks guys for all the suggestions. None of it changed the situation but I'm beginning to think that it might have to do with SNMP not accepting word names in MRTG, or more specifically some kind of language encoding issue. This is because of the following reasons 1. It's been pointed that out that MRTG need to be started with the options env LANG=C because it won't work properly if LANG is UTF8 2. On some options I try in MRTG, the log shows some error about Wide characters returned from SNMP, and I see a chinese character, which obviously shouldn't be a return value. 3. Addressing SNMP variables by name does not work in MRTG, but works from command line. e.g. something like ssRawCpuLoad is fine in command line, but does not work in MRTG config file, only the dot-numeric equivalent would return some kind of data in MRTG. 4. The problem started AFTER I rebooted the system after the update, so the reboot might have possibly allowed some settings to take effect with regards to the server's encoding. Maybe Centos 5.3 went from an EN_US language default to UTF8 default? If this is indeed the case, how would I possible change the interface/shell language settings back to the English one, since I don't typically need to input non-English characters nor view them in shell? I've added a LANG='en_US' and export LANG line in /etc/profile but it doesn't seem to be doing anything. Do I need a reboot for it to work like I am guessing based on #4 above? Thanks! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS/SNMP update breaks MRTG?
Hi, I don't see any similar problem on machines upgraded to Centos5.3 that are monitored with (and running) OpenNMS, so I'd guess that since you didn't change your snmpd.conf settings it is MRTG-specific. I think it's my server, quite possibly I screwed up something during the initial setup two years ago or along the way updating it from 5.0 and so forth until it's not behaving in any recognizable manner anymore. And btw: OpenNMS might be overkill for your purpose, but you might want to take a look: http://www.opennms.org. It looks good and I decided to give it a try in hope that maybe it can be up and running faster than I can get MRTG to work again. Unfortunately, as above mentioned, my server does not behave like a CentOS server anymore. Following the steps at OpenNMS, I get to the install -dis stage where it promptly dies because it cannot find jrrd. downloaded jrrd but it refuses to ./configure because it cannot find rrd_create yum install rrdtool but there was no rrd_create searched online and the only result that was similar... was somebody having the same problem on a Solaris server -- hence making me wonder if I was logging into the wrong server. Using the instructions there however, I at least learnt how to tell configure where rrdtool was... but it still cannot find rrd_create for the ./configure process Having spent almost 5 days on this, I'm officially giving up on monitoring the server with these tools. Writing a PHP script seems a lot faster, I've already gotten a basic script running to pull load figures from exec'ing uptime and emailing warnings if the load figures stay above a certain level. Now I just have to expand the script to exec snmpget for the other metrices I need to keep track of. It's really frustrating that I have to resort to writing my own code when these things worked fine for other people. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] CentOS/SNMP update breaks MRTG?
I got itchy fingers over the weekend and decided to fix what wasn't broken and upgraded one of the older servers from Centos 5.2 to Centos 5.3. Following the recommended process of updating glibc and such before the rest, it appeared to work perfectly and rebooted without problem. However, MRTG 2.15.2 started complaining about unexpected values. I installed/updated both MRTG (2.16.2) and net-snmp to the latest available in hope of fixing it. Subsequently, MRTG stopped working altogether. I've spent the whole weekend and whole Monday morning trying to fix it and thus far have only finally managed to get garbage values showing up in MRTG again as opposed to nothing. And this required learning about SNMP and adding many additional lines to the original MRTG configuration file, none of which I had to do previously. Did anybody else have similar experiences with MRTG failing after the update and what was the simple fix? It does not make any sense that I have to jump through so much hoops to get just the default functionality back. Thus I believe there must be one small thing I'm overlooking. Thanks for any advice. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS/SNMP update breaks MRTG?
Hi, Perhaps the OIDs changed for the interfaces you are monitoring. Have you tried re-running cfgmaker to regenerate mrtg.cfg? It should pick up the correct OIDs again. Yes I did, however the default MRTG configuration appears to contain almost nothing. Consulting with others. it seems to be the norm, MRTG should pick up the standard OIDs for the basics, i.e. load and network traffic if nothing's specified. Currently, I had to manually insert target lines after figuring out the OIDs in order to get garbage data into the log files. Garbage data because while the debug log shows some numbers corresponding to output from top, MRTG is producing graphs that bear no resemblance to it. Reproducing the entire default MRTG configuration would therefore pretty much require a very long config file, as well as coming up with formulas to twist the data into something that would produce sensible graphs... which obviously don't seem like the right way to do it. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS/SNMP update breaks MRTG?
Hi, Did the update overwrite your snmpd.conf file? The 'view' on the default one may not permit access to the things mrtg needs to see. Try changing it to .1 to expose everything. It might have done so. To be honest I have no idea since I've never touched the SNMP configuration before this and simply used the default. Currently there's nothing inside the snmpd.conf except a rocommunity which is the public user. I've added lines from an online source that claims that is the default snmpd configuration and it looks like it should be allowing view all to the public user. In any case, even prior to adding these lines, I could get the relevant values off SNMP using command line with the public community user, so I don't think I was blocking any thing in SNMP --- snmpd.conf -- #existing line rocommunity public localhost #added by me com2sec publicdefault public group publicv1 public group publicv2c public group publicusm public view all included .1 accesspublic any noauthexact all none none end As expected, MRTG behaviour remains unchanged. In fact, looking at the mrtg log, with the default blank mrtg.cfg it does not even appear to be trying to poll SNMP. This is because if I added the target lines myself, MRTG would at least scream at me if SNMP does not return values or cannot find the variable name. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Getting ready for CentOS 5.4
2009/3/27 Spiro Harvey sp...@knossos.net.nz: required? How do you figure anything is *required* of volunteers? Show me your support contract. If you're worried that CentOS is late or is stopping you from fulfilling your own contractual obligations, perhaps you should stop being a tight-arse and pay for RedHat support. When you pay nothing, you have no right to expect anything. Unless they're your slaves, and I'm pretty sure that's not the case here. And as long as CentOS stays a relevant distro the pressure (not only from me) will continue to raise. This is just rude. I think you're over-reacting or maybe just misunderstanding what I believe the OP was trying to put across. Personally, even when I volunteer to do something, I do my best to do a good job of it. If something's worth doing, it's worth doing it right, paid or otherwise. So even on a personal level, there are requirements and pressure. If you are organising a charity event, would you accept a team of helpers who may or not may not show up simply because they are volunteers? Now, I don't think any of us here are demanding the CentOS to meet strict deadlines or some corporate standards of performance here. Nobody's saying the CentOS developers can't take a vacation, can't fall sick, etc. If you read our posts, most of us are wondering where did the snags occur, how we can help to ease such problems, how we can help prevent these from recurring. These are issues that must be tackled if we want the CentOS project to flourish. Like mbneto said, as things grow, pressure expectations will increase. I don't think we want to see the team get frustrated and give up due to these pressures or expectations. One of the best way to deal with expectations/pressure is good communications. It doesn't even matter if the communications is that there are delays due to personal issues. People read it, people understand and nobody bugs the team about what's going on, they will feel less pressured. Similarly, if there's a way for us as non-development-savvy folks to contribute our resources, it would also help relieve pressure on the team. All we are trying to achieve with this discussion, I believe, is to identify problem areas, see if we can help out. So as to keep the project fun for the developers to continue and not one day burn out because they feel so unsupported, unappreciated and harrassed. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Getting ready for CentOS 5.4
On Sun, Mar 29, 2009 at 3:13 AM, William L. Maltby centos4b...@triad.rr.com wrote: As a step to reducing the pressure and dissatisfaction of Are We There Yet? (When will xxx be released?), a simple publication of a projected time line will help. It should be updated as needed. It should understood that this could be another source of pressure as a release date nears and folks realize it may be missed. I'll suggest that instead of a timeline, which would be a source of pressure like you said, a weekly progress update would be just fine. Similar to what Karanbir, IIANW, has done on his twitter/blog recently. Maybe something like CentOS 5.4 Progress: Completed 2/7 Stages. Stage 3 estimated 5% completed. No progress expected for next two weeks due to XYZ convention The main thing is actually the VISIBILITY part. Putting it on CentOS frontpage would cut down a lot of the unnecessary when/where questions and leave the developers in peace :) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Installing on LVM on SW-RAID
On Sat, Mar 28, 2009 at 9:58 PM, Robert Heller hel...@deepsoft.com wrote: At Fri, 27 Mar 2009 23:48:04 -0300 CentOS mailing list centos@centos.org wrote: Yes, the root file system has to be outside of the LVM -- the initrd does not start LVM, so LVM volumes are not available for mounting at that point. As Norberto pointed out, root file system can be inside the LVM. It's /boot that has to be outside. That said, my own unpleasant and unfortunate experience suggests that everything essential to boot/recover the system should be outside lvm since rescue mode is unable to mount lvm without manual intervention after booting. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Getting ready for CentOS 5.4
On Mon, Mar 23, 2009 at 11:49 PM, Ray Van Dolson ra...@bludgeon.org wrote: There maybe needs to be a community leizon of some sort to help leverage these types of offers for help. Many of us are willing to help, but certainly don't have the necessary time cycles to do so as effectively as some of the rest of the core team. If there was a way to make jumping in and helping out with a few mundane tasks or throwing spare CPU cycles at tasks I think a lot of the weekend warriors could be more effectively leveraged. Excellent suggestion! I'm sure I'm not the only one who would love to contribute but quite obvious lack the skills to do anything really advanced. There was a somewhat similar in spirit thread on CentOS forum about PHP5.2 and somebody mentioned things are slow because none of us are willing to help test. When I saw it, the only thing came to mind was How? So if it's possible, I'd be more than happy to throw in spare CPU cycles to help compile some binaries or run automated tests etc! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Being Green, Time to make the servers sleep!
On Thu, Mar 19, 2009 at 10:22 PM, John Hinton webmas...@ew3d.com wrote: ATX, just powers down the computer, leaving the PS in a lowered power state, but apparently this can draw up to 60% of the working power needed. 60% would be a gross exaggeration, off the top of my head, an OFF ATX PSU draws less than 10W, maybe a few more in terms of VA due to inefficiency at really low power. But certainly no way near 60% unless you are referring to one of those new Atoms/Nano platform. Even then, they usually come with PSU optimized for low power operation. It would be interesting to put a wattmeter inline on the power cord to see how much current it's drawing running vs. in sleep state. I guess with an AT machine, one would have to use one of those old timers that switch on a plug something else that uses a bit of electricity, but I bet less than a power supply in sleep mode. Including conversion inefficiency, my gaming PC sucks some 180W on idle, I just sent it into standby and my wattmeter says 4W. It isn't spec'd to be accurate at less than 10W so the actual draw could range from 2W to 8W. Certainly nothing too significant, the total environmental cost including materials and energy is likely less than a new timer :) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Being Green, Time to make the servers sleep!
On Thu, Mar 19, 2009 at 7:13 PM, James Bensley jwbens...@gmail.com wrote: Shadies and Mentlemen; I am trying to be green and put our backup servers to sleep during the day and have them wake on LAN and fire back up at night for our nightly backups as sleep is a sort of low power usage mode. Make sure you are not using Seagate 7200.11 series hard disks for this unless you've somehow obtained and updated the firmwire. Frequent power cycles increases the chances that you will hit their firmware bug that apparently bricks the drive if the drive internal log is at some specific entry number before the power cycle. Was part of the recent Seagate fiasco. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Easiest way to get samba up and working for Windows users?
I was back onsite and trying it again, in vain. Copied the conf from another site's working setup and dumped directly, recreated with the same names and all. No go. So again removed and install samba again, made a blank conf file, fire up SWAT and did the most basic config. Even chmod 777 the directory. Conf file [global] workgroup = MKSC52 netbios name = MKSC52 security = SHARE log level = 2 os level = 35 [staff] comment = Staff Share path = /home/staff valid users = jackie @staff I've changed one of the Windows machine workgroup to a fresh one as above, in case the existing WIndows 2000 domain controller was somehow interfering. The pc name was also changed to the user's name. But no joy either. But at least Samba is logging something after that # [2009/03/06 17:38:31, 2] smbd/reply.c:reply_special(324) netbios connect: name1=MKS2009C52 name2=JACKIE [2009/03/06 17:38:31, 2] smbd/reply.c:reply_special(331) netbios connect: local=mks2009c52 remote=jackie, name type = 0 [2009/03/06 17:40:31, 2] smbd/process.c:timeout_processing(1363) Closing idle connection On the windows side, there was a brief pause before Windows tells me I have no permission to access the network resource. No prompt for password. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Centos 5.x SElinux issues
On Thu, Mar 5, 2009 at 3:09 AM, Chuck Campbell campb...@accelinc.com wrote: Do I need to start over with a clean install again, and how do I avoid this problem the next time I try to run updates after the install? Just my noob opinion, that if there's no practical and definitive benefit from enabling SELinux, for the time being until it is matured, the best thing to do is just set it to off. Otherwise, it just generally causes trouble and runs up tons of log as it is. I'd love to be enlightened on this though :) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Easiest way to get samba up and working for Windows users?
On Fri, Mar 6, 2009 at 3:57 AM, Scott Silva ssi...@sgvwater.com wrote: Learn to use a file editor and edit the configs yourself. That is the only way to have the best control. That's generally how I try to do things, except sometimes hand written doesn't work the way I expect it to. Then I'd like to have a GUI that does works, then learn from the conf file it creates if possible. Unfortunately, in this case they didn't work either! :D Once you have a working config, copy and modify it for the next share. That's the part of the problem I'm facing, getting a working config to be working on another machine where things might not be exactly be the same and the whole voodoo ritual starts anew. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] SELinux resource hog
Spinning off from the other thread about SELinux, I just tried to re-enable SELinux on my personal server hosting just email and forum for a small local community. Average load for this Intel Core 2 Duo box with 2GB of ram (usually with some 1GB free) was generally below 0.4 for the last 24hrs, averaging 0.23 based on MRTG. Once I did setenforce 1, load shot through the roof to fluctuate between 3 to 5. As per my past experience setroubleshootd started chewing up ram more than 600M and 500M worth of virt and res based on top. The server started crawling and php apps stopped communicating with mysql. I had to kill setroubleshootd in order to return things to normal. This again reflects my original experience with SELinux: massive resource hog and this is just a lowly loaded webserver. Naturally it seems to me that this doesn't seem like it should be the norm. What could be going on here or rather what could be wrong here? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] SELinux resource hog
On Fri, Mar 6, 2009 at 12:11 PM, Craig White craigwh...@azapple.com wrote: did you 'relabel' the entire filesystem? - that's pretty much necessary if you've been running the system without having SELinux running, at least in permissive mode. SELinux had been running in permissive. I did not disable during install because of the warning about having to relabel the entire filesystem if I wish to re-enable it subsequently. That seems like a bad idea so I've always ran it in permissive rather than enforcing due to the first experience. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Easiest way to get samba up and working for Windows users?
On Sun, Mar 1, 2009 at 2:21 AM, Les Mikesell lesmikes...@gmail.com wrote: But, if you want to do it the hard way, you probably have an Unfortunately I do want to do it the hard way. While the SME server would make things really easy, the lesson I learnt in the past with easy thing is that, once something break, I will really have no idea what is going on. It's kind of like folks who grew up knowing only GUI, they usually are helpless if the mouse doesn't work. authentication issue. With the default security setting of 'user', the windows users must authenticate before they can even see a share - and things get weird if the name they used to log into windows is not the same as the linux/samba login name. You can still map drives if you explicitly specify \\server\share, 'connect as other user' and fill in the name and password, but browsing for shares often doesn't work. I think we have a winner! This could be it as the names they use to log into their Windows machine are not their own. Most of them are inherited PC, they simply continued using the previous login since no password were set, usually. Where as the other location was a new setup with new PC setup. you aren't too concerned about security, you can change this to 'security = share' and then you can browse before authenticating, and also have the option to authenticate as different users when connecting to different shares on the same machine which you can't do in user or server modes. I'll probably do this since this is what they are used to, and expect. I don't understand the log issue, though. Are you sure smbd is running? Nmbd would be enough to activate the netbios name - maybe you have a syntax error in smb.conf and smbd did not start. Definitely running. I have tail -f on both their logs and ls the log folder every time. The startup message gets logged everytime I did a service restart on trying a different setting. Which was why I was curious why there was no log message whatsoever. The other machine would show new logs for connecting IP/machines (I think as a result of me using the split log function) even if they got rejected. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Easiest way to get samba up and working for Windows users?
I'm seriously befuddled by Samba now. I followed the good advice given and got the previous server set up nicely. I did the same thing on another one and it refuses to work. 1. useradd some users 2. gpasswd -a them to a staff group nd smbpasswd -a them 3. chmod g+s the staff directory 4. tested smbclient -L smbserver works 5. Windows user can see the Netbios name but not the share 6. Trying to access fails after timeout 7. Checked iptables/firewall not blocking 8. tail -f samba logs but nothing happens, it's like samba never see the incoming request. Note that it doesn't log anything with smbclient -L either. 9. mv the smb.conf and used a very basic one, similar to the one suggested in this thread. 10. yum remove and installed samba again just in case Still not working. I'm almost certain now that samba coder snuck in a devious randomizer that requires every single installation to only work after an random sequence of actions is taken. :( Any hints or magic words? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] xen on CentOS 4.7
On Thu, Feb 26, 2009 at 9:41 AM, Agile Aspect agile.asp...@gmail.com wrote: I'm new to Xen and I'm not familiar with the jargon. I'll second John's suggestion to go with VMWare Server. Being also pretty new and noob to all these, my first attempt at running WinXP and Win2003 Server in VMWare server was almost plain sailing. Xen on the other hand, well, let's just say I spent more time on it and that machine was re-installed with a non-Xen kernel. And that was on CentOS5 which supposedly works better with Xen. Maybe it's my noobness, but the same noob skill applied to VMWare worked fine so... Given VMWare's long history, I think Xen probably just needs more time to all the details right. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Easiest way to get samba up and working for Windows users?
On Tue, Feb 24, 2009 at 5:52 AM, Les Mikesell lesmikes...@gmail.com wrote: Is there a windows domain or AD in this picture somewhere? Not at all for all the usual Windows network migrations I've been setting up. Typically small offices with less than 20 people so they simply used workgroups without domains. If you want something nicer, run freenx on the server and the NX Thanks for the suggestion, I discovered freenx just days ago and actually had the packages installed on the new setup, just have not gotten around to using it. Then the samba shares look like: [aaa-share] comment = aaa workspace path = /path/to/aaa-share public = no valid users = @aaa writable = yes printable = no force create mode = 0775 force directory mode = 775 force group = aaa I just had an OMFG moment reading your conf. Does the valid use...@aaa means all users in the group aaa? I thought I had read it to mean exclude hence never tried it, instead I had tried things like valid users = groupAAA which obviously didn't work. If you use smb authentication against a domain controller all you have to do is create the linux users with the same login name. With winbind you might not even have to do that, but then I don't know how you control the groups. Would setting up a domain controller on the CentOS be better in the long run for only 10 to 20 people situation? I've avoided it since I'm still learning to setup Linux based servers and didn't want to bite off more than I can chew. Thanks again for all the suggestions! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Easiest way to get samba up and working for Window users?
On Tue, Feb 24, 2009 at 6:26 AM, Ned Slider n...@unixmail.co.uk wrote: It is documented on the bug tracker and forums so is a well known issue and is fixed in system-config-samba-1.2.41-3.el5. You could always grab the upstream src.rpm now and build it yourself. Thanks for the information, somehow it never struck me to check the bugtracker for this since I always half assumed it must be something I am not doing quite correctly! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Easiest way to get samba up and working for Windows users?
Everytime I have to setup samba to handle Windows users, sometime inadvertently goes wrong or doesn't work the way I expected, or takes forever to setup, especially when there are many users and various policies. So far, the easiest, sureest and quickest method appears to be install WindowsXP into VMWare and use it to handle Windows sharing. Needless to say, this strucks me as rather ironic and stupid. Thus could anybody please suggest a working frontend to samba that makes it easy to add users, set their permissions and get something that works like basic windows file sharing? So far I've tried the following which all don't quite work. 1. CentOS's samba configuration tool - added users never show up on the share configuration so the only shares it could create was for public access. 2. Webmin - thinks it added the users, but again they never show up when checked against the bundled CentOS tool and needless to say, the shares never work too 3. Samba SWAT - Very confusing tool, selecting shares sometimes end up as another share, and again, doesn't seem to work. So I just need a very basic tool that will reliably allow me to do the following - specify user name, specify password, and maybe specify a group - specify a share the user or group has read only or read/write access - force new files/folders to take on group ID so that it behaves like a normal windows share Don't need print services or anything, it's just far easier to dump a hardware print server into the network than to contemplate the additional complexity of making something like CUPS work. Just need to make sure that the Windows users can browse to the folders, get a prompt for their login and password where needed. Thanks! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Easiest way to get samba up and working for Windows users?
On Tue, Feb 24, 2009 at 3:12 AM, Craig White craigwh...@azapple.com wrote: probably not the answer you want to hear but... swat is supposed to be the tool for simple administration. I was afraid of that. By the time I gave up and completed the task manually, I was thinking maybe it might be easier to write my own script to repeat all those useradd, gpasswd -a, smbpasswd and nano smb.conf :( You are asking several questions but lumping them all under one category samba. The concept of UNIX or Linux administration is simple text files that can be manipulated with just about any editor that suits you though I would suggest that you refrain from using Windows editors because they add line endings that often cause issues. No worries about that one, I only edit conf files on my CentOS box using nano. The closest to using Windows for this is to manage my servers are SSH through putty, and writing long php scripts to be uploaded. the group idea is rather simple... let's say that you have a directory /home/samba/files and you set up a share in smb.conf called [Files], and all your users are members of the group 'users' then you would simply 'chgrp users /home/samba/files' and 'chmod g+s /home/samba/files' and that enables the 'group sticky bit' so that all files and folders in that directory are owned by group 'users' For a single common to everybody share it was easy of course. In fact, for something like that, I'll do away with bothering everybody with a login and simply make a single login everybody shares for filesharing. It's when I have 8 people who have to share aaa, then a sub group B have to share bbb, then a subgroup C have to share ccc, then a subgroup of people from B+C need to share ddd and so forth that it becomes untenable to do everything by hand and the tools at the moment just dont cut it. Now adding users is a bit more complicated in that samba users must necessarily be Linux users AND samba users so they would have to be added to both systems. This was one of the caveats I discovered over time, struggling with webmin and the likes. Something like Webmin can help here in that it can be configured to automatically create the samba user at the same time that a Linux user is created but it doesn't do that upon first install. Except of course webmin doesn't actually create the smbuser correctly. Maybe it has to do with how I use it, but maybe again like CentOS's tool, that particular functionality is actually broken. You probably want to check out something like the 'Samba By Example' publication which can be purchased at your favorite bookstore in dead tree form or can be downloaded in PDF form or read online @ http://www.samba.org/samba/docs (see left side) which will walk you through basic steps. Trust me, I did read through that. I usually don't like to bug people for help unless I really cannot find any relevant existing information and cannot figure out what else can I try. Thanks for replying in any case :) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Easiest way to get samba up and working for Windows users?
On Tue, Feb 24, 2009 at 3:23 AM, Ned Slider n...@unixmail.co.uk wrote: The samba configuration tool (system-config-samba) is finally fixed in 5.3 (due out soon) and will now correctly show added samba users :-) Honestly, I'm so glad to see this! Although I won't likely benefit from it until the next server install or re-install, at least I now know it wasn't ME! :D ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] 4 X 500 gb drives - best software raid config for a backup server?
On Sun, Feb 22, 2009 at 7:05 PM, Ian Forde i...@duckland.org wrote: RAID in software, whether RAID1 or RAID5/6, always has manual steps involved in recovery. If one is using standardized hardware, such as HP DL-x80 hardware or Dell x950 boxes, HW RAID obviates the need for a recovery procedure. It's just easier. You can still boot from a single drive, since that's what the bootloader sees. There are no vendor instructions or utilities needed for recovery. Nor is there a backup controller needed. If I have to do hardware raid, I'll definitely spec in a backup controller. Learnt this the hard way when my raid 5 controller died years after I first got it and I could no longer find a replacement. For high budget projects, having the extra raid controller as insurance isn't a big deal. But for most budget setup and cost conscious clients, soft raid obviates that hardware dependency. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] 4 X 500 gb drives - best software raid config for a backup server?
On Sat, Feb 21, 2009 at 6:04 PM, John R Pierce pie...@hogranch.com wrote: Kay Diederichs wrote: hdparm -tT tests one type of disk access, other tools test other aspects. I gave the hdparm numbers because everyone can reproduce them. For RAID0 with two disks you do see - using e.g. hdparm - the doubling of performance from two disks. If you take the time to read (or do) RAID benchmarks you'll discover that Linux software RAID1 is about as fast as a single disk (and RAID0 with two disks is about twice the speed). It's as simple as that. maybe with a simple single threaded application. if there are concurrent read requests pending it will dispatch them to both drives. I'm waiting for a 10 hour backup to be completed before doing recovery on a server (ok recovery is a nice way to put it, truth is I gave up any hope of making the screwed LVM setup work and going to wipe/reinstall after the backup), I'll probably be able to try some tests. However, I don't know enough to do this properly. So some questions: Would running two CP command to copy 2 different set of files to two different targets suffice as a basic two thread test? Is there a way to monitor actual disk transfers from command line without having to do manual timing? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] 4 X 500 gb drives - best software raid config for a backup server?
On Sat, Feb 21, 2009 at 11:42 PM, Chan Chung Hang Christopher christopher.c...@bradbury.edu.hk wrote: Would running two CP command to copy 2 different set of files to two different targets suffice as a basic two thread test? So long as you generate disk access through a file system and not hdparm. Is there a way to monitor actual disk transfers from command line without having to do manual timing? Like I said: iostat Thanks for the information. I checked iostat on one of my older servers running off CentOS 5.0 (2.6.18-53.1.21.el5xen) which was also running md raid 1 and it also confirmed that the md raid 1 was getting reads from both member devices. Although looking at it now, I think I really screwed up that installation, being my first, I had md running on top of LVM PV *slap forehead* ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] 4 X 500 gb drives - best software raid config for a backup server?
On Thu, Feb 19, 2009 at 4:22 AM, Ray Van Dolson ra...@bludgeon.org wrote: The other side of the coin (as I think you mentioned) is that many are not comfortable having LVM handle the mirroring. Are its mirroring abilities as mature or fast as md? It's certainly not documented as well at the very least. :) I remember googling for this before setting up a server some weeks ago and somebody did a benchmark. The general conclusion was stick to md for RAID 1, it has better performance. IIRC, one of the reason was while md1 will read from both disk, LVM mirror apparently only reads from the master unless it fails. Furthermore, given the nightmare of a time I'm having trying to restore a LVM PV sitting across 3 pairs of md RAID 1, I'll strongly recommend against tempting fate by using LVM for mirroring as well. Thankfully for the underlying md mirror, I can at least activate the LVM vg and offload data in rescue mode even if it won't work off a normal boot. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help setting up external drive via Firewire
On Fri, Aug 15, 2008 at 8:56 AM, Filipe Brandenburger [EMAIL PROTECTED]wrote: On Wed, Aug 13, 2008 at 18:43, Bill Campbell [EMAIL PROTECTED] wrote: My experience with Firewire has not been all that good. I figured that since Apple had been using it for years, and it is an IEEE standard, that Firewire would be more reliable than USB. I was also a bit wary as the USB disk drivers on SuSE gave warning messages saying they might not be very reliable. Same here. I just migrated our backups from Firewire 800 to USB2, because the Firewire was causing us a kernel crash per week and we were having to reboot our server because of the backup drives. This on three different machines, one running SuSE 10 and two others with CentOS 5 with the centosplus kernel. I haven't had any problem with the machine since the FW drive was plugged in and left plugged in since I have not been physically back on location. What causes this crash and how would I know it is related to FW or not, in the event but hopefully never, the system does crash? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Help setting up external drive via Firewire
I got a WD 1TB My Book with eSATA/USB/Firewire400 connectivity to backup data on a client Centos 5.1 machine. USB 2.0 works fine out of the box but is rather slow, Nautilus predicts about 1+ hour to fully backup just one day's worth of data or about 100GB. So I was hoping Firewire would be faster, which is why we got the version with all 3 interfaces to experiment with first. Following the suggestions given to another user here http://www.centos.org/modules/newbb/viewtopic.php?topic_id=15767forum=37 I updated the system's kernel to the CentoPlus [EMAIL PROTECTED] ~]$ uname -s -r Linux 2.6.18-92.1.10.el5 After a reboot, everything appears to work as expected, with the motherboard's TI Firewire controller detected [EMAIL PROTECTED] ~]# lspci | grep 1394 04:07.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link) However, now I'm stuck as the system does not appear to detect the drive when I connect the firewire cable and turn it on. I've followed some of the suggestions to check the drive status like fdisk -l but this only shows the drives already installed in the system tail -f /var/log/dmesg shows no new messages when the drive is connected/powered on So I'm at a loss as to what else I should be doing to get Firewire to work and will appreciate any help on this. Thanks! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help setting up external drive via Firewire
On Wed, Aug 13, 2008 at 4:50 PM, Laurence Alexander Hurst [EMAIL PROTECTED] wrote: 2 things jump out: 1. As has already been pointed out that is not a Centos Plus kernel. Did you reboot after installing the new kernel? (You have to reboot for a kernel update in order to be running the new kernel). Thanks Akemi Lawrence for pointing out the obvious that I was blind to! :D I overlooked the exclude line for the Centos Update repo so yum took the wrong kernel update instead. Now downloading 2.6.18-92.1.10.el5.centos.plus and hopes everything will work after this. 2. 1 hour to copy 100GB sounds like a very good speed. Obviously the eSATA interface will be the fastest as it will the the same as having it plugged directly into the SATA controller. For reference I recently copied 73GB from an internal SATA drive to an internal (software) raid0 array (made up of 2 SATA disks), and that took 1.5hours. The first day's transfer just completed and it took about 1hr 10 minutes for 101GB, from du -h, which I think is in terms of 1024. So that's like 24.6MB/s which admittedly appears to be around the maximum real world data transfer rate for USB 2.0. According to some reviews of this WD model, the Firewire was supposedly up to 1/3 faster (they had figures of 35MBps vs 44Mbps). So I am hoping to see a similar speed from the Firewire here to save some 20 minutes of waiting time, a whole week's backup would be almost 2.5 hours of savings! Going to reboot the system now with the new kernel and hopes I don't lose the NIC or something :D ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help setting up external drive via Firewire
On Wed, Aug 13, 2008 at 5:16 PM, Rainer Duffner [EMAIL PROTECTED]wrote: There's a reason someone came up with this eSATA stuff... Unfortunately the machine has no more spare SATA connectors. Installing an eSATA card and such, would probably be yet another learning experience on a machine the client is not particularly keen on seening downtime as it's collecting data 24/7 :( ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help setting up external drive via Firewire
The kernel update was successful and dmesg returns the following ieee1394: The root node is not cycle master capable; selecting a new root node and resetting... ieee1394: Error parsing configrom for node 0-00:1023 ieee1394: Node changed: 0-00:1023 - 0-01:1023 ieee1394: Node added: ID:BUS[0-00:1023] GUID[0090a9f6717e5649] ieee1394: sbp2: Driver forced to serialize I/O (serialize_io=1) ieee1394: sbp2: Try serialize_io=0 for better performance scsi6 : SBP-2 IEEE-1394 ieee1394: sbp2: Logged into SBP-2 device ieee1394: Node 0-00:1023: Max speed [S400] - Max payload [2048] Vendor: WDModel: My Book Rev: 1028 Type: Direct-Access ANSI SCSI revision: 04 SCSI device sde: 1953525168 512-byte hdwr sectors (1000205 MB) sde: Write Protect is off sde: Mode Sense: 10 00 00 00 sde: cache data unavailable sde: assuming drive cache: write through SCSI device sde: 1953525168 512-byte hdwr sectors (1000205 MB) sde: Write Protect is off sde: Mode Sense: 10 00 00 00 sde: cache data unavailable sde: assuming drive cache: write through sde:6sd 6:0:0:0: Device not ready: 6: Current: sense key: Not Ready Add. Sense: Logical unit not ready, initializing command required end_request: I/O error, dev sde, sector 0 Buffer I/O error on device sde, logical block 0 sd 6:0:0:0: Device not ready: 6: Current: sense key: Not Ready Add. Sense: Logical unit not ready, initializing command required end_request: I/O error, dev sde, sector 0 Buffer I/O error on device sde, logical block 0 sd 6:0:0:0: Device not ready: 6: Current: sense key: Not Ready Add. Sense: Logical unit not ready, initializing command required end_request: I/O error, dev sde, sector 0 Buffer I/O error on device sde, logical block 0 sd 6:0:0:0: Device not ready: 6: Current: sense key: Not Ready Add. Sense: Logical unit not ready, initializing command required end_request: I/O error, dev sde, sector 0 Buffer I/O error on device sde, logical block 0 ldm_validate_partition_table(): Disk read failed. Dev sde: unable to read RDB block 0 unable to read partition table sd 6:0:0:0: Attached scsi disk sde sd 6:0:0:0: Attached scsi generic sg4 type 0 scsi7 : SBP-2 IEEE-1394 ieee1394: sbp2: Logged into SBP-2 device ieee1394: Node 0-00:1023: Max speed [S400] - Max payload [2048] Vendor: WDModel: My Book DeviceRev: Type: Enclosure ANSI SCSI revision: 04 scsi 7:0:1:0: Attached scsi generic sg5 type 13 fdisk -l Disk /dev/sde: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sde1 * 1 121601 976760001c W95 FAT32 (LBA) The problem now is when I try to mount /dev/sde1, mount tells me that special device /dev/sde1 does not exist. Neither does trying to mount /dev/sg4 or /dev/sg5 works, mount says they are not a block device. What should I be trying next? Thanks! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help: Server security compromised?
On Thu, Aug 7, 2008 at 11:53 PM, Ray Leventhal [EMAIL PROTECTED] wrote: My US$0.02 on this.I'm a fan of apf as a front-end to iptables...but it takes some reading to understand the switches and the entire RAB (reactive address blocking) configuration options. Sadly, RAB is poorly documented, but with a bit of tinkering, I've enjoyed this feature tremendously as it cuts down on the hammering I used to get to port 22 by the bots and script kiddies. Sad to say my usual tasks keep me sufficiently occupied that I hardly have the time to study what APF actually does. It came with ELS (Easy Linux Security) scripts with directadmin, sounds like A Good Idea (tm) so I just installed it. Personally I'm aghast at the manner in which I'm running the server but practically there is only that much time I can devote to being the server admin. If you've a static IP at your workstation, add your IP address to the apf nicely formed 'allow_hosts.rules' file, usually located in /etc/apf. This is a simple IP address or IP block list (using slash notation, i.e. 192.168.1.0/24) to allow access to an IP or range of IPs. Further, the deny_hosts.rules list is the same format for hosts to always deny. I had considered this allowed only x.x.x.x ip strategy very early on since it appeared to be an obvious way to head off attacks/probes from external parties. Unfortunately, like most folks, I'm on dynamic IP. My primary role also requires me to run around very often, necessitating urgent administration from a variety of potential sub-networks from whichever ISP happens to be providing access at the location. So I figured it would be quite impractical to attempt to limit access to only certain IP addresses. Although thinking about it now, extending the concept from a previous suggestion, I suppose it is theoretically possible to write a privileged script accessible from one of the server hosted domains to activate an allow-host rule addition to the firewall and a cronjob that routinely activates another script to removed added hosts after 1 hour or something. So anytime access is needed, I would hit the website to activate the script to open up SSH access to the IP I am using at the moment and then SSH in. But of course, easier said than done since I barely know shell scripting and allowing exec in PHP had always been met with a big frown personally. :D ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help: Server security compromised?
On Wed, Aug 6, 2008 at 3:06 PM, Bent Terp [EMAIL PROTECTED] wrote: On Wed, Aug 6, 2008 at 8:29 AM, Noob Centos Admin [EMAIL PROTECTED] wrote: Since I followed some of the rules about SSH and used a non-standard port for SSH and disable SSHD listening on the default port 22, I've no way back IMNSHO that's not particularly effective - much better to set up SSH keys and either set 'PermitRootLogin without-password' in /etc/ssh/sshd_config; or set 'PermitRootLogin no', and then su or sudo from your regular user - I know the latter IS more secure, but it's also more annoying to work with I did that too, no root login and everytime I have to su from normal user. It is a pain to work with especially with having to use full pathnames for commands instead of say just doing a service httpd restart. But I figured it was better safe than sorry and as well as I can do since I could not figure out how to properly create a self-sign SSL cert. Remember to reinstall from scratch if your server has been compromised - there are thousands of dark dusty corners for the bugs to hide, once they're inside, so don't expect to be able to flush them out. Well, the thing is I'm not sure if it's compromised since now it became obvious that the iptables is just being reset by the apf settings.. which is at the moment a good thing since on reboot, apf re-added the lines to disable the firewall every 5 minutes so I'm able to get back into the server. Now I just have to figure out where exactly can I add the block for the offending VNSL IP address and have it work without choking up. However, I decided to try whatever it is on Saturday so clients won't be hopping mad why everything's dead. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help: Server security compromised?
Hi, On Wed, Aug 6, 2008 at 3:07 PM, Robert - elists [EMAIL PROTECTED]wrote: If server is not compromised, just edit the smtp configs to deny acceptance from that ip block The EXIM configurations are even more nightmarish than iptables, which at least made some sort of sense. I've been plugging the ip address into the various bad_sender bad_host etc files in the exim configuration directory but it's still not ignoring it. The EXIM smpt/MTA will still accept the connection, then check and realize hey something's not quite right, then issue a reject before the VNSL machine terminates the connection. So the server's still wasting resources handling tens of thousands of such transaction and chewing up log space at the same time. Hence I have to resort to just blocking from iptables. Of course, it could very well be my own admitted incompetence that I'm doing something wrong here so Exim is not working the way I expect. I'm very very wary about messing any deeper with the mail settings because a server that's obviously dead to the world is much easier to notice than client emails mysteriously disappearing for days due to bad config before they realize it. Why doesn't the server have an ILO port or something to that effect? Well, my boss's a cheapskate and his clients are cheapskate so a couple of years back I was assigned the server administration job on top of my regular day role to setup the server with OTS parts. Hence the half baked setup based on a tight budget and whatever information I can glean from the internet and the good folks on forums and mailing lists. So for the ILO? Well, only today did the term enter my mind. Although I did vaguely remember suggestions for a remote reboot button but it was beyond my know how to setup. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help: Server security compromised?
Hi, If you use su only, you assume root privileges without the root environment. Rather do su - which gives you the full root environment, including path. The same holds for other users, i..e su - joe switches the user to the user joe with full environment. Thanks a million for that! Going to save me a ton of time from issuing whereis command to find commands when I need to follow instructions off a website! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help: Server security compromised?
Thanks Steward and Robert for those suggestions, they make plenty of sense!. About the two SSH terminal, if I activate a wrong firewall change that blocks the SSH port, would it not also terminate the existing terminals since new packets going in would be rejected, or does it not affect already established TCP connections? Probably also going to make a script to shutdown the firewall as well as one for reboot. Since so far all 3 times my noobness involves firewalling myself out, although in a slightly different way each time! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help: Server security compromised?
On Thu, Aug 7, 2008 at 1:54 AM, Sorin Srbu [EMAIL PROTECTED] wrote: Seen this? http://www.askbjoernhansen.com/2007/09/18/safely_change_firewall_rules_remotely.html Unfortunately, only after you pointed it out :( But thankfully whoever wrote APF apparently knows this, hence it does insert an automatic reset of the firewall after 5 minutes. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Help: Server security compromised?
Hi, Need some help about this as it's gotten me really concerned. I'm probably reading too much into this but for about two weeks now my daily log has increased by almost 10 times. After running through a couple of days of logs with a script, it seems that I'm getting flooded on SMTP from this IP 219.64.114.52 which belongs to VSNL and appears to be statically assigned IP (219.64.114.52.chn.bb-static.vsnl.net). This IP address is apparently listed in the spamhous.org Policy Block List, eXploit Block List and Composite Block List, which basically indicates it's either an open proxy or a hijacked system. I'm not sure what it's trying to do, but for exactly 10 hours a day which correspond to India 9:30am or so until 7pm or so, I will get massive amounts of SMTP connections from this host. It will attempt to masquerade as domains on my server while trying to send to non-existent accounts on these domains. 2008-08-06 13:32:58 H=(.com) [219.64.114.52] F=[EMAIL PROTECTED] rejected RCPT [EMAIL PROTECTED]: 2008-08-06 13:32:58 H=(.com) [219.64.114.52] incomplete transaction (connection lost) from [EMAIL PROTECTED] 2008-08-06 13:32:58 unexpected disconnection while reading SMTP command from (.com) [219.64.114.52] 2008-08-06 13:32:58 H=(.com) [219.64.114.52] F=[EMAIL PROTECTED] rejected RCPT [EMAIL PROTECTED]: 2008-08-06 13:32:58 H=(.com) [219.64.114.52] incomplete transaction (connection lost) from [EMAIL PROTECTED] 2008-08-06 13:32:58 unexpected disconnection while reading SMTP command from (.com) [219.64.114.52] At this point, I thought it was just a case of a dedicated spamming, until I decided I had enough of multi-megabytes daily logs flooding my mailbox, plus the fact it was probably contributing to an increase server load in the past weeks as the mail daemon had to handle the connections. So I thought I could just block the IP using iptables. I had a bad experience locking myself out by accident after editing the iptables file so for this time I decided to test from command line first using instructions from the Internet like this /sbin/iptables -A RH-Firewall-1-INPUT -s 219.64.114.52 -j DROP and I got an error that chain/command /sbin/iptables -L produces blank output [EMAIL PROTECTED] confused]# /sbin/iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination which was of course a shock to me, since that seems to say that my server firewall is basically non-existent. I did a /sbin/service iptables restart and iptables -L produced the expected output showing all the rules on file. I could then add the new rule from command line without any messages. Minutes later, my tail -f on the exim log started spewing the smtp messages AGAIN. iptables -L again shows NO RULES Everytime I restart, iptables, for a short while, the rules are there. But minutes later, it's wiped. So I'm very concerned that the server had been compromised and something is wiping my iptables. Or am I just badly mistaken about the way iptables -L is supposed to work? If not, what should I do next to find and eliminate this problem? Thanks in advance for any advice! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Help in troubleshoot cause of high kernel activity
On Sat, Mar 29, 2008 at 6:37 PM, Johnny Hughes [EMAIL PROTECTED] wrote: Well .. top says you have 4 processes running ... if that is consistent (4 processes always in a run state) then you should be able to determine the running processes with the command: ps -ef r (I think) I would think one of always running processes is the one that is taking up CPU time. Also while in top, Shift-H might show some hidden threads in the output. Thanks for the advise although I never got a chance to use it. For some inexplicable Murphy-like reason, the server load went back to normal levels shortly after I sent off the email to the list. The only possible explanation I could think of was that I killed the setroubleshootd process because it froze up after I tried to fiddle with the SELinux settings. There was some error in the log about unable to connect to the audit socket. After observing the back to normal loads for a few hours to confirm it wasn't a momentarily drop, I restarted the setroubleshootd process and yet the load remain normal. So my current uneducated guess is that the barrage of undeliverable email messages on the very first day caused SELinux to choke on a system/kernel level until the reporting daemon was killed to whatever was getting tied up to move on? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Help in troubleshoot cause of high kernel activity
Hi, I had been experiencing a problem on our dedicated server running Centos 5, and unable to successfully track down the problem. Since about 6 days ago, I noticed a spike in load/CPU utilization which went from a typical 0.2x-0.3x to 3.x. At the same time, average traffic also went up and so did the log usage. Prior to this, the server was working fine and there had been no changes to the configuration. Initially, I narrowed it down to the mail system. Exim was generating significantly more log data than usual. This was eventually narrowed down to apparently our server and another server playing ping pong between two users who coincidentally were on vacation and had both their mailboxes filled. Thus it caused an endless loop of Message Undelivered and Auto-reply. Once this was identified and cleared up, I had expected things to go back to normal. However, load/traffic remained high. Looking at top output, I noted that %sys was as high and often much higher than %user. However, individual process %CPU just didn't add up to the total top was reporting. Top reports 160~170 sleeping tasks and only 4 active most of the time, which was largely exim then httpd/mysql/php. top Snapshot == top - 17:25:03 up 7 days, 19:16, 1 user, load average: 2.03, 2.84, 3.04 Tasks: 168 total, 4 running, 164 sleeping, 0 stopped, 0 zombie Cpu(s): 26.5%us, 50.3%sy, 0.0%ni, 16.6%id, 6.1%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 1915208k total, 1880256k used,34952k free, 142100k buffers Swap: 16777208k total,66140k used, 16711068k free, 1276564k cached iostat Snapshot avg-cpu: %user %nice %system %iowait %steal %idle 18.960.00 25.57 5.16 0.01 50.30 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 54.1963.31 2460.80 42689802 1659234904 sdb 55.1276.41 2460.80 51521720 1659234904 md1 315.95 139.72 2442.00 94207644 1646554216 md0 0.01 0.00 0.02 1422 14736 dm-0 39.1365.85 292.50 44399402 197219496 dm-1267.1836.18 2110.08 24398010 1422756072 dm-2 9.6437.6839.42 25408576 26578648 fd0 0.00 0.00 0.00 16 0 sr0 0.00 0.00 0.00136 0 Searching around for ways to interpret the output, I tried sar/iostat and essentially, the information off the net indicates there wasn't a disk problem, %io was relatively low and mdadm shows the RAID 1 disks working perfectly fine. Since %sys is consistently highest, it appears that the kernel was doing something outside of norm. The problem is I have no idea what else to do to determine what something is. I've looked at netstat and there doesn't appear to be excessive connections, logwatch summary also does not appear to give any clue as there are no records of unusual failed log in attempts. Please advise what else can I look into or check. Thanks in advance! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
