Re: weirdest problem ticket opened today.
On 21/04/05, Ez-Aton [EMAIL PROTECTED] wrote: It's not a contest I want to win in. It happened once, and the backups were one week old. Yep. Bad luck. Yeah, bad luck. And I quote from your own words: I played with a spare disk (small one) I had, and a backup script, using tar... It happened that I was very drunk that night, and it seemed like the best idea to play with the script Your honor, this is a clear and cut case of DUI - Debugging Under the Influence. The accused was trying to hide his actions, as is plainly clear from his words again: ...a user starts talkint to me, saying he can't login to his home dir... I've explained there are some maintanance works on the server, and that it will be ok by morning. He claimed he can't read his mail using pine (wonder why...), and I've used the same explanation... After this overwhelming evidence, the prosecution demands that the accused will receive the maximum penalty set for DUI in the law: Running Windows 3.10 for 3 years. -- Arik To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: weirdest problem ticket opened today.
On Saturday April 23 2005 10:00, [EMAIL PROTECTED] wrote: ROTFL! Except that Running Windows 3.10 for 3 years might be considered cruel and unusual :-) How about 1.0 then? Given some time, I can provide a 5.25 diskette torture^Winstall set...for the pure sake of justice, of course. Heck, I will even dig out that drive out of my scrap pile and revive the old 486 box I typically rest my bored feet on with the 386 motherboard off the wall just to watch someone use it! *evil grin* -- Off to practice my evil laughter, Vasiliev Michael The day after tomorrow is the third day of the rest of your life. = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: weirdest problem ticket opened today.
It's not a contest I want to win in. It happened once, and the backups were one week old. Yep. Bad luck. Ez. Muli Ben-Yehuda wrote: On Wed, Apr 20, 2005 at 09:41:06AM +0300, Arik Baratz wrote: AAARG! NOW I know what happened to my f-ing files on that server! Your backups were NOT up to date enough!!! *ROTFL* EZ, you win :-) Cheers, Muli
Re: weirdest problem ticket opened today.
On 19/04/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: [snip] Few years ago, while (and still) administrating the Israeli Radio Amature Commette (IARC) server, which is a Linux machine, and back then it was old [snip] rm -Rf home AAARG! NOW I know what happened to my f-ing files on that server! Your backups were NOT up to date enough!!! -- Arik 4z5rx To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: weirdest problem ticket opened today.
On Wed, Apr 20, 2005 at 09:41:06AM +0300, Arik Baratz wrote: AAARG! NOW I know what happened to my f-ing files on that server! Your backups were NOT up to date enough!!! *ROTFL* EZ, you win :-) Cheers, Muli signature.asc Description: Digital signature
Re: weirdest problem ticket opened today.
About a year ago, i wanted to build music server in my house, and the hardware i had was p100 that had some rust on it ;) Ok, started checking it, and saw that the floppy controller don't work, thus i can't boot from cdrom either because it was a very old bios. Went to the manufacture web site, uploaded the firmware and installed it, and... black screen ;) Opened my p100 router and did hot swapping of the bios chip, then burned the old firmware back to the eeprom. Now how can i boot? Called friends from work and drove there to take pxe boot enabled network card. installed tftp server on the router and booted. hehe. the machine was installed and running ;) I returned the network card to work. 2 month later, i'm brousing the hard drive and wanted to clear some space. Well, its 200GB so i left some partitions to try more distros and stuff, so i thoght what the heck, i'll mount one and check if its empty. Mounted and i see my / . I remember that i allways save partition that i installed and install again on another partition, so i guessed that its past install. Quickly did 'rm -Rf /mnt/my_old_partition', and hey Something strange happening ;) no ls, no cd.. Scratching my head if finally got it ;) kfir pgpYx8XFHpUCA.pgp Description: PGP signature
Re: weirdest problem ticket opened today.
I'll call. (raising on this crowd is to risky) working on my first workplace (TAU) we had a similar issue... a sysadmin tryed to format(1) a disk under SunOs 4.1.3 problem format displays a menu 1 /dev/sda0 2 /dev/sda1 3 /dev/sda2 nuff said on my second work place I was a sys admin in eci telecom I was working on a priority project on Solaris 2.5 when suddenly i had to setup an AIX system real quick. (new problem popped up on a customer site). we had 2 AIX systems that had a tape (DDS-2) drive that represented customer machines. we had customer machines backups on tapes (the DDS-2 tapes) that we restored whenever a customer problem occured. since the restore of a complete system takes time I decided to ditch the sun for a second, dig a KVM, connect the AIX to a KVM (they made me use windows for mail so I was connecting machines with PS/2 slots to a KVM with my windows) and get back to the sun ASAP. now the Sun project was very important (but so was the AIX project) so I was very very stressed I stormed through our lab equipment eventually finding a KVM that was ditched to the side (who would put it there? nobody can find it!) and immdietly connected the first AIX system. it refused to boot. no fear! there is another AIX! we will deal with this one later! so i connected the other AIX... this one also doesn't boot hmmm I connect the first one again, check electricity nothing works... I eventually ask the windows system adminstrator to take a look the second he arrives he said where did you take this KVM from I specificall put it aside because it kills mother boards! suddenly things were not urgent. PS later I found out that my windows PC (also connected to the KVM also didn't boot... --- [EMAIL PROTECTED] wrote: I see as raised, and add some more. Few years ago, while (and still) administrating the Israeli Radio Amature Commette (IARC) server, which is a Linux machine, and back then it was old RH5.1 (very old at that time), I played with a spare disk (small one) I had, and a backup script, using tar. It happened that I was very drunk that night, and it seemed like the best idea to play with the script, and try to handle everything in the / partition, where it was anyhow well divided between many partitions. I did the following: cd / tar some commands | (cd /mnt/backup tar xvf -) Oh no, I thought to myself, now it's backing up /home, which is on another partition. Lets clean the space and try again, correctly this time ls /mnt/backup yep, home is there. Not good. Need to remove it, and try again rm -Rf home Had I ran `pwd` I'de seen I'm located in / ... Shit! ^C /me now very sober At that stage I started copying whatever realy resides on /mnt/backup from the home subdir. With luck, I had a week old backup of the home dirs, at home, connected through ADSL, and got to start uploading a 4GB file to the server, to open and restore. (afterwards I've decided to untar it on my computer, and upload only the missing parts). During this upload and restore time, a user starts talkint to me, saying he can't login to his home dir... I've explained there are some maintanance works on the server, and that it will be ok by morning. He claimed he can't read his mail using pine (wonder why...), and I've used the same explanation... That's another way to get real sober, real fast... Ez. On Mon, Apr 18, 2005 at 09:22:22PM +0300, [EMAIL PROTECTED] wrote: I told him I'de sell tickets for his show, if he ever did it again. I'll see this and raise you one. Some time ago I was working on a custom embedded PPC board (running Linux, naturally). After I finished hardening the system against intrusion, I disabled root access and logged off. There was a super secret sneaky method for enabling root access remotely, which I proceeded to try. The method was buggy and root access was not enabled. No worries, I still had serial console access. Which required root access. I also had a couple of open root logins on the board - until my X died. Oh shit. I then proceeded to try and break into the system I just finished hardening to (re)gain root priviledges. A few hours later, I gave up. Cooked up a RiscWatch, sacrificied some blood to the bare hardware gods, hooked it up, and proceeded to reflash a new kernel that should drop me into /bin/sh. Driving the RW was done from a machine several firewalls (and continents over), with the latency you would expect. It was done via a set of shell scripts that usually worked, except when they didn't and completely fried the board. Naturally, they were sensitive to timing. Amazingly, this time they worked. I rebooted the board, dropped into /bin/sh, was happy to discover that everything still worked, restored the old kernel and rebooted. As it was booting, I realized that I haven't enabled root access before rebooting... Cue several
Re: weirdest problem ticket opened today.
Speaking of motherboards-a-la-grill, here's a tip: Never switch the keyboard and the mouse connectors on an old Alpha machine. It burns motherboard when you turn it on. Another thing about those creatures (I'm talking about something as old as OSF/3): It appears that a bug in a specific version of OSF causes the system to hang after a long uptime. It goes something like this: As all kernels, OSF's doesn't like to be left without memory, so it kills a process once it find out it can't allocate pages, until it re-gains enough free memory. Problem is, OSF picks processes randomly (or so it seems). As a result, if you had a leaking process running somewhere on your system, you could be certain that eventually, OSF will decide to kill something important on its killing spree (like init). That version of OSF ships with a leaking idle process... On 4/19/05, Nzer Zaidenberg [EMAIL PROTECTED] wrote: I'll call. (raising on this crowd is to risky) working on my first workplace (TAU) we had a similar issue... a sysadmin tryed to format(1) a disk under SunOs 4.1.3 problem format displays a menu 1 /dev/sda0 2 /dev/sda1 3 /dev/sda2 nuff said on my second work place I was a sys admin in eci telecom I was working on a priority project on Solaris 2.5 when suddenly i had to setup an AIX system real quick. (new problem popped up on a customer site). we had 2 AIX systems that had a tape (DDS-2) drive that represented customer machines. we had customer machines backups on tapes (the DDS-2 tapes) that we restored whenever a customer problem occured. since the restore of a complete system takes time I decided to ditch the sun for a second, dig a KVM, connect the AIX to a KVM (they made me use windows for mail so I was connecting machines with PS/2 slots to a KVM with my windows) and get back to the sun ASAP. now the Sun project was very important (but so was the AIX project) so I was very very stressed I stormed through our lab equipment eventually finding a KVM that was ditched to the side (who would put it there? nobody can find it!) and immdietly connected the first AIX system. it refused to boot. no fear! there is another AIX! we will deal with this one later! so i connected the other AIX... this one also doesn't boot hmmm I connect the first one again, check electricity nothing works... I eventually ask the windows system adminstrator to take a look the second he arrives he said where did you take this KVM from I specificall put it aside because it kills mother boards! suddenly things were not urgent. PS later I found out that my windows PC (also connected to the KVM also didn't boot... --- [EMAIL PROTECTED] wrote: I see as raised, and add some more. Few years ago, while (and still) administrating the Israeli Radio Amature Commette (IARC) server, which is a Linux machine, and back then it was old RH5.1 (very old at that time), I played with a spare disk (small one) I had, and a backup script, using tar. It happened that I was very drunk that night, and it seemed like the best idea to play with the script, and try to handle everything in the / partition, where it was anyhow well divided between many partitions. I did the following: cd / tar some commands | (cd /mnt/backup tar xvf -) Oh no, I thought to myself, now it's backing up /home, which is on another partition. Lets clean the space and try again, correctly this time ls /mnt/backup yep, home is there. Not good. Need to remove it, and try again rm -Rf home Had I ran `pwd` I'de seen I'm located in / ... Shit! ^C /me now very sober At that stage I started copying whatever realy resides on /mnt/backup from the home subdir. With luck, I had a week old backup of the home dirs, at home, connected through ADSL, and got to start uploading a 4GB file to the server, to open and restore. (afterwards I've decided to untar it on my computer, and upload only the missing parts). During this upload and restore time, a user starts talkint to me, saying he can't login to his home dir... I've explained there are some maintanance works on the server, and that it will be ok by morning. He claimed he can't read his mail using pine (wonder why...), and I've used the same explanation... That's another way to get real sober, real fast... Ez. On Mon, Apr 18, 2005 at 09:22:22PM +0300, [EMAIL PROTECTED] wrote: I told him I'de sell tickets for his show, if he ever did it again. I'll see this and raise you one. Some time ago I was working on a custom embedded PPC board (running Linux, naturally). After I finished hardening the system against intrusion, I disabled root access and logged off. There was a super secret sneaky method for enabling root access remotely, which I proceeded to try. The method was buggy and root access was not enabled. No worries, I
Re: weirdest problem ticket opened today.
- Solaris SunCluster 3.1 uses some special SCSI-3 command for quorum keep-alive. Connecting two nodes in Installation Mode (Default, right after install), everything works well. Connecting it to an FC storage (namely IBM FastT200) single port, you can define the quorum, but it's always offline. It brings to the state where high-availability might be compromised: Your max votes are 3, your current votes, at all times, are 2, and if you reboot one of the nodes (to remind you, high availability it is), the 2nd node crushes. I specifically asked the developers _not_ to reset any of the cluster's nodes. Solved when I managed to put my hands on a Store Edge A5000 (old and junky), which supported exactly the required SCSI-3 command. / - IBM AIX High Availability cluster (aka HACMP) connected to the same IBM FastT200 storage (supports it, by the books) has a weird syndrom. Once a while, during resource switch, the volume group gets corrupted. Latest FastT200 firmware deals with sometimes volume corruption occures under AIX. Didn't have the chance to upgrade yet. - IBM HACMP (just the same one) has some small issue - if the cluster service on a machine fails, the computer gets turned off, so no harm could be caused. The cluster failes as a result of a switch to the machine in question. The machine turns off. The resource group moves to the other node. The cluster service fails. The 2nd node turns off. So long for High Availability... Ez. Speaking of motherboards-a-la-grill, here's a tip: Never switch the keyboard and the mouse connectors on an old Alpha machine. It burns motherboard when you turn it on. Another thing about those creatures (I'm talking about something as old as OSF/3): It appears that a bug in a specific version of OSF causes the system to hang after a long uptime. It goes something like this: As all kernels, OSF's doesn't like to be left without memory, so it kills a process once it find out it can't allocate pages, until it re-gains enough free memory. Problem is, OSF picks processes randomly (or so it seems). As a result, if you had a leaking process running somewhere on your system, you could be certain that eventually, OSF will decide to kill something important on its killing spree (like init). That version of OSF ships with a leaking idle process... On 4/19/05, Nzer Zaidenberg [EMAIL PROTECTED] wrote: I'll call. (raising on this crowd is to risky) working on my first workplace (TAU) we had a similar issue... a sysadmin tryed to format(1) a disk under SunOs 4.1.3 problem format displays a menu 1 /dev/sda0 2 /dev/sda1 3 /dev/sda2 nuff said on my second work place I was a sys admin in eci telecom I was working on a priority project on Solaris 2.5 when suddenly i had to setup an AIX system real quick. (new problem popped up on a customer site). we had 2 AIX systems that had a tape (DDS-2) drive that represented customer machines. we had customer machines backups on tapes (the DDS-2 tapes) that we restored whenever a customer problem occured. since the restore of a complete system takes time I decided to ditch the sun for a second, dig a KVM, connect the AIX to a KVM (they made me use windows for mail so I was connecting machines with PS/2 slots to a KVM with my windows) and get back to the sun ASAP. now the Sun project was very important (but so was the AIX project) so I was very very stressed I stormed through our lab equipment eventually finding a KVM that was ditched to the side (who would put it there? nobody can find it!) and immdietly connected the first AIX system. it refused to boot. no fear! there is another AIX! we will deal with this one later! so i connected the other AIX... this one also doesn't boot hmmm I connect the first one again, check electricity nothing works... I eventually ask the windows system adminstrator to take a look the second he arrives he said where did you take this KVM from I specificall put it aside because it kills mother boards! suddenly things were not urgent. PS later I found out that my windows PC (also connected to the KVM also didn't boot... --- [EMAIL PROTECTED] wrote: I see as raised, and add some more. Few years ago, while (and still) administrating the Israeli Radio Amature Commette (IARC) server, which is a Linux machine, and back then it was old RH5.1 (very old at that time), I played with a spare disk (small one) I had, and a backup script, using tar. It happened that I was very drunk that night, and it seemed like the best idea to play with the script, and try to handle everything in the / partition, where it was anyhow well divided between many partitions. I did the following: cd / tar some commands | (cd /mnt/backup tar xvf -) Oh no, I thought to myself, now it's backing up /home, which is on another partition. Lets clean the space and try again, correctly this time ls /mnt/backup yep, home is
weirdest problem ticket opened today.
Sorry for posting an off-topic here, but I hope you enojy it and I secretly hope you have odd stories to add for the sake of entertainment... I just got a call, a guy at the Technion called to open a ticket. on two machines, one running RHEL3 and the other Fedora, every time he hits backspace, while in EMACS, in the mail mode, the machine gets stuck and starts beeping. A service technician is on the way there to help me remote-debug this, but until we discover it's something as rediculous as a faulty KVM (they claim there isn't one, we'll see). Got a funny incident to overshaddow this? :-) -- Crazy/Beutiful Ira Abramov http://ira.abramov.org/email/ = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: weirdest problem ticket opened today.
On Mon, 18 Apr 2005, Ira Abramov wrote: Sorry for posting an off-topic here, but I hope you enojy it and I secretly hope you have odd stories to add for the sake of entertainment... I just got a call, a guy at the Technion called to open a ticket. on two machines, one running RHEL3 and the other Fedora, every time he hits backspace, while in EMACS, in the mail mode, the machine gets stuck and starts beeping. A service technician is on the way there to help me remote-debug this, but until we discover it's something as rediculous as a faulty KVM (they claim there isn't one, we'll see). Got a funny incident to overshaddow this? :-) Sure! One bright day somebody opened ticket complaining that ADANET socket in his room is faulty and it prevents him from doing his research. I was new at the place then and thought that the guy, probably knew what he wanted and there's kind of special network in his room. Only after i came to check i saw that ADANET was printed on his network socket. Kind Regards, Greg -- Crazy/Beutiful Ira Abramov http://ira.abramov.org/email/ = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED] = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: weirdest problem ticket opened today.
NOt long ago, one of our clients, running Windows 2000 Server, in a very complicated testing environment (testing our product), installed the product, and since then, his server was unable to boot. During the debug procedure, he was asked to send us his registry, and so he did. He exported his registry to a reg file, and sent it to us, only for one of our tech supporters/QA to double-click on it, and say yes. Since installing a faulty server's registy seemed like a bad idea, he immediately rebooted, and since then, his computer, just like the client's one, was unable to boot. I told him I'de sell tickets for his show, if he ever did it again. Ez. On Mon, 18 Apr 2005, Ira Abramov wrote: Sorry for posting an off-topic here, but I hope you enojy it and I secretly hope you have odd stories to add for the sake of entertainment... I just got a call, a guy at the Technion called to open a ticket. on two machines, one running RHEL3 and the other Fedora, every time he hits backspace, while in EMACS, in the mail mode, the machine gets stuck and starts beeping. A service technician is on the way there to help me remote-debug this, but until we discover it's something as rediculous as a faulty KVM (they claim there isn't one, we'll see). Got a funny incident to overshaddow this? :-) Sure! One bright day somebody opened ticket complaining that ADANET socket in his room is faulty and it prevents him from doing his research. I was new at the place then and thought that the guy, probably knew what he wanted and there's kind of special network in his room. Only after i came to check i saw that ADANET was printed on his network socket. Kind Regards, Greg -- Crazy/Beutiful Ira Abramov http://ira.abramov.org/email/ = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED] = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED] = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: weirdest problem ticket opened today.
On Mon, Apr 18, 2005 at 09:22:22PM +0300, [EMAIL PROTECTED] wrote: I told him I'de sell tickets for his show, if he ever did it again. I'll see this and raise you one. Some time ago I was working on a custom embedded PPC board (running Linux, naturally). After I finished hardening the system against intrusion, I disabled root access and logged off. There was a super secret sneaky method for enabling root access remotely, which I proceeded to try. The method was buggy and root access was not enabled. No worries, I still had serial console access. Which required root access. I also had a couple of open root logins on the board - until my X died. Oh shit. I then proceeded to try and break into the system I just finished hardening to (re)gain root priviledges. A few hours later, I gave up. Cooked up a RiscWatch, sacrificied some blood to the bare hardware gods, hooked it up, and proceeded to reflash a new kernel that should drop me into /bin/sh. Driving the RW was done from a machine several firewalls (and continents over), with the latency you would expect. It was done via a set of shell scripts that usually worked, except when they didn't and completely fried the board. Naturally, they were sensitive to timing. Amazingly, this time they worked. I rebooted the board, dropped into /bin/sh, was happy to discover that everything still worked, restored the old kernel and rebooted. As it was booting, I realized that I haven't enabled root access before rebooting... Cue several more hours of alternately massaging RiscWatch and banging head against wall. Eventually, root access is restored and I go home. Some mistakes you only make once. Cheers, Muli -- Muli Ben-Yehuda http://www.mulix.org | http://mulix.livejournal.com/ signature.asc Description: Digital signature
Re: weirdest problem ticket opened today.
I see as raised, and add some more. Few years ago, while (and still) administrating the Israeli Radio Amature Commette (IARC) server, which is a Linux machine, and back then it was old RH5.1 (very old at that time), I played with a spare disk (small one) I had, and a backup script, using tar. It happened that I was very drunk that night, and it seemed like the best idea to play with the script, and try to handle everything in the / partition, where it was anyhow well divided between many partitions. I did the following: cd / tar some commands | (cd /mnt/backup tar xvf -) Oh no, I thought to myself, now it's backing up /home, which is on another partition. Lets clean the space and try again, correctly this time ls /mnt/backup yep, home is there. Not good. Need to remove it, and try again rm -Rf home Had I ran `pwd` I'de seen I'm located in / ... Shit! ^C /me now very sober At that stage I started copying whatever realy resides on /mnt/backup from the home subdir. With luck, I had a week old backup of the home dirs, at home, connected through ADSL, and got to start uploading a 4GB file to the server, to open and restore. (afterwards I've decided to untar it on my computer, and upload only the missing parts). During this upload and restore time, a user starts talkint to me, saying he can't login to his home dir... I've explained there are some maintanance works on the server, and that it will be ok by morning. He claimed he can't read his mail using pine (wonder why...), and I've used the same explanation... That's another way to get real sober, real fast... Ez. On Mon, Apr 18, 2005 at 09:22:22PM +0300, [EMAIL PROTECTED] wrote: I told him I'de sell tickets for his show, if he ever did it again. I'll see this and raise you one. Some time ago I was working on a custom embedded PPC board (running Linux, naturally). After I finished hardening the system against intrusion, I disabled root access and logged off. There was a super secret sneaky method for enabling root access remotely, which I proceeded to try. The method was buggy and root access was not enabled. No worries, I still had serial console access. Which required root access. I also had a couple of open root logins on the board - until my X died. Oh shit. I then proceeded to try and break into the system I just finished hardening to (re)gain root priviledges. A few hours later, I gave up. Cooked up a RiscWatch, sacrificied some blood to the bare hardware gods, hooked it up, and proceeded to reflash a new kernel that should drop me into /bin/sh. Driving the RW was done from a machine several firewalls (and continents over), with the latency you would expect. It was done via a set of shell scripts that usually worked, except when they didn't and completely fried the board. Naturally, they were sensitive to timing. Amazingly, this time they worked. I rebooted the board, dropped into /bin/sh, was happy to discover that everything still worked, restored the old kernel and rebooted. As it was booting, I realized that I haven't enabled root access before rebooting... Cue several more hours of alternately massaging RiscWatch and banging head against wall. Eventually, root access is restored and I go home. Some mistakes you only make once. Cheers, Muli -- Muli Ben-Yehuda http://www.mulix.org | http://mulix.livejournal.com/ = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
