Re: Other migration problems (was Unable to login with ssh)
>From what Alan and Mark said, I understand that not working ssh might have been just a small result of bigger problem - data corruption. So practically anything else might go wrong too. Gregory 2016-06-29 14:01 GMT-04:00 Duerbusch, Tom: > I didn't want to hijack the original thread so > > The conclusion on the original thread was that you need to be on a > supported release of Suse when you go to a new z13. This involved SSH and > I assume crypto hardware. > > We have been planing to go to a z13 this year. > I've said that many times before as we are on a z890, and have planed to go > to a z9, z10, z114, z12 and now z13. > > We have SLES 9, SLES 10 and SLES 11 running. > (After WAVV in 2011, I did manage to sunset our SLES 7 and SLES 8 guests) > > I'm ok with not being able to use new features/functions that are on a > z13/zVM 6, on the unsupported software, but I'm now concerned about what I > might loose, or crash with the new hardware. > > SSH isn't much of a problem, as there are so few people that actually log > on to Linux or use SSH with Linux applications. > > So, the question is: > Is there any other problems that others have come across, when migrating > unsupported SLES systems to a z13? > > Thanks > > Tom Duerbusch > THD Consulting > > -- > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Unable to login with ssh
On Wednesday, 06/29/2016 at 04:58 GMT, "van Sleeuwen, Berry"wrote: > There is a first for everything, this is the first time in 20+ years I have > such problems with unsupported versions on new hardware. Up until last year we even ran VSE 2.2 in > a z10. And we have always has older (unsupported) Linux systems on more recent hardware (SLES8/9 on > z10). I include here by reference, all of my previous posts on the subject of "support". When an OS isn't supported, it means that (a) it failed during testing, or (b) it wasn't tested. In either case, you are exposed to risk. Time to tighten up your OS lifecycle management process. Having current maintenance and support is an important element of high availability. Alan Altmark Senior Managing z/VM and Linux Consultant Lab Services System z Delivery Practice IBM Systems & Technology Group ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Other migration problems (was Unable to login with ssh)
I didn't want to hijack the original thread so The conclusion on the original thread was that you need to be on a supported release of Suse when you go to a new z13. This involved SSH and I assume crypto hardware. We have been planing to go to a z13 this year. I've said that many times before as we are on a z890, and have planed to go to a z9, z10, z114, z12 and now z13. We have SLES 9, SLES 10 and SLES 11 running. (After WAVV in 2011, I did manage to sunset our SLES 7 and SLES 8 guests) I'm ok with not being able to use new features/functions that are on a z13/zVM 6, on the unsupported software, but I'm now concerned about what I might loose, or crash with the new hardware. SSH isn't much of a problem, as there are so few people that actually log on to Linux or use SSH with Linux applications. So, the question is: Is there any other problems that others have come across, when migrating unsupported SLES systems to a z13? Thanks Tom Duerbusch THD Consulting -- -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Unable to login with ssh
On Wednesday, 06/29/2016 at 04:17 GMT, Mark Postwrote: > >>> On 6/29/2016 at 05:42 AM, "van Sleeuwen, Berry" > wrote: > > A few months ago we have moved to a z13 and zVM 6.3. All machines were moved > > successfully. But we now have problems with SLES10 SP4 machines. > > Ah, this is most likely the cause of your problem. Because SLES10 was EOL (and we don't have a > z13), we didn't test it on a z13. IBM, however, has told us to expect problems with data > corruption/loss in such an environment. If your OS doesn't support the z13, don't run it on the z13. Depending on your z13 hw configuration, any operating system that does not explicitly support the z13 will very likely fail. It could be a noisy abend, a hang, or silent data corruption. The only solution is to upgrade the OS to a supported level. One of my clients hit this when they went to GA2 and upgraded to a bigger model. Alan Altmark Senior Managing z/VM and Linux Consultant Lab Services System z Delivery Practice IBM Systems & Technology Group ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Unable to login with ssh
>>> On 6/29/2016 at 12:57 PM, "van Sleeuwen, Berry"wrote: > Hi Mark, > > I was afraid of that. There is a first for everything, this is the first > time in 20+ years I have such problems with unsupported versions on new > hardware. Up until last year we even ran VSE 2.2 in a z10. And we have always > has older (unsupported) Linux systems on more recent hardware (SLES8/9 on > z10). This is the first time the underlying processor architecture has changed radically enough to be of concern. > Anyway. As you mention it might lead to data corruption, got it. But I can > still read the files from those disks in a SLES11 machine. And we don't see > similar problems with SLES10 SP2 (though perhaps we are just lucky). So a > case of YMMV? And would this also affect ssh related issues as we've seen at > the start of the problems? I suppose the corruption would happen on a read access and then made permanent on a write. IBM didn't say it was 100% guaranteed, every time. Just that it _would_ happen. > Is there any way to resolve it? Other than an emergency migration to SLES11 > and hope for the best? (Luckily it's only one production system so the damage > is not too big.) SLES11 SP4 is functionally stabilized, so it might make more sense to go to something more recent and supported for a longer period, such as SLES12 SP1. But yes, I think that's the only way to truly fix this. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Unable to login with ssh
Hi Mark, I was afraid of that. There is a first for everything, this is the first time in 20+ years I have such problems with unsupported versions on new hardware. Up until last year we even ran VSE 2.2 in a z10. And we have always has older (unsupported) Linux systems on more recent hardware (SLES8/9 on z10). Anyway. As you mention it might lead to data corruption, got it. But I can still read the files from those disks in a SLES11 machine. And we don't see similar problems with SLES10 SP2 (though perhaps we are just lucky). So a case of YMMV? And would this also affect ssh related issues as we've seen at the start of the problems? Is there any way to resolve it? Other than an emergency migration to SLES11 and hope for the best? (Luckily it's only one production system so the damage is not too big.) Met vriendelijke groet/With kind regards/Mit freundlichen Grüßen, Berry van Sleeuwen -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Mark Post Sent: Wednesday, June 29, 2016 6:17 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: Unable to login with ssh >>> On 6/29/2016 at 05:42 AM, "van Sleeuwen, Berry" >>>wrote: > A few months ago we have moved to a z13 and zVM 6.3. All machines were > moved successfully. But we now have problems with SLES10 SP4 machines. Ah, this is most likely the cause of your problem. Because SLES10 was EOL (and we don't have a z13), we didn't test it on a z13. IBM, however, has told us to expect problems with data corruption/loss in such an environment. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request.
Request for Session Chairs for Atlanta
We are still in need of individuals to chair sessions at SHARE in Atlanta. If you are planning to attend, why not jump in and help out by moderating a session. If you are going to be sitting in the session anyway... why not? Don't let the thought of chairing a session intimidate you? It's really easy. You just introduce the speaker and the session, pass along some relevant announcements, count the number of attendees, and report that number after the session. Oh, and let the speaker now when the session is almost over. There are still plenty of sessions to choose from, so don't procrastinate and jump right in. Please respond to me directly with the sessions that you would like to chair. Session ID Session Title Day Date and Time Speakers 19325 Lab: Linux for Beginners, Part 1 of 3 Mon 2016-08-01, 10:00:00 Neale Ferguson (Speaker) 19366 Introduction to Virtualization Mon 2016-08-01, 10:00:00 Romney White (Speaker) 19552 Docker for Mainframers Mon 2016-08-01, 10:00:00 Wilhelm Mild (Speaker) 19326 Lab: Linux for Beginners, Part 2 of 3 Mon 2016-08-01, 11:15:00 Neale Ferguson (Speaker) 19444 z/VM - The Leader in Virtualization Mon 2016-08-01, 11:15:00 Bill Bitner (Speaker) 19327 Lab: Linux for Beginners, Part 3 of 3 Mon 2016-08-01, 12:30:00 Neale Ferguson (Speaker) 19320 Hadoop and Data Integration with z Systems Mon 2016-08-01, 13:45:00 Cameron Seay, Doctorate, Educational-Psychology, Master's, Business, Information-Systems, Economics (Speaker), Mike Combs (Speaker) 19363 Introduction to RACF on z/VM Mon 2016-08-01, 13:45:00 Bruce Hayden (Speaker) 19337 What's New with SLES 11 on System z Mon 2016-08-01, 15:15:00 Mark Post (Speaker) 19364 Backup Strategies for z/VM, LinuxONE, and Linux on z Systems Mon 2016-08-01, 15:15:00 Tracy Dean (Speaker) 19365 Overview of CA VM:Secure for z/VM in a Single System Image (SSI) Cluster Mon 2016-08-01, 15:15:00 Brian Jagos (Speaker) 19331 LinuxONE and z Systems Solutions and the Open Source Ecosystem Mon 2016-08-01, 16:30:00 Wilhelm Mild (Speaker), Erich Amrehn (Speaker) 19752 Oracle Database on z Systems High Availability and DR Examples Mon 2016-08-01, 16:30:00 Sam Amsavelu (Speaker), Gaylan Braselton (Speaker) 19324 KVM Concepts and Components for z/VM Lovers Tue 2016-08-02, 10:00:00 Stefan Reimbold, Dr. (Speaker) 19368 How Do You Spell SMT on z/VM? Tue 2016-08-02, 10:00:00 John Franciscovich (Speaker) 19063 Docker: The (complete) Tutorial Tue 2016-08-02, 11:15:00 Alejandro Bonilla (Speaker) 19437 A Look at SMT Performance Tue 2016-08-02, 11:15:00 Bill Bitner (Speaker) 19448 Lab: Managing a LinuxONE, z/VM, and Linux on z Systems Environment Using IBM Solutions Tue 2016-08-02, 12:30:00 Tracy Dean (Speaker), Michael Sine (Speaker), Eduardo C. Oliveira, IBM, Wave, for, z/VM, Tiger, Team, Lead (Speaker) 19322 Introducing IBM LinuxONE and What It Means to You Tue 2016-08-02, 15:15:00 Bryan Foley (Speaker) 19391 Advanced Systems Management of LinuxONE and Linux on z Systems using IBM Wave for z/VM Tue 2016-08-02, 15:15:00 Eduardo C. Oliveira, IBM, Wave, for, z/VM, Tiger, Team, Lead (Speaker) 19323 SHARE Live! - Introduction and Positioning of KVM for IBM z Systems Tue 2016-08-02, 16:30:00 Steffen Thoss (Speaker) 19334 Penguins Behind the Affordable Care Act Tue 2016-08-02, 16:30:00 Mike Giglio, z/VM, Engineer (Speaker) 19441 z/VM Security Essentials Tue 2016-08-02, 16:30:00 Alan Altmark (Speaker) 19394 Lab: Introduction to REXX Workshop (Part 1 of 2) (BYO Device) Wed 2016-08-03, 08:30:00 John Franciscovich (Speaker) 19395 Lab: Introduction to REXX Workshop (Part 2 of 2) (BYO Device) Wed 2016-08-03, 10:00:00 John Franciscovich (Speaker) 19329 Linux Performance Tools in 60 Minutes Wed 2016-08-03, 11:15:00 Stefan Reimbold, Dr. (Speaker) 19393 z/VM Upgrade In Place Installation Wed 2016-08-03, 11:15:00 Richard Lewis (Speaker) 19435 Gen-x and the Mainframe - I Ain't Doing 3270 Wed 2016-08-03, 11:15:00 Jack Acton, Software, Developer (Speaker) 19309 IBM LinuxONE as an Enterprise Cloud Server Wed 2016-08-03, 13:45:00 Jim Elliott (Speaker) 19440 Top Ten Most Listed z/VM Issues from the VM Listserv Since the Last Meeting Wed 2016-08-03, 13:45:00 Brian Jagos (Speaker) 19450 Driving Virtual Mobile Devices Through the zEnterprise Wed 2016-08-03, 13:45:00 Matthew T. Cousens (Speaker) 19396 z/VM Virtual Switch (Part 1): The Basics Wed 2016-08-03, 15:15:00 Alan Altmark (Speaker) 19397 Migrate Running Linux and z/VM Systems to New DASD Without Disruption Wed 2016-08-03, 15:15:00 Mehdi Fadaifard (Speaker) 19551 A Variety of Uses for z Systems Wed 2016-08-03, 15:15:00 Wilhelm Mild (Speaker) 19146 SUSE Linux Technical Update Wed 2016-08-03, 16:30:00 Mark Post (Speaker) 19439 Top 20 Problems to Look for on z/VM and Linux for z Systems Wed 2016-08-03, 16:30:00 Rich Smrcina (Speaker) 19114 Linux & VM Program Dinner Wed 2016-08-03, 19:00:00 Brian Jagos (Speaker), Rick Barlow (Speaker) 19332 Managing Linux Resources with cgroups and Containers Thu 2016-08-04, 08:30:00 Richard G. Young (Speaker) 19433
Re: Unable to login with ssh
>>> On 6/29/2016 at 05:42 AM, "van Sleeuwen, Berry"wrote: > A few months ago we have moved to a z13 and zVM 6.3. All machines were moved > successfully. But we now have problems with SLES10 SP4 machines. Ah, this is most likely the cause of your problem. Because SLES10 was EOL (and we don't have a z13), we didn't test it on a z13. IBM, however, has told us to expect problems with data corruption/loss in such an environment. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Unable to login with ssh
That looks unusual. How did you move those machines? Did you move it from one storage to another as well? How? Or did they stay on the same storage and you've just ipl-ed those on a new system Z? Do those sles10 SP4 live on the same storage as other linux guests? Is it dasd? Did you try to run fsck on those disks after you linked it (W) to other guest? Gregory 2016-06-29 5:42 GMT-04:00 van Sleeuwen, Berry: > I did trace the login ssh process. I haven't tried the top sshd process > yet. But that's because we have another problem now. (Apart from being ill > the last few weeks.) We can't login to the machine at all anymore. So the > problem with the machine gets worse. > > A few months ago we have moved to a z13 and zVM 6.3. All machines were > moved successfully. But we now have problems with SLES10 SP4 machines. We > could logon to them right after the move. At some point ssh failed. And > later on the login fails entirely. Even during boot some services can’t be > started anymore. It looks like the passwd is corrupted. But when I link the > disks in another linux guest I can still read files like passwd, group and > shadow. > > During boot we see some errors, such as: > > Starting D-BUS daemon Could not get password database information for UID > of current process: > User "???" unknown or no memory to allocate password entry > Unknown username "haldaemon" in message bus configuration file > > Starting SSH daemon Privilege separation user sshd does not exist > startproc: exit status of parent of /usr/sbin/sshd: 255 > > When I try to login to the affected machines users are not accepted > anymore. Even in a 3270 console I can't login anymore. > > We have a couple of SLES10 SP2 machines without any problems, all SLES11 > machines function correctly. It looks like to be a problem specific to > SLES10 SP4. > > Could there be an issue specifically related to SLES10 SP4 on zVM 6.3/z13? > > Met vriendelijke groet/With kind regards/Mit freundlichen Grüßen, > Berry van Sleeuwen > > -Original Message- > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of > Mark Post > Sent: Friday, May 27, 2016 7:29 PM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: Unable to login with ssh > > >>> On 5/27/2016 at 08:57 AM, Grzegorz Powiedziuk > wrote: > > One other thing you could try (saved me many times) but a bit trouble > > some is doing some stracing. > > This was something I was going to suggest as well. > > > 1. ssh to the user@server and let it sit on the login 2. on the > > server, do ps auxwww |grep sshd and look for a new spawned process > > Personally I would just pick the "top most" ssh process started at boot > time _before_ trying to connect over the network. > > -snip- > > 3. strace -p 36203 &> logfile.x > > I would probably try strace -f -p 36203 -s500 -o strace.sshd > > You'll need to break out of it using ^c since the process won't terminate > on its own. > > -snip-> > 5. Now examine the the trace by looking at the logile.x (it will be a > big > > file). > > If you don't have terminal server access you'll probably need to access it > via FTP since scp is not likely to work. Which raises a perhaps > interesting question. From the 3270 console are you able to ssh/scp _out_ > of the system? That would at least allow you to send things off the system > to some place you can use "normal" tools. > > > Mark Post > > -- > For LINUX-390 subscribe / signoff / archive access instructions, send > email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- > For more information on Linux on System z, visit http://wiki.linuxvm.org/ > This e-mail and the documents attached are confidential and intended > solely for the addressee; it may also be privileged. If you receive this > e-mail in error, please notify the sender immediately and destroy it. As > its integrity cannot be secured on the Internet, Atos’ liability cannot be > triggered for the message content. Although the sender endeavours to > maintain a computer virus-free network, the sender does not warrant that > this transmission is virus-free and will not be liable for any damages > resulting from any virus transmitted. On all offers and agreements under > which Atos Nederland B.V. supplies goods and/or services of whatever > nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. > The Terms of Delivery shall be promptly submitted to you on your request. > -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Unable to login with ssh
I did trace the login ssh process. I haven't tried the top sshd process yet. But that's because we have another problem now. (Apart from being ill the last few weeks.) We can't login to the machine at all anymore. So the problem with the machine gets worse. A few months ago we have moved to a z13 and zVM 6.3. All machines were moved successfully. But we now have problems with SLES10 SP4 machines. We could logon to them right after the move. At some point ssh failed. And later on the login fails entirely. Even during boot some services can’t be started anymore. It looks like the passwd is corrupted. But when I link the disks in another linux guest I can still read files like passwd, group and shadow. During boot we see some errors, such as: Starting D-BUS daemon Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry Unknown username "haldaemon" in message bus configuration file Starting SSH daemon Privilege separation user sshd does not exist startproc: exit status of parent of /usr/sbin/sshd: 255 When I try to login to the affected machines users are not accepted anymore. Even in a 3270 console I can't login anymore. We have a couple of SLES10 SP2 machines without any problems, all SLES11 machines function correctly. It looks like to be a problem specific to SLES10 SP4. Could there be an issue specifically related to SLES10 SP4 on zVM 6.3/z13? Met vriendelijke groet/With kind regards/Mit freundlichen Grüßen, Berry van Sleeuwen -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Mark Post Sent: Friday, May 27, 2016 7:29 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: Unable to login with ssh >>> On 5/27/2016 at 08:57 AM, Grzegorz Powiedziukwrote: > One other thing you could try (saved me many times) but a bit trouble > some is doing some stracing. This was something I was going to suggest as well. > 1. ssh to the user@server and let it sit on the login 2. on the > server, do ps auxwww |grep sshd and look for a new spawned process Personally I would just pick the "top most" ssh process started at boot time _before_ trying to connect over the network. -snip- > 3. strace -p 36203 &> logfile.x I would probably try strace -f -p 36203 -s500 -o strace.sshd You'll need to break out of it using ^c since the process won't terminate on its own. -snip-> 5. Now examine the the trace by looking at the logile.x (it will be a big > file). If you don't have terminal server access you'll probably need to access it via FTP since scp is not likely to work. Which raises a perhaps interesting question. From the 3270 console are you able to ssh/scp _out_ of the system? That would at least allow you to send things off the system to some place you can use "normal" tools. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request.