Re: Cloudstack 4.7 password reset issue - resolved.

[Rafael Weingärtner]

Those are great improvements, please create the PRs that we will review and
help you prepare the code to be merged. Just be aware that we are going to
have a feature freeze starting next Monday (if I am not mistaken) until the
4.9 version is closed.

On Fri, Apr 15, 2016 at 3:12 PM, Richard Klein (RSI) <rkl...@rsitex.com>
wrote:

> Thanks for the replay and I like the 3 PR as well.  The changes are
> related to the environment we are using.  We are using CS 4.7.0 on CentOS7
> and VXLAN with a two 10Mb NICs per host that is bonded via multi-stack LACP
> switch for performance and redundancy.  We ran into the following 3 issues
> with this configuration.
>
> 1) The bond0 interface had 2 bridge interfaces.  The "cloudbr0" is bridged
> to bond0.  It's used for management and VLAN related guest networks. We
> were limited to 1500 MTU in order to be compatible with our infrastructure
> switches.   The "cloudbr1" is bridge to "bond0." where "" is an
> internal VLAN ID that is restricted to Gluster and VXLAN traffic as well as
> using jumbo frames.  The problem was with the logic in the "modifyvxlan.sh"
> script.  When creating a VXLAN isolated network it would always use the
> "cloudbr0" interface even though the traffic label was defined as
> "cloudbr1".  This was due to how the physical interface search was being
> performed.  There was even a "TODO" in the script that suggesting passing
> the traffic label instead of doing a search.  So I modified the Java back
> end code and the "modifyvslan.sh" script to pass the traffic label and this
> corrected the problem and used the "cloudbr1" interface.
>
> 2) We discovered a very odd problem when migrating instances that use
> VXLAN.  For some reason we always got an error like "brbond0" not found.  I
> can look up the exact verbiage but basically it was not able to create the
> proper VXLAN bridges and such on the target host because it couldn't find
> the physical interface of "brbond0" which is true.  The bridge interface
> should have been "brvx-x".  This turned out to be a problem with
> libvirt and not cloudstack.  You could replicate the problem completely
> isolated from the Cloudstack environment.  For some reason I still can't
> explain, libvirt would always fail migrating a VXLAN.  Oddly enough this
> only happened if the VXLAN bridge interface created by CS started with
> "brvx-" (- is the VXLAN ID). Even if manually created and migrated
> (outside of CS control) we found that this would happen so it behaved as
> bug in libvirt not CS.  The solution was quite simple.  We change the code
> that builds the VXLAN bridges to use "vxbr-".  I really can't explain
> why but the name changed resolved the issue.
>
> 3) This last one is the password change issue we've been discussing which
> as you know was resolved by changing the "configure.py" script.
>
> I will try a pull request on number 3 first.  This is the simplest way for
> me to get use to the process.  I will then follow up on the others as time
> permits.  We are about to implement CS in a production environment so
> hopefully I can get to the PR soon.
>
>
> Richard Klein  <rkl...@rsitex.com>
> RSI
> 5426 Guadalupe, Suite 100
> Austin TX 78751
>
>
>
>
>
> > -Original Message-
> > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> > Sent: Friday, April 15, 2016 12:02 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> >
> > Would it be possible for you to explain a little bit these changes?
> >
> > I believe a PR per change would be the best way to go.
> >
> > On Fri, Apr 15, 2016 at 1:31 PM, Richard Klein (RSI) <rkl...@rsitex.com>
> > wrote:
> >
> > > I would be happy to submit a pull request but I am relatively new to
> > > using Git and GitHub.  I have a lot of experience with SVN and CVS.  I
> > > have read the following link about the process:
> > >
> > >* https://cloudstack.apache.org/developers.html
> > >* https://help.github.com/articles/creating-a-pull-request/
> > >
> > > I have forked the apache/cloudstack on GitHub and have been making
> > > changes to the 4.7.0 version on a separate branch.  This branch
> > > contains several code changes we had to make in order to get CS to run
> in
> > our environment.
> > > Since I am not familiar with Mavin I have created some non-standard
> > > version numbers in order to distinguish the RPMs and use a private
&

RE: Cloudstack 4.7 password reset issue - resolved.

[Richard Klein (RSI)]

Thanks for the replay and I like the 3 PR as well.  The changes are related to 
the environment we are using.  We are using CS 4.7.0 on CentOS7 and VXLAN with 
a two 10Mb NICs per host that is bonded via multi-stack LACP switch for 
performance and redundancy.  We ran into the following 3 issues with this 
configuration.

1) The bond0 interface had 2 bridge interfaces.  The "cloudbr0" is bridged to 
bond0.  It's used for management and VLAN related guest networks. We were 
limited to 1500 MTU in order to be compatible with our infrastructure switches. 
  The "cloudbr1" is bridge to "bond0." where "" is an internal VLAN ID 
that is restricted to Gluster and VXLAN traffic as well as using jumbo frames.  
The problem was with the logic in the "modifyvxlan.sh" script.  When creating a 
VXLAN isolated network it would always use the "cloudbr0" interface even though 
the traffic label was defined as "cloudbr1".  This was due to how the physical 
interface search was being performed.  There was even a "TODO" in the script 
that suggesting passing the traffic label instead of doing a search.  So I 
modified the Java back end code and the "modifyvslan.sh" script to pass the 
traffic label and this corrected the problem and used the "cloudbr1" interface.

2) We discovered a very odd problem when migrating instances that use VXLAN.  
For some reason we always got an error like "brbond0" not found.  I can look up 
the exact verbiage but basically it was not able to create the proper VXLAN 
bridges and such on the target host because it couldn't find the physical 
interface of "brbond0" which is true.  The bridge interface should have been 
"brvx-x".  This turned out to be a problem with libvirt and not cloudstack. 
 You could replicate the problem completely isolated from the Cloudstack 
environment.  For some reason I still can't explain, libvirt would always fail 
migrating a VXLAN.  Oddly enough this only happened if the VXLAN bridge 
interface created by CS started with "brvx-" (- is the VXLAN ID). Even 
if manually created and migrated (outside of CS control) we found that this 
would happen so it behaved as bug in libvirt not CS.  The solution was quite 
simple.  We change the code that builds the VXLAN bridges to use "vxbr-".  
I really can't explain why but the name changed resolved the issue.

3) This last one is the password change issue we've been discussing which as 
you know was resolved by changing the "configure.py" script.

I will try a pull request on number 3 first.  This is the simplest way for me 
to get use to the process.  I will then follow up on the others as time 
permits.  We are about to implement CS in a production environment so hopefully 
I can get to the PR soon.


Richard Klein  <rkl...@rsitex.com> 
RSI 
5426 Guadalupe, Suite 100 
Austin TX 78751 





> -Original Message-----
> From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> Sent: Friday, April 15, 2016 12:02 PM
> To: users@cloudstack.apache.org
> Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> 
> Would it be possible for you to explain a little bit these changes?
> 
> I believe a PR per change would be the best way to go.
> 
> On Fri, Apr 15, 2016 at 1:31 PM, Richard Klein (RSI) <rkl...@rsitex.com>
> wrote:
> 
> > I would be happy to submit a pull request but I am relatively new to
> > using Git and GitHub.  I have a lot of experience with SVN and CVS.  I
> > have read the following link about the process:
> >
> >* https://cloudstack.apache.org/developers.html
> >* https://help.github.com/articles/creating-a-pull-request/
> >
> > I have forked the apache/cloudstack on GitHub and have been making
> > changes to the 4.7.0 version on a separate branch.  This branch
> > contains several code changes we had to make in order to get CS to run in
> our environment.
> > Since I am not familiar with Mavin I have created some non-standard
> > version numbers in order to distinguish the RPMs and use a private
> > repository so we can control the upgrade process.
> >
> > I see 2 options on submitting a pull request.  First is to submit it
> > from the existing branch that contains all the modified code we've
> > made to 4.7.0.  The only downside is it contains a lot of "pom.xml"
> > version number changes as well.  The Second option is to create a
> > branch for each of the 3 types of fixes we have made and do a pull request
> for each one.
> >
> > Let me know if there are any additional resources I need to read up on
> > and the proper method of submitting a pull request.
> >
> > Thanks!
> >
> > Richard 

Re: Cloudstack 4.7 password reset issue - resolved.

[Rafael Weingärtner]

Would it be possible for you to explain a little bit these changes?

I believe a PR per change would be the best way to go.

On Fri, Apr 15, 2016 at 1:31 PM, Richard Klein (RSI) <rkl...@rsitex.com>
wrote:

> I would be happy to submit a pull request but I am relatively new to using
> Git and GitHub.  I have a lot of experience with SVN and CVS.  I have read
> the following link about the process:
>
>* https://cloudstack.apache.org/developers.html
>* https://help.github.com/articles/creating-a-pull-request/
>
> I have forked the apache/cloudstack on GitHub and have been making changes
> to the 4.7.0 version on a separate branch.  This branch contains several
> code changes we had to make in order to get CS to run in our environment.
> Since I am not familiar with Mavin I have created some non-standard version
> numbers in order to distinguish the RPMs and use a private repository so we
> can control the upgrade process.
>
> I see 2 options on submitting a pull request.  First is to submit it from
> the existing branch that contains all the modified code we've made to
> 4.7.0.  The only downside is it contains a lot of "pom.xml" version number
> changes as well.  The Second option is to create a branch for each of the 3
> types of fixes we have made and do a pull request for each one.
>
> Let me know if there are any additional resources I need to read up on and
> the proper method of submitting a pull request.
>
> Thanks!
>
> Richard Klein  <rkl...@rsitex.com>
> RSI
> 5426 Guadalupe, Suite 100
> Austin TX 78751
>
>
>
> > -Original Message-
> > From: Remi Bergsma [mailto:rberg...@schubergphilis.com]
> > Sent: Thursday, April 14, 2016 3:35 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> >
> > Hi Richard,
> >
> > Great you fixed it! Can you send the patch of your fix as a spul request
> on
> > github?
> >
> > Required upgrade is yes when the router reports a version older than the
> > minreq.sysvm.version (or similar) global setting. It's used to upgrade
> systemvm
> > templates.
> >
> > Regards, Remi
> >
> > Sent from my iPhone
> >
> > > On 13 Apr 2016, at 22:23, Richard Klein (RSI) <rkl...@rsitex.com>
> wrote:
> > >
> > > I finally found the problem and resolved the issue.  The problem was
> in the
> > Python code change I made.  I had a flag variable that indicated to save
> data
> > when it was changed while processing a list.  This worked fine as long
> as it
> > executed the logic and defined the flag variable.  The problem was during
> > startup when it doesn't go through the loop and the flag variable was
> > undefined.  This cause the "update_config.py" to fail which then bubbled
> back
> > up as an error and prevent the router from starting.
> > >
> > > Once I changed the code and rebuilt the project all worked well and
> the bug
> > is fixed.  Thanks so much for everyone's help.  This process was very
> > educational and looking forward to learning more.
> > >
> > > I do have one question just out of curiosity.  What makes the "Requires
> > Upgrade" column on the Home->Infrastructure->Virtual Router page
> indicated
> > "Yes"?
> > >
> > > Thanks again,
> > >
> > >
> > > Richard Klein  <rkl...@rsitex.com>
> > > RSI
> > > 5426 Guadalupe, Suite 100
> > > Austin TX 78751
> > >
> > >
> > >
> > >
> > >> -Original Message-
> > >> From: Rajani Karuturi [mailto:raj...@apache.org]
> > >> Sent: Tuesday, April 12, 2016 6:15 AM
> > >> To: users@cloudstack.apache.org
> > >> Subject: Re: Cloudstack 4.7 password reset issue.
> > >>
> > >> Can you check the vm_instance table for the VR entry and update
> > >> vm_template_id?
> > >>
> > >> This might be helpful
> > >> https://gist.github.com/terbolous/102ae8edd1cda192561c
> > >>
> > >> ~Rajani
> > >>
> > >> On Sat, Apr 9, 2016 at 5:45 AM, Richard Klein (RSI)
> > >> <rkl...@rsitex.com>
> > >> wrote:
> > >>
> > >>> I found the password reset issue and it ended up being a Python
> > >>> script on the VR.  I ended up modifying the
> > >>> "/opt/cloud/bin/configure.py" to resolve the issue.  Basically there
> is a
> > "/etc/cloud/vmpassword.json"
> > >>> 

RE: Cloudstack 4.7 password reset issue - resolved.

[Richard Klein (RSI)]

I would be happy to submit a pull request but I am relatively new to using Git 
and GitHub.  I have a lot of experience with SVN and CVS.  I have read the 
following link about the process:

   * https://cloudstack.apache.org/developers.html
   * https://help.github.com/articles/creating-a-pull-request/

I have forked the apache/cloudstack on GitHub and have been making changes to 
the 4.7.0 version on a separate branch.  This branch contains several code 
changes we had to make in order to get CS to run in our environment.  Since I 
am not familiar with Mavin I have created some non-standard version numbers in 
order to distinguish the RPMs and use a private repository so we can control 
the upgrade process.

I see 2 options on submitting a pull request.  First is to submit it from the 
existing branch that contains all the modified code we've made to 4.7.0.  The 
only downside is it contains a lot of "pom.xml" version number changes as well. 
 The Second option is to create a branch for each of the 3 types of fixes we 
have made and do a pull request for each one.

Let me know if there are any additional resources I need to read up on and the 
proper method of submitting a pull request.

Thanks!

Richard Klein  <rkl...@rsitex.com> 
RSI 
5426 Guadalupe, Suite 100 
Austin TX 78751 



> -Original Message-
> From: Remi Bergsma [mailto:rberg...@schubergphilis.com]
> Sent: Thursday, April 14, 2016 3:35 PM
> To: users@cloudstack.apache.org
> Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> 
> Hi Richard,
> 
> Great you fixed it! Can you send the patch of your fix as a spul request on
> github?
> 
> Required upgrade is yes when the router reports a version older than the
> minreq.sysvm.version (or similar) global setting. It's used to upgrade 
> systemvm
> templates.
> 
> Regards, Remi
> 
> Sent from my iPhone
> 
> > On 13 Apr 2016, at 22:23, Richard Klein (RSI) <rkl...@rsitex.com> wrote:
> >
> > I finally found the problem and resolved the issue.  The problem was in the
> Python code change I made.  I had a flag variable that indicated to save data
> when it was changed while processing a list.  This worked fine as long as it
> executed the logic and defined the flag variable.  The problem was during
> startup when it doesn't go through the loop and the flag variable was
> undefined.  This cause the "update_config.py" to fail which then bubbled back
> up as an error and prevent the router from starting.
> >
> > Once I changed the code and rebuilt the project all worked well and the bug
> is fixed.  Thanks so much for everyone's help.  This process was very
> educational and looking forward to learning more.
> >
> > I do have one question just out of curiosity.  What makes the "Requires
> Upgrade" column on the Home->Infrastructure->Virtual Router page indicated
> "Yes"?
> >
> > Thanks again,
> >
> >
> > Richard Klein  <rkl...@rsitex.com>
> > RSI
> > 5426 Guadalupe, Suite 100
> > Austin TX 78751
> >
> >
> >
> >
> >> -Original Message-
> >> From: Rajani Karuturi [mailto:raj...@apache.org]
> >> Sent: Tuesday, April 12, 2016 6:15 AM
> >> To: users@cloudstack.apache.org
> >> Subject: Re: Cloudstack 4.7 password reset issue.
> >>
> >> Can you check the vm_instance table for the VR entry and update
> >> vm_template_id?
> >>
> >> This might be helpful
> >> https://gist.github.com/terbolous/102ae8edd1cda192561c
> >>
> >> ~Rajani
> >>
> >> On Sat, Apr 9, 2016 at 5:45 AM, Richard Klein (RSI)
> >> <rkl...@rsitex.com>
> >> wrote:
> >>
> >>> I found the password reset issue and it ended up being a Python
> >>> script on the VR.  I ended up modifying the
> >>> "/opt/cloud/bin/configure.py" to resolve the issue.  Basically there is a
> "/etc/cloud/vmpassword.json"
> >>> file that is updated with the IP/password pair when the GUI password
> >>> change is performed.  During the power on process the VM
> >>> configuration info is sent to the router which reads the
> >>> vmpassword.json file and sends the password changes to the password
> >>> server cache file.  When the client retrieved the password it was
> >>> cleared from the password cache file but not the vmpassword.json
> >>> file.  So every time a VM started the last password reset was sent to the
> password server again.
> >>>
> >>> The question I have now is how do I get the system VM template
> >>> updated with the ch

Re: Cloudstack 4.7 password reset issue - resolved.

[Remi Bergsma]

Hi Richard,

Great you fixed it! Can you send the patch of your fix as a spul request on 
github?

Required upgrade is yes when the router reports a version older than the 
minreq.sysvm.version (or similar) global setting. It's used to upgrade systemvm 
templates. 

Regards, Remi 

Sent from my iPhone

> On 13 Apr 2016, at 22:23, Richard Klein (RSI) <rkl...@rsitex.com> wrote:
> 
> I finally found the problem and resolved the issue.  The problem was in the 
> Python code change I made.  I had a flag variable that indicated to save data 
> when it was changed while processing a list.  This worked fine as long as it 
> executed the logic and defined the flag variable.  The problem was during 
> startup when it doesn't go through the loop and the flag variable was 
> undefined.  This cause the "update_config.py" to fail which then bubbled back 
> up as an error and prevent the router from starting.
> 
> Once I changed the code and rebuilt the project all worked well and the bug 
> is fixed.  Thanks so much for everyone's help.  This process was very 
> educational and looking forward to learning more.
> 
> I do have one question just out of curiosity.  What makes the "Requires 
> Upgrade" column on the Home->Infrastructure->Virtual Router page indicated 
> "Yes"?
> 
> Thanks again,
> 
> 
> Richard Klein  <rkl...@rsitex.com> 
> RSI 
> 5426 Guadalupe, Suite 100 
> Austin TX 78751 
> 
> 
> 
> 
>> -Original Message-
>> From: Rajani Karuturi [mailto:raj...@apache.org]
>> Sent: Tuesday, April 12, 2016 6:15 AM
>> To: users@cloudstack.apache.org
>> Subject: Re: Cloudstack 4.7 password reset issue.
>> 
>> Can you check the vm_instance table for the VR entry and update
>> vm_template_id?
>> 
>> This might be helpful
>> https://gist.github.com/terbolous/102ae8edd1cda192561c
>> 
>> ~Rajani
>> 
>> On Sat, Apr 9, 2016 at 5:45 AM, Richard Klein (RSI) <rkl...@rsitex.com>
>> wrote:
>> 
>>> I found the password reset issue and it ended up being a Python script
>>> on the VR.  I ended up modifying the "/opt/cloud/bin/configure.py" to
>>> resolve the issue.  Basically there is a "/etc/cloud/vmpassword.json"
>>> file that is updated with the IP/password pair when the GUI password
>>> change is performed.  During the power on process the VM configuration
>>> info is sent to the router which reads the vmpassword.json file and
>>> sends the password changes to the password server cache file.  When
>>> the client retrieved the password it was cleared from the password
>>> cache file but not the vmpassword.json file.  So every time a VM
>>> started the last password reset was sent to the password server again.
>>> 
>>> The question I have now is how do I get the system VM template updated
>>> with the change?  Since we are using CS v4.7 we used the system
>>> template for v4.6 per the installation instructions for CentOS7 and
>>> KVM.  I performed the following steps to use a new system VM template:
>>> 
>>> * I copied the system vm template QCOW2 file from secondary storage to
>>> a work server and made a backup of it.
>>> * On the work server I mounted the QCOW2 template file using
>> "guestmount"
>>> tools and made the code changes to the template.
>>> * I then copied this modified template file to a web server and
>>> registered the template in cloudstack with all checkboxes off except for
>> "routing".
>>> * Then we set the cloudstack global value of "router.template.kvm" to
>>> the name of the new template.
>>> * The management services were restarted.
>>> * I picked a test VR, powered it off, destroyed it then let the system
>>> recreate it.
>>> * When I look at the code I changed on the new VR it does not appear.
>>> 
>>> I even doubled checked the database and the vm_instance table for the
>>> test VR showed the new template ID.  I must be missing something or I
>>> don't really understand how the system templates are created.  Any
>>> help/suggestions would be appreciated.
>>> 
>>> 
>>> 
>>> Richard Klein  <rkl...@rsitex.com>
>>> RSI
>>> 5426 Guadalupe, Suite 100
>>> Austin TX 78751
>>> 
>>> 
>>> 
>>>> -Original Message-
>>>> From: Richard Klein (RSI)
>>>> Sent: Tuesday, April 05, 2016 2:32 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: RE: Cloudstack 4.

RE: Cloudstack 4.7 password reset issue - resolved.

[Richard Klein (RSI)]

I finally found the problem and resolved the issue.  The problem was in the 
Python code change I made.  I had a flag variable that indicated to save data 
when it was changed while processing a list.  This worked fine as long as it 
executed the logic and defined the flag variable.  The problem was during 
startup when it doesn't go through the loop and the flag variable was 
undefined.  This cause the "update_config.py" to fail which then bubbled back 
up as an error and prevent the router from starting.

Once I changed the code and rebuilt the project all worked well and the bug is 
fixed.  Thanks so much for everyone's help.  This process was very educational 
and looking forward to learning more.

I do have one question just out of curiosity.  What makes the "Requires 
Upgrade" column on the Home->Infrastructure->Virtual Router page indicated 
"Yes"?

Thanks again,


Richard Klein  <rkl...@rsitex.com> 
RSI 
5426 Guadalupe, Suite 100 
Austin TX 78751 




> -Original Message-
> From: Rajani Karuturi [mailto:raj...@apache.org]
> Sent: Tuesday, April 12, 2016 6:15 AM
> To: users@cloudstack.apache.org
> Subject: Re: Cloudstack 4.7 password reset issue.
> 
> Can you check the vm_instance table for the VR entry and update
> vm_template_id?
> 
> This might be helpful
> https://gist.github.com/terbolous/102ae8edd1cda192561c
> 
> ~Rajani
> 
> On Sat, Apr 9, 2016 at 5:45 AM, Richard Klein (RSI) <rkl...@rsitex.com>
> wrote:
> 
> > I found the password reset issue and it ended up being a Python script
> > on the VR.  I ended up modifying the "/opt/cloud/bin/configure.py" to
> > resolve the issue.  Basically there is a "/etc/cloud/vmpassword.json"
> > file that is updated with the IP/password pair when the GUI password
> > change is performed.  During the power on process the VM configuration
> > info is sent to the router which reads the vmpassword.json file and
> > sends the password changes to the password server cache file.  When
> > the client retrieved the password it was cleared from the password
> > cache file but not the vmpassword.json file.  So every time a VM
> > started the last password reset was sent to the password server again.
> >
> > The question I have now is how do I get the system VM template updated
> > with the change?  Since we are using CS v4.7 we used the system
> > template for v4.6 per the installation instructions for CentOS7 and
> > KVM.  I performed the following steps to use a new system VM template:
> >
> > * I copied the system vm template QCOW2 file from secondary storage to
> > a work server and made a backup of it.
> > * On the work server I mounted the QCOW2 template file using
> "guestmount"
> > tools and made the code changes to the template.
> > * I then copied this modified template file to a web server and
> > registered the template in cloudstack with all checkboxes off except for
> "routing".
> > * Then we set the cloudstack global value of "router.template.kvm" to
> > the name of the new template.
> > * The management services were restarted.
> > * I picked a test VR, powered it off, destroyed it then let the system
> > recreate it.
> > * When I look at the code I changed on the new VR it does not appear.
> >
> > I even doubled checked the database and the vm_instance table for the
> > test VR showed the new template ID.  I must be missing something or I
> > don't really understand how the system templates are created.  Any
> > help/suggestions would be appreciated.
> >
> >
> >
> > Richard Klein  <rkl...@rsitex.com>
> > RSI
> > 5426 Guadalupe, Suite 100
> > Austin TX 78751
> >
> >
> >
> > > -Original Message-
> > > From: Richard Klein (RSI)
> > > Sent: Tuesday, April 05, 2016 2:32 PM
> > > To: users@cloudstack.apache.org
> > > Subject: RE: Cloudstack 4.7 password reset issue.
> > >
> > > The snippets for before and after the reboot via console look the
> > > same
> > so I
> > > pasted the 2nd set of message instead of the first.  Sorry about that.
> > I did
> > > discover that the /var/lib/dhclient/dhclient.leases existed but was
> > empty.  I've
> > > run across an issue with CentOS 7 where the lease file is missing so
> > > I
> > wrote a
> > > "cloud-dhcp-check" service that makes sure it exists but now I need
> > > to
> > validate
> > > its content.  That being said, I have insured that the
> > > dhclient_leases
> > was valid
> > > and replic

Re: Cloudstack 4.7 password reset issue.

[Rajani Karuturi]

Can you check the vm_instance table for the VR entry and update
vm_template_id?

This might be helpful https://gist.github.com/terbolous/102ae8edd1cda192561c

~Rajani

On Sat, Apr 9, 2016 at 5:45 AM, Richard Klein (RSI) <rkl...@rsitex.com>
wrote:

> I found the password reset issue and it ended up being a Python script on
> the VR.  I ended up modifying the "/opt/cloud/bin/configure.py" to resolve
> the issue.  Basically there is a "/etc/cloud/vmpassword.json" file that is
> updated with the IP/password pair when the GUI password change is
> performed.  During the power on process the VM configuration info is sent
> to the router which reads the vmpassword.json file and sends the password
> changes to the password server cache file.  When the client retrieved the
> password it was cleared from the password cache file but not the
> vmpassword.json file.  So every time a VM started the last password reset
> was sent to the password server again.
>
> The question I have now is how do I get the system VM template updated
> with the change?  Since we are using CS v4.7 we used the system template
> for v4.6 per the installation instructions for CentOS7 and KVM.  I
> performed the following steps to use a new system VM template:
>
> * I copied the system vm template QCOW2 file from secondary storage to a
> work server and made a backup of it.
> * On the work server I mounted the QCOW2 template file using "guestmount"
> tools and made the code changes to the template.
> * I then copied this modified template file to a web server and registered
> the template in cloudstack with all checkboxes off except for "routing".
> * Then we set the cloudstack global value of "router.template.kvm" to the
> name of the new template.
> * The management services were restarted.
> * I picked a test VR, powered it off, destroyed it then let the system
> recreate it.
> * When I look at the code I changed on the new VR it does not appear.
>
> I even doubled checked the database and the vm_instance table for the test
> VR showed the new template ID.  I must be missing something or I don't
> really understand how the system templates are created.  Any
> help/suggestions would be appreciated.
>
>
>
> Richard Klein  <rkl...@rsitex.com>
> RSI
> 5426 Guadalupe, Suite 100
> Austin TX 78751
>
>
>
> > -Original Message-
> > From: Richard Klein (RSI)
> > Sent: Tuesday, April 05, 2016 2:32 PM
> > To: users@cloudstack.apache.org
> > Subject: RE: Cloudstack 4.7 password reset issue.
> >
> > The snippets for before and after the reboot via console look the same
> so I
> > pasted the 2nd set of message instead of the first.  Sorry about that.
> I did
> > discover that the /var/lib/dhclient/dhclient.leases existed but was
> empty.  I've
> > run across an issue with CentOS 7 where the lease file is missing so I
> wrote a
> > "cloud-dhcp-check" service that makes sure it exists but now I need to
> validate
> > its content.  That being said, I have insured that the dhclient_leases
> was valid
> > and replicated the problem.
> >
> > The cloud-set-guest- scripts are from the master branch GitHub
> repository
> > for apaches/cloudstack using the
> > "
> https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-set-
> > guest-password.in" and the
> > "
> https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-set-
> > guest-sshkey.in" links.
> >
> > I have attached the entire log from the VR but have some snippets below
> along
> > with the VM client logs and the issue still occurs after fixing the dhcp
> lease file.
> > I did not perform any password resets via the GUI during this process.
> >
>
>

RE: Cloudstack 4.7 password reset issue.

[Richard Klein (RSI)]

Thanks for the explanation, which really clarified a lot.  I did as you 
suggested and changed the code in a local development WS and then rebuilt the 
RPMs using the following steps:

* Updated code in a local git repository.
* Ran the " mvn clean install -P systemvm ".
* Ran the "./package.sh -d centos7" to create the new RPM's
* Copied the new RPMs to a local repository we have to control updates.
* Stopped the cloudstack-management, cloudstack-usage and cloudstack-agent 
services
* Reinstalled the "cloudstack-common" on all management and hosts using the 
"yum reinstall cloudstack-common".
* Reinstalled the "cloudstack-agent" on the hosts only using "yum reinstall 
cloudstack-agent"
* Started the agents on all hosts.
* Started management services.
* Stopped and then started a test VR but get the following error:  Resource 
[Host:31] is unreachable: Host 31: Unable to start instance due to Unable to 
start VM[DomainRouter|r-1187-VM] due to error in finalizeStart, not retrying

I also restored the system vm template on secondary storage from a backup I 
made before starting any changes.  I tried 2 different VRs and I get this same 
message with a different Host:xx value.

I reverted back to the original code by repeating the above process using the 
original RPMs.  The VR start normally but I still have the original password 
reset issue. Ether I am missing something with the upgrade to a new VM system 
template process or I guess the code change could have caused this problem.  I 
did test the code change by using a live VR and manually make the change.  All 
worked well then but I can't test the stop/restart process cause it reverts 
back to the original scripts.

Did I miss a step or do something incorrectly with the new system VM update?  
Thanks.

--
Richard Klein  <rkl...@rsitex.com> 
RSI 
5426 Guadalupe, Suite 100 
Austin TX 78751 




> -Original Message-
> From: Remi Bergsma [mailto:rberg...@schubergphilis.com]
> Sent: Saturday, April 09, 2016 9:48 AM
> To: users@cloudstack.apache.org
> Subject: Re: Cloudstack 4.7 password reset issue.
> 
> Hi,
> 
> It’s best to make a change in the source of CloudStack, then send it as a Pull
> Request to get it included.
> 
> When you run: `mvn clean install -P systemvm` a new systemvm.iso is
> generated.
> 
> However, the management server will check the systemvm.iso’s signature, so
> I’m not sure if replacing works without also copying it to the management
> server and restart it (so it generates a new signature).
> 
> Another trick is to alter the scripts on live routers with Ansible.
> 
> Regards,
> Remi
> 
> 
> 
> 
> On 09/04/16 03:42, "Simon Weller" <swel...@ena.com> wrote:
> 
> >
> >Richard,
> >
> >The system vms are bootstrapped with an iso that is mounted and all the
> scripts are installed from there. The reason for this is to reduce the number 
> of
> times you have to completely redeploy routers (we used to have to do it for
> every release).
> >My guess is the file you need to change is within that iso.
> >The iso is part of the cloudstack-common package that is installed onto all 
> >the
> hosts (along with the cloudstack-agent).
> >Take a look in /usr/share/cloudstack-common/vms. There should be an iso
> >in there called systemvm.iso
> >
> >You'll probably need to mount the iso make your changes to the filesystem,
> rebuild the iso with mkisofs, copy that new iso to all of your hosts and then 
> stop
> and start your VR.
> >
> >Hope this helps.
> >
> >- Si
> >
> >
> >
> >
> >
> >From: Richard Klein (RSI) <rkl...@rsitex.com>
> >Sent: Friday, April 8, 2016 7:15 PM
> >To: users@cloudstack.apache.org
> >Subject: RE: Cloudstack 4.7 password reset issue.
> >
> >I found the password reset issue and it ended up being a Python script on the
> VR.  I ended up modifying the "/opt/cloud/bin/configure.py" to resolve the
> issue.  Basically there is a "/etc/cloud/vmpassword.json" file that is updated
> with the IP/password pair when the GUI password change is performed.  During
> the power on process the VM configuration info is sent to the router which
> reads the vmpassword.json file and sends the password changes to the
> password server cache file.  When the client retrieved the password it was
> cleared from the password cache file but not the vmpassword.json file.  So
> every time a VM started the last password reset was sent to the password
> server again.
> >
> >The question I have now is how do I get the system VM template updated with
> the chang

Re: Cloudstack 4.7 password reset issue.

[Remi Bergsma]

Hi,

It’s best to make a change in the source of CloudStack, then send it as a Pull 
Request to get it included.

When you run: `mvn clean install -P systemvm` a new systemvm.iso is generated.

However, the management server will check the systemvm.iso’s signature, so I’m 
not sure if replacing works without also copying it to the management server 
and restart it (so it generates a new signature).

Another trick is to alter the scripts on live routers with Ansible.

Regards,
Remi




On 09/04/16 03:42, "Simon Weller" <swel...@ena.com> wrote:

>
>Richard,
>
>The system vms are bootstrapped with an iso that is mounted and all the 
>scripts are installed from there. The reason for this is to reduce the number 
>of times you have to completely redeploy routers (we used to have to do it for 
>every release).
>My guess is the file you need to change is within that iso.
>The iso is part of the cloudstack-common package that is installed onto all 
>the hosts (along with the cloudstack-agent).
>Take a look in /usr/share/cloudstack-common/vms. There should be an iso in 
>there called systemvm.iso
>
>You'll probably need to mount the iso make your changes to the filesystem, 
>rebuild the iso with mkisofs, copy that new iso to all of your hosts and then 
>stop and start your VR.
>
>Hope this helps.
>
>- Si
>
>
>
>
>
>From: Richard Klein (RSI) <rkl...@rsitex.com>
>Sent: Friday, April 8, 2016 7:15 PM
>To: users@cloudstack.apache.org
>Subject: RE: Cloudstack 4.7 password reset issue.
>
>I found the password reset issue and it ended up being a Python script on the 
>VR.  I ended up modifying the "/opt/cloud/bin/configure.py" to resolve the 
>issue.  Basically there is a "/etc/cloud/vmpassword.json" file that is updated 
>with the IP/password pair when the GUI password change is performed.  During 
>the power on process the VM configuration info is sent to the router which 
>reads the vmpassword.json file and sends the password changes to the password 
>server cache file.  When the client retrieved the password it was cleared from 
>the password cache file but not the vmpassword.json file.  So every time a VM 
>started the last password reset was sent to the password server again.
>
>The question I have now is how do I get the system VM template updated with 
>the change?  Since we are using CS v4.7 we used the system template for v4.6 
>per the installation instructions for CentOS7 and KVM.  I performed the 
>following steps to use a new system VM template:
>
>* I copied the system vm template QCOW2 file from secondary storage to a work 
>server and made a backup of it.
>* On the work server I mounted the QCOW2 template file using "guestmount" 
>tools and made the code changes to the template.
>* I then copied this modified template file to a web server and registered the 
>template in cloudstack with all checkboxes off except for "routing".
>* Then we set the cloudstack global value of "router.template.kvm" to the name 
>of the new template.
>* The management services were restarted.
>* I picked a test VR, powered it off, destroyed it then let the system 
>recreate it.
>* When I look at the code I changed on the new VR it does not appear.
>
>I even doubled checked the database and the vm_instance table for the test VR 
>showed the new template ID.  I must be missing something or I don't really 
>understand how the system templates are created.  Any help/suggestions would 
>be appreciated.
>
>
>
>Richard Klein  <rkl...@rsitex.com>
>RSI
>5426 Guadalupe, Suite 100
>Austin TX 78751
>
>
>
>> -Original Message-
>> From: Richard Klein (RSI)
>> Sent: Tuesday, April 05, 2016 2:32 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: Cloudstack 4.7 password reset issue.
>>
>> The snippets for before and after the reboot via console look the same so I
>> pasted the 2nd set of message instead of the first.  Sorry about that.  I did
>> discover that the /var/lib/dhclient/dhclient.leases existed but was empty.  
>> I've
>> run across an issue with CentOS 7 where the lease file is missing so I wrote 
>> a
>> "cloud-dhcp-check" service that makes sure it exists but now I need to 
>> validate
>> its content.  That being said, I have insured that the dhclient_leases was 
>> valid
>> and replicated the problem.
>>
>> The cloud-set-guest- scripts are from the master branch GitHub repository
>> for apaches/cloudstack using the
>> "https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-set-
>> guest-password.in" and the
>> "https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-set-
>> guest-sshkey.in" links.
>>
>> I have attached the entire log from the VR but have some snippets below along
>> with the VM client logs and the issue still occurs after fixing the dhcp 
>> lease file.
>> I did not perform any password resets via the GUI during this process.
>>
>

Re: Cloudstack 4.7 password reset issue.

[Simon Weller]

Richard,

The system vms are bootstrapped with an iso that is mounted and all the scripts 
are installed from there. The reason for this is to reduce the number of times 
you have to completely redeploy routers (we used to have to do it for every 
release).
My guess is the file you need to change is within that iso.
The iso is part of the cloudstack-common package that is installed onto all the 
hosts (along with the cloudstack-agent).
Take a look in /usr/share/cloudstack-common/vms. There should be an iso in 
there called systemvm.iso

You'll probably need to mount the iso make your changes to the filesystem, 
rebuild the iso with mkisofs, copy that new iso to all of your hosts and then 
stop and start your VR.

Hope this helps.

- Si





From: Richard Klein (RSI) <rkl...@rsitex.com>
Sent: Friday, April 8, 2016 7:15 PM
To: users@cloudstack.apache.org
Subject: RE: Cloudstack 4.7 password reset issue.

I found the password reset issue and it ended up being a Python script on the 
VR.  I ended up modifying the "/opt/cloud/bin/configure.py" to resolve the 
issue.  Basically there is a "/etc/cloud/vmpassword.json" file that is updated 
with the IP/password pair when the GUI password change is performed.  During 
the power on process the VM configuration info is sent to the router which 
reads the vmpassword.json file and sends the password changes to the password 
server cache file.  When the client retrieved the password it was cleared from 
the password cache file but not the vmpassword.json file.  So every time a VM 
started the last password reset was sent to the password server again.

The question I have now is how do I get the system VM template updated with the 
change?  Since we are using CS v4.7 we used the system template for v4.6 per 
the installation instructions for CentOS7 and KVM.  I performed the following 
steps to use a new system VM template:

* I copied the system vm template QCOW2 file from secondary storage to a work 
server and made a backup of it.
* On the work server I mounted the QCOW2 template file using "guestmount" tools 
and made the code changes to the template.
* I then copied this modified template file to a web server and registered the 
template in cloudstack with all checkboxes off except for "routing".
* Then we set the cloudstack global value of "router.template.kvm" to the name 
of the new template.
* The management services were restarted.
* I picked a test VR, powered it off, destroyed it then let the system recreate 
it.
* When I look at the code I changed on the new VR it does not appear.

I even doubled checked the database and the vm_instance table for the test VR 
showed the new template ID.  I must be missing something or I don't really 
understand how the system templates are created.  Any help/suggestions would be 
appreciated.



Richard Klein  <rkl...@rsitex.com>
RSI
5426 Guadalupe, Suite 100
Austin TX 78751



> -Original Message-
> From: Richard Klein (RSI)
> Sent: Tuesday, April 05, 2016 2:32 PM
> To: users@cloudstack.apache.org
> Subject: RE: Cloudstack 4.7 password reset issue.
>
> The snippets for before and after the reboot via console look the same so I
> pasted the 2nd set of message instead of the first.  Sorry about that.  I did
> discover that the /var/lib/dhclient/dhclient.leases existed but was empty.  
> I've
> run across an issue with CentOS 7 where the lease file is missing so I wrote a
> "cloud-dhcp-check" service that makes sure it exists but now I need to 
> validate
> its content.  That being said, I have insured that the dhclient_leases was 
> valid
> and replicated the problem.
>
> The cloud-set-guest- scripts are from the master branch GitHub repository
> for apaches/cloudstack using the
> "https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-set-
> guest-password.in" and the
> "https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-set-
> guest-sshkey.in" links.
>
> I have attached the entire log from the VR but have some snippets below along
> with the VM client logs and the issue still occurs after fixing the dhcp 
> lease file.
> I did not perform any password resets via the GUI during this process.
>

RE: Cloudstack 4.7 password reset issue.

[Richard Klein (RSI)]

I found the password reset issue and it ended up being a Python script on the 
VR.  I ended up modifying the "/opt/cloud/bin/configure.py" to resolve the 
issue.  Basically there is a "/etc/cloud/vmpassword.json" file that is updated 
with the IP/password pair when the GUI password change is performed.  During 
the power on process the VM configuration info is sent to the router which 
reads the vmpassword.json file and sends the password changes to the password 
server cache file.  When the client retrieved the password it was cleared from 
the password cache file but not the vmpassword.json file.  So every time a VM 
started the last password reset was sent to the password server again.

The question I have now is how do I get the system VM template updated with the 
change?  Since we are using CS v4.7 we used the system template for v4.6 per 
the installation instructions for CentOS7 and KVM.  I performed the following 
steps to use a new system VM template:

* I copied the system vm template QCOW2 file from secondary storage to a work 
server and made a backup of it.
* On the work server I mounted the QCOW2 template file using "guestmount" tools 
and made the code changes to the template.
* I then copied this modified template file to a web server and registered the 
template in cloudstack with all checkboxes off except for "routing".
* Then we set the cloudstack global value of "router.template.kvm" to the name 
of the new template.
* The management services were restarted.
* I picked a test VR, powered it off, destroyed it then let the system recreate 
it.
* When I look at the code I changed on the new VR it does not appear.

I even doubled checked the database and the vm_instance table for the test VR 
showed the new template ID.  I must be missing something or I don't really 
understand how the system templates are created.  Any help/suggestions would be 
appreciated.



Richard Klein  <rkl...@rsitex.com> 
RSI 
5426 Guadalupe, Suite 100 
Austin TX 78751 



> -Original Message-
> From: Richard Klein (RSI)
> Sent: Tuesday, April 05, 2016 2:32 PM
> To: users@cloudstack.apache.org
> Subject: RE: Cloudstack 4.7 password reset issue.
> 
> The snippets for before and after the reboot via console look the same so I
> pasted the 2nd set of message instead of the first.  Sorry about that.  I did
> discover that the /var/lib/dhclient/dhclient.leases existed but was empty.  
> I've
> run across an issue with CentOS 7 where the lease file is missing so I wrote a
> "cloud-dhcp-check" service that makes sure it exists but now I need to 
> validate
> its content.  That being said, I have insured that the dhclient_leases was 
> valid
> and replicated the problem.
> 
> The cloud-set-guest- scripts are from the master branch GitHub repository
> for apaches/cloudstack using the
> "https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-set-
> guest-password.in" and the
> "https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-set-
> guest-sshkey.in" links.
> 
> I have attached the entire log from the VR but have some snippets below along
> with the VM client logs and the issue still occurs after fixing the dhcp 
> lease file.
> I did not perform any password resets via the GUI during this process.
> 

RE: Cloudstack 4.7 password reset issue.

[Richard Klein (RSI)]

al cloud-init job (metadata 
service crawler)...
Apr  5 13:33:18 rsi-test-c7 systemd: Started Initial cloud-init job (metadata 
service crawler).
Apr  5 13:33:18 rsi-test-c7 systemd: Starting Apply the settings specified in 
cloud-config...
Apr  5 13:33:19 rsi-test-c7 systemd: Started Apply the settings specified in 
cloud-config.
Apr  5 13:33:19 rsi-test-c7 systemd: Starting Execute cloud user/final 
scripts...
Apr  5 13:33:19 rsi-test-c7 systemd: Started Execute cloud user/final scripts.

-- Changed root password via VM console then powered off via GUI --
Apr  5 13:37:10 rsi-test-c7 systemd: Stopped Execute cloud user/final scripts.
Apr  5 13:37:10 rsi-test-c7 systemd: Stopping Execute cloud user/final 
scripts...
Apr  5 13:37:10 rsi-test-c7 systemd: Stopped Apply the settings specified in 
cloud-config.
Apr  5 13:37:10 rsi-test-c7 systemd: Stopping Apply the settings specified in 
cloud-config...

-- Powered on via GUI --
Apr  5 13:38:47 rsi-test-c7 systemd: Starting Initial cloud-init job 
(pre-networking)...
Apr  5 13:38:49 rsi-test-c7 systemd: Started Initial cloud-init job 
(pre-networking).
Apr  5 13:38:51 rsi-test-c7 NetworkManager[535]: domain name 
'cs2cloud.internal'
Apr  5 13:38:56 rsi-test-c7 systemd: Starting LSB: cloud-dhcp-check...
Apr  5 13:38:56 rsi-test-c7 cloud-dhcp-check: Checking for DHCP leas file...
Apr  5 13:38:57 rsi-test-c7 cloud-dhcp-check: DHCP lease found: 
/var/lib/dhclient/dhclient.leases
Apr  5 13:38:57 rsi-test-c7 systemd: Started LSB: cloud-dhcp-check.
Apr  5 13:38:57 rsi-test-c7 systemd: Starting LSB: cloud-set-guest-sshkey...
Apr  5 13:38:57 rsi-test-c7 systemd: Starting LSB: cloud-set-guest-password...
Apr  5 13:38:57 rsi-test-c7 cloud: Found ssh key server at 10.1.1.1
Apr  5 13:38:57 rsi-test-c7 cloud: Found password server IP 10.1.1.1 in 
/var/lib/dhclient/dhclient.leases
Apr  5 13:38:57 rsi-test-c7 cloud: Sending request to ssh key server at 10.1.1.1
Apr  5 13:38:57 rsi-test-c7 cloud: Sending request to password server at 
10.1.1.1
Apr  5 13:38:57 rsi-test-c7 cloud: Got response from server at 10.1.1.1
Apr  5 13:38:57 rsi-test-c7 cloud: Did not receive any keys from any server
Apr  5 13:38:57 rsi-test-c7 systemd: cloud-set-guest-sshkey.service: control 
process exited, code=exited status=1
Apr  5 13:38:57 rsi-test-c7 systemd: Failed to start LSB: 
cloud-set-guest-sshkey.
Apr  5 13:38:57 rsi-test-c7 systemd: Unit cloud-set-guest-sshkey.service 
entered failed state.
Apr  5 13:38:57 rsi-test-c7 systemd: cloud-set-guest-sshkey.service failed.
Apr  5 13:38:57 rsi-test-c7 cloud: Got response from server at 10.1.1.1
Apr  5 13:38:57 rsi-test-c7 cloud: VM got a valid password from server at 
10.1.1.1
Apr  5 13:38:57 rsi-test-c7 cloud: Changing password for user root
Apr  5 13:38:57 rsi-test-c7 cloud: Sending acknowledgment to password server at 
10.1.1.1
Apr  5 13:38:57 rsi-test-c7 cloud-set-guest-password: saved_password
Apr  5 13:38:57 rsi-test-c7 systemd: Started LSB: cloud-set-guest-password.
Apr  5 13:38:57 rsi-test-c7 systemd: Starting Initial cloud-init job (metadata 
service crawler)...
Apr  5 13:38:58 rsi-test-c7 systemd: Started Initial cloud-init job (metadata 
service crawler).
Apr  5 13:38:58 rsi-test-c7 systemd: Starting Apply the settings specified in 
cloud-config...
Apr  5 13:38:58 rsi-test-c7 systemd: Started Apply the settings specified in 
cloud-config.
Apr  5 13:38:58 rsi-test-c7 systemd: Starting Execute cloud user/final 
scripts...
Apr  5 13:38:59 rsi-test-c7 systemd: Started Execute cloud user/final scripts.


Let me know if I've missed something or you need further info.


Richard Klein  <rkl...@rsitex.com>
RSI
5426 Guadalupe, Suite 100
Austin TX 78751


> -Original Message-
> From: Erik Weber [mailto:terbol...@gmail.com]
> Sent: Tuesday, April 05, 2016 12:30 AM
> To: users@cloudstack.apache.org
> Subject: Re: Cloudstack 4.7 password reset issue.
>
> On Mon, Apr 4, 2016 at 9:17 PM, Richard Klein (RSI) <rkl...@rsitex.com>
> wrote:
>
> > Thanks for the reply and sorry for the delay.  You know how it is on
> > Monday's.
> >
> > I checked the VR password file as suggested after the password reset
> > via the GUI but before powering on the VM.  I did see the password
> > reset there as expected.  I powered on the VM and the password was reset as
> expected.
> > The test VM is built from a CentOS7 template with password enabled.  I
> > checked the /var/log/messages and you can see from the client
> > perspective the change went as expected.  Below is "snippet1" from the
> > message log on the VM (there are no SSH key so it fails as expected):
> >
> >
> This is your first VM snippet:
>
>
> > Apr  4 13:45:35 rsi-test-c7 systemd: Starting LSB:
> > cloud-set-guest-sshkey...
> > Apr  4 13:45:35 rsi-test-c7 systemd: Starting LSB:
> > cloud-set-guest-password...
> > Apr  4 13:45:35 rsi-test-c

Re: Cloudstack 4.7 password reset issue.

[Erik Weber]

On Mon, Apr 4, 2016 at 9:17 PM, Richard Klein (RSI) 
wrote:

> Thanks for the reply and sorry for the delay.  You know how it is on
> Monday's.
>
> I checked the VR password file as suggested after the password reset via
> the GUI but before powering on the VM.  I did see the password reset there
> as expected.  I powered on the VM and the password was reset as expected.
> The test VM is built from a CentOS7 template with password enabled.  I
> checked the /var/log/messages and you can see from the client perspective
> the change went as expected.  Below is "snippet1" from the message log on
> the VM (there are no SSH key so it fails as expected):
>
>
This is your first VM snippet:


> Apr  4 13:45:35 rsi-test-c7 systemd: Starting LSB:
> cloud-set-guest-sshkey...
> Apr  4 13:45:35 rsi-test-c7 systemd: Starting LSB:
> cloud-set-guest-password...
> Apr  4 13:45:35 rsi-test-c7 cloud: Unable to determine the password
> server, falling back to data-server
> Apr  4 13:45:35 rsi-test-c7 cloud: Could not find ssh key server IP in
> /var/lib/dhclient/dhclient.leases
> Apr  4 13:45:36 rsi-test-c7 cloud: Unable to determine the password
> server, falling back to data-server
> Apr  4 13:45:36 rsi-test-c7 cloud: Sending request to password server at
> data-server
> Apr  4 13:45:36 rsi-test-c7 cloud: Sending request to ssh key server at
> data-server
> Apr  4 13:45:36 rsi-test-c7 cloud: Got response from server at data-server
> Apr  4 13:45:36 rsi-test-c7 cloud: Did not receive any keys from any server
> Apr  4 13:45:36 rsi-test-c7 systemd: cloud-set-guest-sshkey.service:
> control process exited, code=exited status=1
> Apr  4 13:45:36 rsi-test-c7 systemd: Failed to start LSB:
> cloud-set-guest-sshkey.
> Apr  4 13:45:36 rsi-test-c7 systemd: Unit cloud-set-guest-sshkey.service
> entered failed state.
> Apr  4 13:45:36 rsi-test-c7 systemd: cloud-set-guest-sshkey.service failed.
> Apr  4 13:45:36 rsi-test-c7 cloud: Got response from server at data-server
> Apr  4 13:45:36 rsi-test-c7 cloud: VM got a valid password from server at
> data-server
> Apr  4 13:45:36 rsi-test-c7 cloud: Changing password for user root
> Apr  4 13:45:36 rsi-test-c7 cloud: Sending acknowledgment to password
> server at data-server
> Apr  4 13:45:36 rsi-test-c7 cloud-set-guest-password: saved_password
> Apr  4 13:45:36 rsi-test-c7 systemd: Started LSB: cloud-set-guest-password.
> Apr  4 13:45:36 rsi-test-c7 systemd: Starting Initial cloud-init job
> (metadata service crawler)...
> Apr  4 13:45:37 rsi-test-c7 systemd: Started Initial cloud-init job
> (metadata service crawler).
> Apr  4 13:45:37 rsi-test-c7 systemd: Starting Apply the settings specified
> in cloud-config...
> Apr  4 13:45:37 rsi-test-c7 systemd: Started Apply the settings specified
> in cloud-config.
> Apr  4 13:45:37 rsi-test-c7 systemd: Starting Execute cloud user/final
> scripts...
> Apr  4 13:45:37 rsi-test-c7 systemd: Started Execute cloud user/final
> scripts.
>
> I checked the password file on the VR and it was empty now.  I then
> manually reset the password on the VM console only (not CS GUI) and
> performed an OS reboot from the console.  I could log in with the password
> I change with the console.  Below is the system messages "snippet2":
>
>
This is your second VM snippet, happening before snippet1


> Apr  4 13:36:27 rsi-test-c7 systemd: Starting LSB:
> cloud-set-guest-sshkey...
> Apr  4 13:36:27 rsi-test-c7 systemd: Starting LSB:
> cloud-set-guest-password...
> Apr  4 13:36:27 rsi-test-c7 cloud: Could not find ssh key server IP in
> /var/lib/dhclient/dhclient.leases
>

Do you have such file, or is there a configuration error in the
cloud-set-guest-password?
You seem to be running centos7, and that wasn't around when these scripts
where made and thus might have to be modified for them to work.


> Apr  4 13:36:27 rsi-test-c7 cloud: Unable to determine the password
> server, falling back to data-server
> Apr  4 13:36:27 rsi-test-c7 cloud: Unable to determine the password
> server, falling back to data-server
> Apr  4 13:36:27 rsi-test-c7 cloud: Sending request to password server at
> data-server
> Apr  4 13:36:27 rsi-test-c7 cloud: Sending request to ssh key server at
> data-server
> Apr  4 13:36:27 rsi-test-c7 cloud: Got response from server at data-server
> Apr  4 13:36:27 rsi-test-c7 cloud: Did not receive any keys from any server
> Apr  4 13:36:27 rsi-test-c7 systemd: cloud-set-guest-sshkey.service:
> control process exited, code=exited status=1
> Apr  4 13:36:27 rsi-test-c7 systemd: Failed to start LSB:
> cloud-set-guest-sshkey.
> Apr  4 13:36:27 rsi-test-c7 systemd: Unit cloud-set-guest-sshkey.service
> entered failed state.
> Apr  4 13:36:27 rsi-test-c7 systemd: cloud-set-guest-sshkey.service failed.
> Apr  4 13:36:27 rsi-test-c7 cloud: Got response from server at data-server
> Apr  4 13:36:27 rsi-test-c7 cloud: VM has already saved a password from
> the password server at data-server
> Apr  4 13:36:27 rsi-test-c7 cloud: Did not need to change 

Re: Cloudstack 4.7 password reset issue.

[Shweta Agarwal]

Hi Richard,
This looks like a bug to me. The password should be pushed only first time of  
VM start and only after User reset password of VM via CS.

Thanks
Shweta
Principal Product Engineer, CloudPlatform
Accelerite


From: Richard Klein (RSI) <rkl...@rsitex.com>
Sent: Tuesday, April 5, 2016 12:47 AM
To: users@cloudstack.apache.org
Subject: RE: Cloudstack 4.7 password reset issue.

Thanks for the reply and sorry for the delay.  You know how it is on Monday's.

I checked the VR password file as suggested after the password reset via the 
GUI but before powering on the VM.  I did see the password reset there as 
expected.  I powered on the VM and the password was reset as expected.  The 
test VM is built from a CentOS7 template with password enabled.  I checked the 
/var/log/messages and you can see from the client perspective the change went 
as expected.  Below is "snippet1" from the message log on the VM (there are no 
SSH key so it fails as expected):

Apr  4 13:45:35 rsi-test-c7 systemd: Starting LSB: cloud-set-guest-sshkey...
Apr  4 13:45:35 rsi-test-c7 systemd: Starting LSB: cloud-set-guest-password...
Apr  4 13:45:35 rsi-test-c7 cloud: Unable to determine the password server, 
falling back to data-server
Apr  4 13:45:35 rsi-test-c7 cloud: Could not find ssh key server IP in 
/var/lib/dhclient/dhclient.leases
Apr  4 13:45:36 rsi-test-c7 cloud: Unable to determine the password server, 
falling back to data-server
Apr  4 13:45:36 rsi-test-c7 cloud: Sending request to password server at 
data-server
Apr  4 13:45:36 rsi-test-c7 cloud: Sending request to ssh key server at 
data-server
Apr  4 13:45:36 rsi-test-c7 cloud: Got response from server at data-server
Apr  4 13:45:36 rsi-test-c7 cloud: Did not receive any keys from any server
Apr  4 13:45:36 rsi-test-c7 systemd: cloud-set-guest-sshkey.service: control 
process exited, code=exited status=1
Apr  4 13:45:36 rsi-test-c7 systemd: Failed to start LSB: 
cloud-set-guest-sshkey.
Apr  4 13:45:36 rsi-test-c7 systemd: Unit cloud-set-guest-sshkey.service 
entered failed state.
Apr  4 13:45:36 rsi-test-c7 systemd: cloud-set-guest-sshkey.service failed.
Apr  4 13:45:36 rsi-test-c7 cloud: Got response from server at data-server
Apr  4 13:45:36 rsi-test-c7 cloud: VM got a valid password from server at 
data-server
Apr  4 13:45:36 rsi-test-c7 cloud: Changing password for user root
Apr  4 13:45:36 rsi-test-c7 cloud: Sending acknowledgment to password server at 
data-server
Apr  4 13:45:36 rsi-test-c7 cloud-set-guest-password: saved_password
Apr  4 13:45:36 rsi-test-c7 systemd: Started LSB: cloud-set-guest-password.
Apr  4 13:45:36 rsi-test-c7 systemd: Starting Initial cloud-init job (metadata 
service crawler)...
Apr  4 13:45:37 rsi-test-c7 systemd: Started Initial cloud-init job (metadata 
service crawler).
Apr  4 13:45:37 rsi-test-c7 systemd: Starting Apply the settings specified in 
cloud-config...
Apr  4 13:45:37 rsi-test-c7 systemd: Started Apply the settings specified in 
cloud-config.
Apr  4 13:45:37 rsi-test-c7 systemd: Starting Execute cloud user/final 
scripts...
Apr  4 13:45:37 rsi-test-c7 systemd: Started Execute cloud user/final scripts.

I checked the password file on the VR and it was empty now.  I then manually 
reset the password on the VM console only (not CS GUI) and performed an OS 
reboot from the console.  I could log in with the password I change with the 
console.  Below is the system messages "snippet2":

Apr  4 13:36:27 rsi-test-c7 systemd: Starting LSB: cloud-set-guest-sshkey...
Apr  4 13:36:27 rsi-test-c7 systemd: Starting LSB: cloud-set-guest-password...
Apr  4 13:36:27 rsi-test-c7 cloud: Could not find ssh key server IP in 
/var/lib/dhclient/dhclient.leases
Apr  4 13:36:27 rsi-test-c7 cloud: Unable to determine the password server, 
falling back to data-server
Apr  4 13:36:27 rsi-test-c7 cloud: Unable to determine the password server, 
falling back to data-server
Apr  4 13:36:27 rsi-test-c7 cloud: Sending request to password server at 
data-server
Apr  4 13:36:27 rsi-test-c7 cloud: Sending request to ssh key server at 
data-server
Apr  4 13:36:27 rsi-test-c7 cloud: Got response from server at data-server
Apr  4 13:36:27 rsi-test-c7 cloud: Did not receive any keys from any server
Apr  4 13:36:27 rsi-test-c7 systemd: cloud-set-guest-sshkey.service: control 
process exited, code=exited status=1
Apr  4 13:36:27 rsi-test-c7 systemd: Failed to start LSB: 
cloud-set-guest-sshkey.
Apr  4 13:36:27 rsi-test-c7 systemd: Unit cloud-set-guest-sshkey.service 
entered failed state.
Apr  4 13:36:27 rsi-test-c7 systemd: cloud-set-guest-sshkey.service failed.
Apr  4 13:36:27 rsi-test-c7 cloud: Got response from server at data-server
Apr  4 13:36:27 rsi-test-c7 cloud: VM has already saved a password from the 
password server at data-server
Apr  4 13:36:27 rsi-test-c7 cloud: Did not need to change password.
Apr  4 13:36:27 rsi-test-c7 systemd: Started LSB: cloud-set-guest-password.

RE: Cloudstack 4.7 password reset issue.

[Richard Klein (RSI)]

ollowing 2 
messages stood out to me:

2016-04-04 18:45:06,937  merge.py save:72 {u'id': u'dhcpentry', u'10.1.1.39': 
{u'default_gateway': u'10.1.1.1', u'ipv6_duid': u'00:03:00:01:
02:00:26:c0:00:07', u'default_entry': True, u'ipv4_adress': u'10.1.1.39', 
u'host_name': u'rsi-test-c7', u'mac_address': u'02:00:26:c0:00:07'
, u'type': u'dhcpentry', u'dns_adresses': u'10.1.1.1'}}

2016-04-04 18:45:07,006  CsHelper.py execute:160 Executing: curl --header 
"DomU_Request: save_password" "http://10.1.1.1:8080/; -F "ip=10.1.1.39" -F 
"password=a8Q7Bs" -F "token=ae199529efcd26f7811f36ae7c0807b1" >/dev/null 
2>/dev/null &

The VM I powered on IP address is 10.1.1.39 and it appears to me that the 
CsHelper.py curl command is setting the password again?  Making an educated 
guess it looks like the management process is sending the reset password again 
to the VR after the power up?

Please let me know what you think or if you need further information just let 
me know.  Thanks.


Richard Klein  <rkl...@rsitex.com> 
RSI 
5426 Guadalupe, Suite 100 
Austin TX 78751 



>-Original Message-
>From: Suresh Sadhu [mailto:suresh.sa...@accelerite.com] 
>Sent: Monday, April 04, 2016 5:21 AM
>To: users@cloudstack.apache.org
>Subject: RE: Cloudstack 4.7 password reset issue.
>
>This look like an issue, please check the VR password file 
>(/var/cache/cloud/passwords) and entries look like below: Guest_vm_ip=Password)
>Eg:
>10.1.1.145=sX3qvbstr   --- here, In this case it  successfully 
>generated the password  but didn't applied on Guest vm
>10.1.1.135=saved_password   -- if it  applied successfully on VM  then it 
>will show as  guest_vm_ip=Saved_password.) 
>
>So please check you entries as well check your logs for any errors. 
>
>Regards
>Sadhu
>Chief Product Engineer, Accelerite
>suresh.sa...@accelerite.com


>>-Original Message-
>>From: Erik Weber [mailto:terbol...@gmail.com]
>>Sent: Monday, April 4, 2016 11:54 AM
>>To: users@cloudstack.apache.org
>>Subject: Re: Cloudstack 4.7 password reset issue.
>>
>>On Mon, Apr 4, 2016 at 7:28 AM, Shweta Agarwal < 
>>shweta.agarw...@accelerite.com> wrote:
>>
>> Hi Richard,
>> Cloudstack is working as expected at every steps you mentioned .
>>
>> when your template is password enabled  then every time you reset 
>> password via CS and stop start the VM then router will push new 
>> password to the user VM .
>>

This works, but if you read his example you will see that the password is reset 
after an additional reboot, which it should not.

Looks like a bug, but more info is needed to state if it is in CloudStack or 
the password reset script.

--
Erik



DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

RE: Cloudstack 4.7 password reset issue.

[Suresh Sadhu]

This look like an issue, please check the VR password file 
(/var/cache/cloud/passwords) and entries look like below: Guest_vm_ip=Password)
Eg:
10.1.1.145=sX3qvbstr--- here, In this case it  successfully 
generated the password  but didn't applied on Guest vm
10.1.1.135=saved_password-- if it  applied successfully on VM  then it 
will show as  guest_vm_ip=Saved_password.)

So please check you entries as well check your logs for any errors. 

Regards
Sadhu
Chief Product Engineer, Accelerite
suresh.sa...@accelerite.com


-Original Message-
From: Erik Weber [mailto:terbol...@gmail.com] 
Sent: Monday, April 4, 2016 11:54 AM
To: users@cloudstack.apache.org
Subject: Re: Cloudstack 4.7 password reset issue.

On Mon, Apr 4, 2016 at 7:28 AM, Shweta Agarwal < 
shweta.agarw...@accelerite.com> wrote:

> Hi Richard,
> Cloudstack is working as expected at every steps you mentioned .
>
> when your template is password enabled  then every time you reset 
> password via CS and stop start the VM then router will push new 
> password to the user VM .
>

This works, but if you read his example you will see that the password is reset 
after an additional reboot, which it should not.

Looks like a bug, but more info is needed to state if it is in CloudStack or 
the password reset script.

--
Erik



DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

Re: Cloudstack 4.7 password reset issue.

[Erik Weber]

On Mon, Apr 4, 2016 at 7:28 AM, Shweta Agarwal <
shweta.agarw...@accelerite.com> wrote:

> Hi Richard,
> Cloudstack is working as expected at every steps you mentioned .
>
> when your template is password enabled  then every time you reset password
> via CS and stop start the VM then router will push new password to the user
> VM .
>

This works, but if you read his example you will see that the password is
reset after an additional reboot, which it should not.

Looks like a bug, but more info is needed to state if it is in CloudStack
or the password reset script.

-- 
Erik

Re: Cloudstack 4.7 password reset issue.

[Shweta Agarwal]

Hi Richard,
Cloudstack is working as expected at every steps you mentioned .

when your template is password enabled  then every time you reset password via 
CS and stop start the VM then router will push new password to the user VM .
If you want a VM with your set password then create a VM  with a template 
having no password enabled check . In that case when you login to VM and change 
then password via VM console then on stop start of VM the password will remain 
what you set via consol.

Hope it  helps to understand thing



Thanks
Shweta
Principal Product Engineer, CloudPlatform
Accelerite


From: Erik Weber <terbol...@gmail.com>
Sent: Sunday, April 3, 2016 12:40 AM
To: users@cloudstack.apache.org
Subject: Re: Cloudstack 4.7 password reset issue.

On Sat, Apr 2, 2016 at 9:04 PM, Richard Klein (RSI) <rkl...@rsitex.com>
wrote:

> First time mailer here, hello everyone.



Hi, and welcome!



> We are implementing CS 4.7 cluster with CentOS7.  All is going well with a
> few bumps that we have resolved.  During testing we are getting an unusual
> behavior from the password reset process.  We have setup the templates
> using the instructions from the documentation including installing the
> guest password and SSH setup scripts.  Below is what happens:
>
>  * Create a VM from template that has the password option checked and the
> root password is set as expected.
>  * Log in with 1st password and then change password of root via the
> console of the VM
>  * Perform a reboot from the VM console and the password stays as set
> manually.
>  * Power off VM, change PW with CS web GUI.  When the VM is powered on the
> password is the new one (as expected).
>  * Log in with new password on the console and then manually change root
> again.
>  * Reboot from the console and the password remains as manually set.
>  * Power off VM via web GUI and then power it back on.
>  * The password reverts to the last password reset operation from the web
> GUI.
>
> I've looked at the message logs and the guest password and SSH scripts are
> running normally.  The virtual router is sending the password reset again
> when the service query it.  But this only occurs after the VM is powered
> off then on again.  I expected the password to remain as the manually set
> password but it always reverts to the last password reset operation when
> the VM is powered off then on again.
>

Check the logs on the relevant VR. If the password script does not
successfully send the acknowledgment that the password was saved it won't
get removed from the VR and be set on subsequent reboots.

Don't recall at the moment if is /var/log/cloud.log, /var/log/syslog, but
just look around.


--
Erik



DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

Re: Cloudstack 4.7 password reset issue.

[Erik Weber]

On Sat, Apr 2, 2016 at 9:04 PM, Richard Klein (RSI) 
wrote:

> First time mailer here, hello everyone.



Hi, and welcome!



> We are implementing CS 4.7 cluster with CentOS7.  All is going well with a
> few bumps that we have resolved.  During testing we are getting an unusual
> behavior from the password reset process.  We have setup the templates
> using the instructions from the documentation including installing the
> guest password and SSH setup scripts.  Below is what happens:
>
>  * Create a VM from template that has the password option checked and the
> root password is set as expected.
>  * Log in with 1st password and then change password of root via the
> console of the VM
>  * Perform a reboot from the VM console and the password stays as set
> manually.
>  * Power off VM, change PW with CS web GUI.  When the VM is powered on the
> password is the new one (as expected).
>  * Log in with new password on the console and then manually change root
> again.
>  * Reboot from the console and the password remains as manually set.
>  * Power off VM via web GUI and then power it back on.
>  * The password reverts to the last password reset operation from the web
> GUI.
>
> I've looked at the message logs and the guest password and SSH scripts are
> running normally.  The virtual router is sending the password reset again
> when the service query it.  But this only occurs after the VM is powered
> off then on again.  I expected the password to remain as the manually set
> password but it always reverts to the last password reset operation when
> the VM is powered off then on again.
>

Check the logs on the relevant VR. If the password script does not
successfully send the acknowledgment that the password was saved it won't
get removed from the VR and be set on subsequent reboots.

Don't recall at the moment if is /var/log/cloud.log, /var/log/syslog, but
just look around.


-- 
Erik