Re: [openstack-dev] Future Trove development

[Amrith Kumar]

Luke, please see inline below.

--
Amrith Kumar
mailto:amrith.ku...@gmail.com


>From: Luke Browning [mailto:lukebrown...@us.ibm.com] 
>Sent: Thursday, August 31, 2017 1:00 PM
>To: openstack-dev@lists.openstack.org
>Subject: Re: [openstack-dev] Future Trove development
>
>Amrith, 
>
>See answers to your questions below.
>
>Cheers,
>Luke  
>
>>>
>>> I understand that the Trove project will be going into maintenance mode
>>> soon. I have a number of Trove related enhancements and bug fixes that I
>>> would like to submit to the community, but I don't know if they would be
>>> considered given that the project is going into maintenance mode. Please
>>> elaborate on what you mean by maintenance mode.
>>>
>>
>> See: https://review.openstack.org/#/c/488947/
>>
>> It is not a foregone conclusion that the project will go into maintenance
>> mode. Thierry and I (for example) are not convinced that this is the right
>> course of action, but if there are no offers to contribute to the project,
>> it may be a decision by default.
>>
>> Do you subscribe to the OpenStack-dev mailing list. Please also see
>> http://openstack.markmail.org/thread/4p6gp263fei4mbru
>>
>> Finally, for a description of what maintenance mode means, please refer:
>> https://governance.openstack.org/tc/reference/tags/status_ma
>> intenance-mode.html
>>
>>>
>>>
>>> Would Trove be updated to support new OpenStack versions?
>>>
>> Would support be provided for Trove gate testing?
>>>
>> Would elements be enhanced to support Xenial and newer versions of
>>> databases?
>>>
>> Is there a time limit associated with maintenance mode? For example, would
>>> it remain in effect for a year or two after the new stuff is released?
>>>
>>> For the four questions above, I direct your attention to the definition
>> of the maintenance-mode tag (https://governance.openstack/.
>> org/tc/reference/tags/status_maintenance-mode.html) and remind you that
>> support for different versions from the community, for gate testing and for
>> elements is dependent on people participating (and their employers allowing
>> them to do this). At this point, I have no one who has stepped up to do
>> this in any significant way. There are a couple of people from China Mobile
>> who want to help but who are largely in the weeds, trying to fix this and
>> that bug but have no idea what Trove is all about. IBM has a core reviewer
>> on the project (Mariam is copied on this email). I am happy that she's
>> able to devote what time she can to the project but absent competent
>> reviewers, whether your changes ever get merged or not remains an
>> interesting question. Since you will be contributing elements and propose
>> to contribute a tool, will you be (or will IBM be) offering to support it?
>>
>>  Let me provide some back ground on my code changes, so you will have a
>> better idea of what might be submitted in a patch.
>>
>>>
>>> I have written a fully automated command that creates a virtual guest
>>> image based on Ubuntu 16.04 containing a user specified version of a
>>> database. The user's selection is validated against an internal list of
>>> databases built into the tool. Once validated, the tool creates the image,
>>> registers it with Glance, and then creates a Trove Datastore definition for
>>> it. This is done in a single command invocation that typically takes about
>>> 25 minutes to complete. For some of the databases, it takes considerably
>>> longer (2 hours for mongodb) as I have to compile source code. The guest
>>> image that is produced is complete from a Trove perspective. It includes
>>> the Cloud specific trove-guestagent.conf file and SSH public keys for the
>>> DBA and OpenStack controller nodes.
>>>
>>> What mechanism does it use to develop the image? is it diskimage-builder
>> or some other?
>
>Yes, diskimage-builder and Trove elements
>
>>
>>
>> Is there something the matter with the existing tool that exists, it
>> already works, it does more than just build an image, and it is already
>> integrated in the gate. Why not use that?
>


I should have said this before (like in the last reply) and I may not have, but 
I am broadly in favor of the approach you proposed (I did look briefly at the 
repository that you provided). The place(s) where I had reservations relate 
primarily to how this new tool would interface/interact with the existing 
Trove. More details about that below.

Re: [openstack-dev] Future Trove development

[Luke Browning]

Amrith, 

See answers to your questions below.

Cheers,
Luke 

>>
>> I understand that the Trove project will be going into maintenance mode
>> soon. I have a number of Trove related enhancements and bug fixes that 
I
>> would like to submit to the community, but I don't know if they would 
be
>> considered given that the project is going into maintenance mode. 
Please
>> elaborate on what you mean by maintenance mode.
>>
>
> ​See: https://review.openstack.org/#/c/488947/
>
> It is not a foregone conclusion that the project will go into 
maintenance
> mode. Thierry and I (for example) are not convinced that this is the 
right
> course of action, but if there are no offers to contribute to the 
project,
> it may be a decision by default.
>
> Do you subscribe to the OpenStack-dev mailing list​. Please also see
> http://openstack.markmail.org/thread/4p6gp263fei4mbru
>
> Finally, for a description of what maintenance mode means, please refer:
> https://governance.openstack.org/tc/reference/tags/status_ma
> intenance-mode.html
>
>>
>>
>> Would Trove be updated to support new OpenStack versions?
>>
> Would support be provided for Trove gate testing?
>>
> Would elements be enhanced to support Xenial and newer versions of
>> databases?
>>
> Is there a time limit associated with maintenance mode? For example, 
would
>> it remain in effect for a year or two after the new stuff is released?
>>
>> ​For the four questions above, I direct your attention to the 
definition
> of the maintenance-mode tag (https://governance.openstack.
> org/tc/reference/tags/status_maintenance-mode.html) and remind you that
> support for different versions from the community, for gate testing and 
for
> elements is dependent on people participating (and their employers 
allowing
> them to do this). At this point, I have no one who has stepped up to do
> this in any significant way. There are a couple of people from China 
Mobile
> who want to help but who are largely in the weeds, trying to fix this 
and
> that bug but have no idea what Trove is all about. IBM has a core 
reviewer
> on the project ​(Mariam is copied on this email). I am happy that she's
> able to devote what time she can to the project but absent competent
> reviewers, whether your changes ever get merged or not remains an
> interesting question. Since you will be contributing elements and 
propose
> to contribute a tool, will you be (or will IBM be) offering to support 
it?
>
>  Let me provide some back ground on my code changes, so you will have a
> better idea of what might be submitted in a patch.
>
>>
>> I have written a fully automated command that creates a virtual guest
>> image based on Ubuntu 16.04 containing a user specified version of a
>> database. The user's selection is validated against an internal list of
>> databases built into the tool. Once validated, the tool creates the 
image,
>> registers it with Glance, and then creates a Trove Datastore definition 
for
>> it. This is done in a single command invocation that typically takes 
about
>> 25 minutes to complete. For some of the databases, it takes 
considerably
>> longer (2 hours for mongodb) as I have to compile source code. The 
guest
>> image that is produced is complete from a Trove perspective. It 
includes
>> the Cloud specific trove-guestagent.conf file and SSH public keys for 
the
>> DBA and OpenStack controller nodes.
>>
>> ​What mechanism does it use to develop the image? is it 
diskimage-builder
> or some other?

Yes, diskimage-builder and Trove elements

>
>
> Is there something the matter with the existing tool that exists, it
> already works, it does more than just build an image, and it is already
> integrated in the gate. Why not use that?​

Is your question why don't we use trovestack?If so, we were looking 
for
something that was more streamlined with fewer dependencies.   The tool
would need to interface with the client's cloud.   Our tool is directed at
customers and IBM lab based services, neither of whom is expected to be
an OpenStack or Trove expert.

Beyond that, there is a difference in how are images our built.   First, 
we need
to control which versions of databases are built as opposed to letting the 
stack
determine that.ppc64le support in the database communities is at 
varying
stages.   In some cases, there are packages and in others there are not. 
In
addition, there are specific versions that we have tested and enhanced 
from
a performance perspective that we would like to include in our offering. 
 
The second part of this is related to the Trove Guest Agent.   We needed 
to
make a few code changes to support different versions and to fix bugs that
we have found.   Anyway, this led to us carrying patches in the tool, 
which is
part of a broader strategy to stage our database deliverables.   We would
like to add support for new databases over time.   To deliver these to our
clients without them having to re-install Trove on the controller side.
Fundamentally, 

Re: [openstack-dev] Future Trove development

[Amrith Kumar]

​The email thread below is an one that I have been having with Luke and
some others at IBM who have been working on Trove.

I feel that these kind(s) of conversations are better had in the mailing
list, and in replying to this email, I let the participants know that
absent any objections, I would like to continue this conversation on the
ML. Hearing nothing in the past two days, I am forwarding this to the ML.

-amrith
​


> On Thu, Aug 17, 2017 at 4:24 PM, Luke Browning 
> wrote:
>
>> Hi Amrith,
>>
>> I understand that the Trove project will be going into maintenance mode
>> soon. I have a number of Trove related enhancements and bug fixes that I
>> would like to submit to the community, but I don't know if they would be
>> considered given that the project is going into maintenance mode. Please
>> elaborate on what you mean by maintenance mode.
>>
>
> ​See: https://review.openstack.org/#/c/488947/
>
> It is not a foregone conclusion that the project will go into maintenance
> mode. Thierry and I (for example) are not convinced that this is the right
> course of action, but if there are no offers to contribute to the project,
> it may be a decision by default.
>
> Do you subscribe to the OpenStack-dev mailing list​. Please also see
> http://openstack.markmail.org/thread/4p6gp263fei4mbru
>
> Finally, for a description of what maintenance mode means, please refer:
> https://governance.openstack.org/tc/reference/tags/status_ma
> intenance-mode.html
>
>>
>>
>> Would Trove be updated to support new OpenStack versions?
>>
> Would support be provided for Trove gate testing?
>>
> Would elements be enhanced to support Xenial and newer versions of
>> databases?
>>
> Is there a time limit associated with maintenance mode? For example, would
>> it remain in effect for a year or two after the new stuff is released?
>>
>> ​For the four questions above, I direct your attention to the definition
> of the maintenance-mode tag (https://governance.openstack.
> org/tc/reference/tags/status_maintenance-mode.html) and remind you that
> support for different versions from the community, for gate testing and for
> elements is dependent on people participating (and their employers allowing
> them to do this). At this point, I have no one who has stepped up to do
> this in any significant way. There are a couple of people from China Mobile
> who want to help but who are largely in the weeds, trying to fix this and
> that bug but have no idea what Trove is all about. IBM has a core reviewer
> on the project ​(Mariam is copied on this email). I am happy that she's
> able to devote what time she can to the project but absent competent
> reviewers, whether your changes ever get merged or not remains an
> interesting question. Since you will be contributing elements and propose
> to contribute a tool, will you be (or will IBM be) offering to support it?
>
>  Let me provide some back ground on my code changes, so you will have a
> better idea of what might be submitted in a patch.
>
>>
>> I have written a fully automated command that creates a virtual guest
>> image based on Ubuntu 16.04 containing a user specified version of a
>> database. The user's selection is validated against an internal list of
>> databases built into the tool. Once validated, the tool creates the image,
>> registers it with Glance, and then creates a Trove Datastore definition for
>> it. This is done in a single command invocation that typically takes about
>> 25 minutes to complete. For some of the databases, it takes considerably
>> longer (2 hours for mongodb) as I have to compile source code. The guest
>> image that is produced is complete from a Trove perspective. It includes
>> the Cloud specific trove-guestagent.conf file and SSH public keys for the
>> DBA and OpenStack controller nodes.
>>
>> ​What mechanism does it use to develop the image? is it diskimage-builder
> or some other?
>
>
> Is there something the matter with the existing tool that exists, it
> already works, it does more than just build an image, and it is already
> integrated in the gate. Why not use that?​
>
> ​
> This approach is nice (having a full guest image with the guest agent and
> all that but as you observe, it takes about 2 hours per build and from a
> developers perspective, this is a pain.
>
> I assume that the tool is therefore for production use cases.
>
> Could the elements that you have developed be used with the existing tool
> or is your tool a complete replacement for the one that already exists?​
>
> ​Is there some reason that this tool was not developed in the open, using
> the normal open development process?
>
> There are ways to accelerate the compiling of source code and such through
> the use of image layers (if you are using DIB). I have elements and code
> that will build most of the trove elements in minutes and I'm waiting to
> see whether there is any point in contributing them to the upstream or not,
> at this point. I could show