Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Jay Bryant]

Mike,

This effort has taken quite some time and was going to require hard
decisions to be made at some point.   You have been more than patient in
this process and I commend you for that as well as all the communication.

Thank you for continuing to drive this!

Jay
On Mar 24, 2015 10:55 PM, "Monty Taylor"  wrote:

> On 03/24/2015 06:05 PM, Kyle Mestery wrote:
> > On Tue, Mar 24, 2015 at 9:56 AM, Doug Hellmann 
> > wrote:
> >
> >> Excerpts from Mark McClain's message of 2015-03-24 10:25:31 -0400:
> >>>
> >>> Echoing both Thierry and John.  I support Mike’s decision to enforce
> the
> >> requirement. Maintaining drivers in the tree comes with
> responsibilities to
> >> the community and 3rd party CI is one of the them.  Mike enforcing this
> >> requirement was the right action even if a hard one to take.
> >>>
> >>> mark
> >>
> >> Indeed. The deadlines were communicated clearly, and frequently.
> >> Making phone calls to reach out to contributors for issues like
> >> this is exceptional.
> >>
> >> +1000. Enabling contributors is great, but CI systems are an important
> > part of that enablement. I appreciate what Mike has done to drive Cinder
> > towards a quality level for all contributors here. It's a hard stance to
> > take, but saying "No" can sometimes be the right decision.
>
> I'm just going to pile on.
>
> People keep asking us to focus on quality over landing features. It was
> the #1 request from EVERY operator at the recent Ops summit. This is one
> of that facets of doing that. It's hard, and it doesn't always feel good
> to all the parties involved - but it's important.
>
> Thank you, Mike, for sticking to your deadline. OpenStack will be better
> for it.
>
> "If it's not tested, it's broken"
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Monty Taylor]

On 03/24/2015 06:05 PM, Kyle Mestery wrote:
> On Tue, Mar 24, 2015 at 9:56 AM, Doug Hellmann 
> wrote:
> 
>> Excerpts from Mark McClain's message of 2015-03-24 10:25:31 -0400:
>>>
>>> Echoing both Thierry and John.  I support Mike’s decision to enforce the
>> requirement. Maintaining drivers in the tree comes with responsibilities to
>> the community and 3rd party CI is one of the them.  Mike enforcing this
>> requirement was the right action even if a hard one to take.
>>>
>>> mark
>>
>> Indeed. The deadlines were communicated clearly, and frequently.
>> Making phone calls to reach out to contributors for issues like
>> this is exceptional.
>>
>> +1000. Enabling contributors is great, but CI systems are an important
> part of that enablement. I appreciate what Mike has done to drive Cinder
> towards a quality level for all contributors here. It's a hard stance to
> take, but saying "No" can sometimes be the right decision.

I'm just going to pile on.

People keep asking us to focus on quality over landing features. It was
the #1 request from EVERY operator at the recent Ops summit. This is one
of that facets of doing that. It's hard, and it doesn't always feel good
to all the parties involved - but it's important.

Thank you, Mike, for sticking to your deadline. OpenStack will be better
for it.

"If it's not tested, it's broken"

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Kyle Mestery]

On Tue, Mar 24, 2015 at 9:56 AM, Doug Hellmann 
wrote:

> Excerpts from Mark McClain's message of 2015-03-24 10:25:31 -0400:
> >
> > Echoing both Thierry and John.  I support Mike’s decision to enforce the
> requirement. Maintaining drivers in the tree comes with responsibilities to
> the community and 3rd party CI is one of the them.  Mike enforcing this
> requirement was the right action even if a hard one to take.
> >
> > mark
>
> Indeed. The deadlines were communicated clearly, and frequently.
> Making phone calls to reach out to contributors for issues like
> this is exceptional.
>
> +1000. Enabling contributors is great, but CI systems are an important
part of that enablement. I appreciate what Mike has done to drive Cinder
towards a quality level for all contributors here. It's a hard stance to
take, but saying "No" can sometimes be the right decision.

Thanks,
Kyle



> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Alessandro Pilotti]

I also absolutely agree that Mike did a great job on the communication
with the driver maintainers and a lot more, especially in the hectic days
around the K-3 deadline.

Removing any driver lacking CI testing was just the right thing to do, even if
this affected our SMB3 driver. Hopefully this is just temporary as the related
CI is currently under test as well (whose delays are totally unrelated), so I
dont see it as a particularly dramatic decision. It also aligns well with the
policies applied it other major OpenStack projects like Nova or Neutron.

I'd be even in favor of a driver decomposition approach as Neutron did, but
that's another topic.

So, thanks again for all your great work and help!

Alessandro


> On 24 Mar 2015, at 16:41, Joshua Harlow  wrote:
> 
> +10 to mike; I have no doubt this is an uneasy and tough task.
> 
> Thanks mike for pushing this through; given all the challenges and hard work 
> (and likely not fun work) that had to be done.
> 
> I salute u! :)
> 
> -Josh
> 
> Duncan Thomas wrote:
>> On 23 March 2015 at 22:50, Mike Perez > > wrote:
>> 
>> 
>>I've talked to folks at the OpenStack foundation to get feedback on my
>>communication, and was told this was good, and even better than previous
>>communication to controversial changes.
>> 
>>I expected nevertheless people to be angry with me and blame me
>>regardless of
>>my attempts to help people be successful and move the community forward.
>> 
>> 
>> As somebody who has previously attempted to drive 3rd party CI in Cinder
>> forward and completely burnt out on the process, I have to applaud
>> Mike's efforts. We needed a line in the sand to force the issue, or we
>> were never going to get to 100% coverage of drivers, which was what we
>> desperately needed.
>> 
>> For those who've have issues getting CI to be stable, this is a genuine
>> reflection of the stability of Openstack in general, but it is not
>> something we're going to be able to make progress on without CI systems
>> exposing problems. That's the entire point of the 3rd party CI program -
>> we knew there were bugs, stability issues and drivers that just plain
>> didn't work, but we couldn't  do much about it without being able to
>> test changes.
>> 
>> Thanks to Mike for the finally push on this, and to all of the various
>> people in both cinder and infra who've been very active in helping
>> people get their CI running, sometimes in very trying circumstances.
>> 
>> --
>> Duncan Thomas
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Arkady_Kanevsky]

Dell - Internal Use - Confidential
+2 on Mike’s job

From: Duncan Thomas [mailto:duncan.tho...@gmail.com]
Sent: Tuesday, March 24, 2015 3:07 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: 
[cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

On 23 March 2015 at 22:50, Mike Perez 
mailto:thin...@gmail.com>> wrote:

I've talked to folks at the OpenStack foundation to get feedback on my
communication, and was told this was good, and even better than previous
communication to controversial changes.

I expected nevertheless people to be angry with me and blame me regardless of
my attempts to help people be successful and move the community forward.

As somebody who has previously attempted to drive 3rd party CI in Cinder 
forward and completely burnt out on the process, I have to applaud Mike's 
efforts. We needed a line in the sand to force the issue, or we were never 
going to get to 100% coverage of drivers, which was what we desperately needed.

For those who've have issues getting CI to be stable, this is a genuine 
reflection of the stability of Openstack in general, but it is not something 
we're going to be able to make progress on without CI systems exposing 
problems. That's the entire point of the 3rd party CI program - we knew there 
were bugs, stability issues and drivers that just plain didn't work, but we 
couldn't  do much about it without being able to test changes.

Thanks to Mike for the finally push on this, and to all of the various people 
in both cinder and infra who've been very active in helping people get their CI 
running, sometimes in very trying circumstances.

--
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Rochelle Grober]

One other item I’d like to bring up.

While Nova and Neutron are well distributed around the globe and have Core 
Reviewers on IRC in the Asia daytime, some other projects are not so well 
distributed as yet.  A problem I noticed a number of times is that an Asia 
based developer will post to the mailing list to get some attention for  
his/her patch.  This is frowned upon in the community, but when there are few 
to no Core Reviewers in IRC, getting that first core review can be difficult.  
Emailing the PTL is something I’m sure the PTLs would like to limit as they are 
already swamped.  So, how do we get timely first core review of patches in 
areas of the world where Core presence in IRC is slim to none?

I can think of a few options but they don’t seem great:

· A filter for dashboards that flags reviews with multiple +1s and no 
core along with a commitment of the Core team to perform a review within x 
number of days

· A separate mailing list for project review requests

· Somehow queueing requests in the IRC channel so that offline 
developers can easily find review requests when looking at channel logs

· ???

Solving this issue could help not just Third Party developers, but all of 
OpenStack and make the community more inviting to Asian and Australian (and 
maybe European and African) developers.

--Rocky

From: Rochelle Grober [mailto:rochelle.gro...@huawei.com]
Sent: Monday, March 23, 2015 14:51
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: 
[cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

I’d like to suggest that the myriad wiki pages and spreadsheets for Third Party 
CI also be consolidated to a more manageable count.  Just looking for 
maintainers contact, you can find information (often conflicting) in 
Stackalytics, on the ThirdPartyDrivers page, on the Cinder PTL’s google doc and 
who knows where else for the Neutron maintainers.  Even finding which tests to 
run takes linking through a number of Cinder wiki pages.

The teams have done a great job documenting a process that started out as lore, 
but I think the beginning of L would be a great time to revisit and reorganize 
the documentation for clarity, conciseness and single locations (version 
controlled) of critical information.

--Rocky

From: Patrick East [mailto:patrick.e...@purestorage.com]
Sent: Monday, March 23, 2015 14:38
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: 
[cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

On Mon, Mar 23, 2015 at 12:59 PM, Stefano Maffulli 
mailto:stef...@openstack.org>> wrote:
On Mon, 2015-03-23 at 11:43 -0700, Mike Perez wrote:
> We've been talking about CI's for a year. We started talking about CI 
> deadlines
> in August. If you post a driver for Kilo, it was communicated that you're
> required to have a CI by the end of Kilo [1][2][3][4][5][6][7][8]. This
> should've been known by your engineers regardless of when you submitted your
> driver.

Let's work to fix the CI bits for Liberty and beyond. I have the feeling
that despite your best effort to communicate deadlines, some quite
visible failure has happened.

I think the only failure is on the side of any driver maintainers who did not 
make the deadlines. From my perspective (as one of the driver maintainers who 
did setup a CI system and a developer working on Cinder) this whole process has 
been a success. The test coverage has sky rocketed for Cinder, driver 
maintainers are forced to be a bit more active in the community, and the code 
base (in theory) no longer has volume drivers in tree that we don't know if 
they actually work or not. This is, in my opinion, a huge win for the project.

You've been clear about Cinder's deadlines, I've been trying to add them
also to the weekly newsletter, too.

To the people whose drivers don't have their CI completed in time: what
do you suggest should change so that you won't miss the deadlines in the
future? How should the processes and tool be different so you'll be
successful with your OpenStack-based products?

For anyone who struggled with getting a CI system operational there are 
numerous resources at your disposal (all of which have been advertised in 
Cinder meetings and the #openstack-cinder IRC channel). There are three 
meetings every week where you can get help setting them up [1]. There are a few 
different Cinder developers who have set up their own CI systems and shared 
code/instructions [2][3]. I have seen those same devs supporting them via IRC 
and have enabled several other companies to successfully use their tools. 
Between these resources I don't think anyone who has actually showed up at the 
meetings, asked for help, a

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Joshua Harlow]

+10 to mike; I have no doubt this is an uneasy and tough task.

Thanks mike for pushing this through; given all the challenges and hard 
work (and likely not fun work) that had to be done.


I salute u! :)

-Josh

Duncan Thomas wrote:

On 23 March 2015 at 22:50, Mike Perez mailto:thin...@gmail.com>> wrote:


I've talked to folks at the OpenStack foundation to get feedback on my
communication, and was told this was good, and even better than previous
communication to controversial changes.

I expected nevertheless people to be angry with me and blame me
regardless of
my attempts to help people be successful and move the community forward.


As somebody who has previously attempted to drive 3rd party CI in Cinder
forward and completely burnt out on the process, I have to applaud
Mike's efforts. We needed a line in the sand to force the issue, or we
were never going to get to 100% coverage of drivers, which was what we
desperately needed.

For those who've have issues getting CI to be stable, this is a genuine
reflection of the stability of Openstack in general, but it is not
something we're going to be able to make progress on without CI systems
exposing problems. That's the entire point of the 3rd party CI program -
we knew there were bugs, stability issues and drivers that just plain
didn't work, but we couldn't  do much about it without being able to
test changes.

Thanks to Mike for the finally push on this, and to all of the various
people in both cinder and infra who've been very active in helping
people get their CI running, sometimes in very trying circumstances.

--
Duncan Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Doug Hellmann]

Excerpts from Mark McClain's message of 2015-03-24 10:25:31 -0400:
> 
> Echoing both Thierry and John.  I support Mike’s decision to enforce the 
> requirement. Maintaining drivers in the tree comes with responsibilities to 
> the community and 3rd party CI is one of the them.  Mike enforcing this 
> requirement was the right action even if a hard one to take.
> 
> mark

Indeed. The deadlines were communicated clearly, and frequently.
Making phone calls to reach out to contributors for issues like
this is exceptional.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Mark McClain]

> On Mar 24, 2015, at 9:30 AM, John Griffith  wrote:
> 
> 
> 
> On Tue, Mar 24, 2015 at 4:52 AM, Thierry Carrez  > wrote:
> Walter A. Boring IV wrote:
> > Thanks Mike for all of your efforts on this,
> 
> +1
> 
> I think Mike checked all the possible boxes to give fair warning to
> driver owners.
> 
> It's hard to say "no" in the name of quality. It's so much easier to
> just say "yes" and avoid all the hatemail and the pressure.
> 
> Mike did the right thing, he did it the right way, and he needs all the
> support the rest of our community can give him.
> 
> --
> Thierry Carrez (ttx)
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 
> 
> ​Just adding my support to the very hard thing that Mike is doing here.  As 
> mentioned discussions and warnings have gone on ad nauseum over the last 
> year.  I may not agree with some of the wording or depictions, ​and I 
> certainly am empathetic here; but the fact is this has been a year long 
> process that was communicated, discussed and help provided.  NOTE this 
> started at the summit in Atlanta!!!
> 
> CI can be hard, the work of a lot of people in Cinder, the Infra team and 
> others have made it a lot easier.  People have also spent countless hours 
> writing code for this, setting up their own systems and helping others out 
> via IRC and even a dedicated weekly meeting as well as a time slot every week 
> in Cinders meeting.  
> 
> If the reasons were different than "my data center went offline" or "I can't 
> host a public web server" I might have a different opinion.  But I have a 
> really hard time with this sort of thing coming from companies the size and 
> scale of Microsoft, NetApp and Oracle.
> 
> Anyway, I do feel bad but not as bad as I'd feel for everybody that worked 
> their butts off on this whole topic for the last year if we turn around and 
> punt it again.
> 
> John
> 

Echoing both Thierry and John.  I support Mike’s decision to enforce the 
requirement. Maintaining drivers in the tree comes with responsibilities to the 
community and 3rd party CI is one of the them.  Mike enforcing this requirement 
was the right action even if a hard one to take.

mark__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[John Griffith]

On Tue, Mar 24, 2015 at 4:52 AM, Thierry Carrez 
wrote:

> Walter A. Boring IV wrote:
> > Thanks Mike for all of your efforts on this,
>
> +1
>
> I think Mike checked all the possible boxes to give fair warning to
> driver owners.
>
> It's hard to say "no" in the name of quality. It's so much easier to
> just say "yes" and avoid all the hatemail and the pressure.
>
> Mike did the right thing, he did it the right way, and he needs all the
> support the rest of our community can give him.
>
> --
> Thierry Carrez (ttx)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

​Just adding my support to the very hard thing that Mike is doing here.  As
mentioned discussions and warnings have gone on ad nauseum over the last
year.  I may not agree with some of the wording or depictions, ​and I
certainly am empathetic here; but the fact is this has been a year long
process that was communicated, discussed and help provided.  NOTE this
started at the summit in Atlanta!!!

CI can be hard, the work of a lot of people in Cinder, the Infra team and
others have made it a lot easier.  People have also spent countless hours
writing code for this, setting up their own systems and helping others out
via IRC and even a dedicated weekly meeting as well as a time slot every
week in Cinders meeting.

If the reasons were different than "my data center went offline" or "I
can't host a public web server" I might have a different opinion.  But I
have a really hard time with this sort of thing coming from companies the
size and scale of Microsoft, NetApp and Oracle.

Anyway, I do feel bad but not as bad as I'd feel for everybody that worked
their butts off on this whole topic for the last year if we turn around and
punt it again.

John
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Thierry Carrez]

Walter A. Boring IV wrote:
> Thanks Mike for all of your efforts on this,

+1

I think Mike checked all the possible boxes to give fair warning to
driver owners.

It's hard to say "no" in the name of quality. It's so much easier to
just say "yes" and avoid all the hatemail and the pressure.

Mike did the right thing, he did it the right way, and he needs all the
support the rest of our community can give him.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Deepak Shetty]

On Tue, Mar 24, 2015 at 2:51 AM, Walter A. Boring IV 
wrote:

> On 03/23/2015 01:50 PM, Mike Perez wrote:
>
>> On 12:59 Mon 23 Mar , Stefano Maffulli wrote:
>>
>>> On Mon, 2015-03-23 at 11:43 -0700, Mike Perez wrote:
>>>
 We've been talking about CI's for a year. We started talking about CI
 deadlines
 in August. If you post a driver for Kilo, it was communicated that
 you're
 required to have a CI by the end of Kilo [1][2][3][4][5][6][7][8]. This
 should've been known by your engineers regardless of when you submitted
 your
 driver.

>>> Let's work to fix the CI bits for Liberty and beyond. I have the feeling
>>> that despite your best effort to communicate deadlines, some quite
>>> visible failure has happened.
>>>
>>> You've been clear about Cinder's deadlines, I've been trying to add them
>>> also to the weekly newsletter, too.
>>>
>>> To the people whose drivers don't have their CI completed in time: what
>>> do you suggest should change so that you won't miss the deadlines in the
>>> future? How should the processes and tool be different so you'll be
>>> successful with your OpenStack-based products?
>>>
>> Just to be clear, here's all the communication attempts made to vendors:
>>
>> 1) Talks during the design summit and the meetup on Friday at the summit.
>>
>> 2) Discussions at the Cinder midcycle meetups in Fort Collins and Austin.
>>
>> 4) Individual emails to driver maintainers. This includes anyone else who
>> has
>> worked on the driver file according to the git logs.
>>
>> 5) Reminders on the mailing list.
>>
>> 6) Reminders on IRC and Cinder IRC meetings every week.
>>
>> 7) If you submitted a new driver in Kilo, you had the annoying reminder
>> from
>> reviewers that your driver needs to have a CI by Kilo.
>>
>> And lastly I have made phone calls to companies that have shown zero
>> responses
>> to my emails or giving me updates. This is very difficult with larger
>> companies because you're redirected from one person to another of who
>> their
>> "OpenStack person" is.  I've left reminders on given voice mail
>> extensions.
>>
>> I've talked to folks at the OpenStack foundation to get feedback on my
>> communication, and was told this was good, and even better than previous
>> communication to controversial changes.
>>
>> I expected nevertheless people to be angry with me and blame me
>> regardless of
>> my attempts to help people be successful and move the community forward.
>>
>>  I completely agree here Mike.   The Cinder cores, PTL, and the rest of
> the
> community have been talking about getting CI as a requirement for quite
> some time now.
> It's really not the fault of the Cinder PTL, or core members, that your
> driver got pulled from the Kilo
> release, because you had issues getting your CI up and stable in the
> required time frame.
> Mike made every possible attempt to let folks know, up front, that the
> deadline was going to happen.
>
> Getting CI in place is critical for the stability of Cinder in general.
>  We have already benefited from
> having 3rd Party CI in place.  It wasn't but a few weeks ago that a change
> that was submitted actually
> broke the HP drivers.   The CI we had in place discovered it, and brought
> it to the surface.   Without
> having that CI in place for our drivers, we would be in a bad spot now.


+1, we (GlusterFS) too discovered issues with live snapshot (being one of
the very few that uses it in cinder)
tests failing as part of CI and we fixed it [1]

[1]: https://review.openstack.org/#/c/156940/

thanx,
deepak
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Duncan Thomas]

On 23 March 2015 at 22:50, Mike Perez  wrote:

>
> I've talked to folks at the OpenStack foundation to get feedback on my
> communication, and was told this was good, and even better than previous
> communication to controversial changes.
>
> I expected nevertheless people to be angry with me and blame me regardless
> of
> my attempts to help people be successful and move the community forward.
>

As somebody who has previously attempted to drive 3rd party CI in Cinder
forward and completely burnt out on the process, I have to applaud Mike's
efforts. We needed a line in the sand to force the issue, or we were never
going to get to 100% coverage of drivers, which was what we desperately
needed.

For those who've have issues getting CI to be stable, this is a genuine
reflection of the stability of Openstack in general, but it is not
something we're going to be able to make progress on without CI systems
exposing problems. That's the entire point of the 3rd party CI program - we
knew there were bugs, stability issues and drivers that just plain didn't
work, but we couldn't  do much about it without being able to test changes.

Thanks to Mike for the finally push on this, and to all of the various
people in both cinder and infra who've been very active in helping people
get their CI running, sometimes in very trying circumstances.

-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Avishay Traeger]

On Mon, Mar 23, 2015 at 9:23 PM, Anita Kuno  wrote:

> I'm really disappointed that there hasn't been more support for Mike in
> this process. I can see how everyone thought I was the problem last year
> when I had to endure this kind of treatment in Neutron, but I would
> think after seeing the exact same kind of behaviour a second time folks
> might be starting to see the pattern.
>

This CI is in my opinion one of the most important undertakings in Cinder
to date.  Cinder is basically an abstraction over a set of drivers, so
testing those drivers is very important.  This is especially true for those
that deploy OpenStack at various customer sites, each with different
hardware, and up to now just had to pray that things worked.

The discussion about the CI has been going on forever.  Mike brought it up
so many times in every forum possible and did a great job with this
difficult task.  While I understand that setting this up is not a simple
task, I think there was enough time.  We have been discussing this CI
forever, and if action is not taken now, it will never happen.

This is not the end of the world for drivers that are removed.  Some
drivers are already hosted in their own github repos as well as in
Cinder's, so vendors can go with that route.  Or maybe there will be an
exception made to allow backports for removed drivers (I'm not sure this is
a good idea).

Anyway, I'm very happy to finally have a release where I know that all
drivers are more-or-less working.  Kudos to Mike, and to all of the Cinder
folks that pioneered the effort and provided support to those that followed.


-- 
*Avishay Traeger*
*Storage R&D*

Mobile: +972 54 447 1475
E-mail: avis...@stratoscale.com



Web  | Blog 
 | Twitter  | Google+

 | Linkedin 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Alex Meade]

As an Engineer running the NetApp CI, I thought it would be a good time to
chime in here. While I have many opinions around this whole process, I will
try my best to avoid any judgement and minimize ratholes.

Over the past year, we have implemented a scalable CI system that is now
running tests against 5 NetApp drivers for every patch (including stable
branches). We have already prevented numerous bugs, some that would
completely break OpenStack/Netapp integrations and other times we caught
issues with the gate before it had time to break. We promptly worked with
the code contributors in each of those cases. We have run over 50k test
runs in total.

Now I realize *those* CI tests have nothing to do with the FibreChannel
drivers specifically, however, it took significant resources to get where
we are now and CI has been our top priority. In the case of FC, we do test
it regularly but atm we only have one FC capable server and 2 FC capable
storage controllers. We have been working diligently with IT to acquire
more for CI use but as most of you know, FC gear is not cheap and IT can
take time. I would agree that we haven’t been panicking and calling IT
every day at 8am under the belief that the community was aware of our
situation and was ok with these drivers taking a bit longer. I might add
that the FC drivers are just a wrapper around our iscsi drivers (the only
difference being the zoning decorator).

Now enough about NetApp, I wish folks would consider their perspective on
the situation. It’s a huge ask to implement a CI system that tests every
patch and not everyone has unlimited resources. Third party CI systems
should be a huge deal for the third party, they should care way more than
Cinder core. Although I understand Cinder not wanting deployers to attempt
to use broken code in their project, I am certain a vendor does not want
broker integration.

Given the nights and weekends I have spent on third party CI, I would
appreciate some empathy on the matter. I’m sure there are plenty of other
folks that would agree. Please ask me any questions out right and I will
give you an answer. I may have been confused but I was under the impression
that NetApp FC had an exception to the deadline given the many
conversations that had occurred.

-Alex

On Mon, Mar 23, 2015 at 6:30 PM, Mike Perez  wrote:

> On 21:51 Mon 23 Mar , Rochelle Grober wrote:
> > I’d like to suggest that the myriad wiki pages and spreadsheets for Third
> > Party CI also be consolidated to a more manageable count.  Just looking
> for
> > maintainers contact, you can find information (often conflicting) in
> > Stackalytics, on the ThirdPartyDrivers page, on the Cinder PTL’s google
> doc
> > and who knows where else for the Neutron maintainers.  Even finding which
> > tests to run takes linking through a number of Cinder wiki pages.
> >
> > The teams have done a great job documenting a process that started out as
> > lore, but I think the beginning of L would be a great time to revisit and
> > reorganize the documentation for clarity, conciseness and single
> locations
> > (version controlled) of critical information.
>
> More than happy to update my doc. My doc was purely for me to expose
> a non-editable doc of what I was seeing since I made myself the point of
> contact for Cinder CI's. My spreadsheet came before the thirdparty CI
> maintainer wiki page had any useful information on it. I got the contacts
> from
> the git logs of the people who actually worked on the drivers. I also was
> dealing with cases where a vendor hired an outside company to do their
> driver,
> which made things difficult for contact.
>
> The one wiki page people should pay attention to is the Cinder Third Party
> wiki
> page which now has a link to the status page:
>
> https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers
>
> --
> Mike Perez
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Mike Perez]

On 21:51 Mon 23 Mar , Rochelle Grober wrote:
> I’d like to suggest that the myriad wiki pages and spreadsheets for Third
> Party CI also be consolidated to a more manageable count.  Just looking for
> maintainers contact, you can find information (often conflicting) in
> Stackalytics, on the ThirdPartyDrivers page, on the Cinder PTL’s google doc
> and who knows where else for the Neutron maintainers.  Even finding which
> tests to run takes linking through a number of Cinder wiki pages.
> 
> The teams have done a great job documenting a process that started out as
> lore, but I think the beginning of L would be a great time to revisit and
> reorganize the documentation for clarity, conciseness and single locations
> (version controlled) of critical information.

More than happy to update my doc. My doc was purely for me to expose
a non-editable doc of what I was seeing since I made myself the point of
contact for Cinder CI's. My spreadsheet came before the thirdparty CI
maintainer wiki page had any useful information on it. I got the contacts from
the git logs of the people who actually worked on the drivers. I also was
dealing with cases where a vendor hired an outside company to do their driver,
which made things difficult for contact.

The one wiki page people should pay attention to is the Cinder Third Party wiki
page which now has a link to the status page: 

https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Anita Kuno]

On 03/23/2015 05:51 PM, Rochelle Grober wrote:
> I’d like to suggest that the myriad wiki pages and spreadsheets for Third 
> Party CI also be consolidated to a more manageable count.  Just looking for 
> maintainers contact, you can find information (often conflicting) in 
> Stackalytics, on the ThirdPartyDrivers page, on the Cinder PTL’s google doc 
> and who knows where else for the Neutron maintainers.  Even finding which 
> tests to run takes linking through a number of Cinder wiki pages.
> 
> The teams have done a great job documenting a process that started out as 
> lore, but I think the beginning of L would be a great time to revisit and 
> reorganize the documentation for clarity, conciseness and single locations 
> (version controlled) of critical information.
> 
> --Rocky
Sure. Since that is the first goal of the third-party meetings, everyone
is welcome to attend and assist with this ongoing effort:
https://wiki.openstack.org/wiki/Meetings/ThirdParty#Goals_for_Third_Party_meetings

Thanks Rocky,
Anita.
> 
> From: Patrick East [mailto:patrick.e...@purestorage.com]
> Sent: Monday, March 23, 2015 14:38
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: 
> [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))
> 
> On Mon, Mar 23, 2015 at 12:59 PM, Stefano Maffulli 
> mailto:stef...@openstack.org>> wrote:
> On Mon, 2015-03-23 at 11:43 -0700, Mike Perez wrote:
>> We've been talking about CI's for a year. We started talking about CI 
>> deadlines
>> in August. If you post a driver for Kilo, it was communicated that you're
>> required to have a CI by the end of Kilo [1][2][3][4][5][6][7][8]. This
>> should've been known by your engineers regardless of when you submitted your
>> driver.
> 
> Let's work to fix the CI bits for Liberty and beyond. I have the feeling
> that despite your best effort to communicate deadlines, some quite
> visible failure has happened.
> 
> I think the only failure is on the side of any driver maintainers who did not 
> make the deadlines. From my perspective (as one of the driver maintainers who 
> did setup a CI system and a developer working on Cinder) this whole process 
> has been a success. The test coverage has sky rocketed for Cinder, driver 
> maintainers are forced to be a bit more active in the community, and the code 
> base (in theory) no longer has volume drivers in tree that we don't know if 
> they actually work or not. This is, in my opinion, a huge win for the project.
> 
> You've been clear about Cinder's deadlines, I've been trying to add them
> also to the weekly newsletter, too.
> 
> To the people whose drivers don't have their CI completed in time: what
> do you suggest should change so that you won't miss the deadlines in the
> future? How should the processes and tool be different so you'll be
> successful with your OpenStack-based products?
> 
> For anyone who struggled with getting a CI system operational there are 
> numerous resources at your disposal (all of which have been advertised in 
> Cinder meetings and the #openstack-cinder IRC channel). There are three 
> meetings every week where you can get help setting them up [1]. There are a 
> few different Cinder developers who have set up their own CI systems and 
> shared code/instructions [2][3]. I have seen those same devs supporting them 
> via IRC and have enabled several other companies to successfully use their 
> tools. Between these resources I don't think anyone who has actually showed 
> up at the meetings, asked for help, and make a good faith effort to keep 
> everyone in the loop and show progress failed to get their system online and 
> keep their driver in Cinder its not a coincidence.
> 
> There are also efforts to provide an easier to use CI system that is shared 
> with the OpenStack infra team [4]. I would recommend anyone who wants to help 
> ease this process for new drivers/maintainers to help contribute to this 
> effort. I think this is going to be the best route forward to ensure people 
> have the tools they need to setup and operate a stable third party ci system.
> 
> 
> 1 - 
> https://wiki.openstack.org/wiki/Meetings/ThirdParty#Weekly_Third_Party_meetings
> 2 - https://github.com/rasselin/os-ext-testing
> 3 - https://github.com/j-griffith/sos-ci
> 4 - https://review.openstack.org/#/c/139745/
> 
> 
> -Patrick
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Rochelle Grober]

I’d like to suggest that the myriad wiki pages and spreadsheets for Third Party 
CI also be consolidated to a more manageable count.  Just looking for 
maintainers contact, you can find information (often conflicting) in 
Stackalytics, on the ThirdPartyDrivers page, on the Cinder PTL’s google doc and 
who knows where else for the Neutron maintainers.  Even finding which tests to 
run takes linking through a number of Cinder wiki pages.

The teams have done a great job documenting a process that started out as lore, 
but I think the beginning of L would be a great time to revisit and reorganize 
the documentation for clarity, conciseness and single locations (version 
controlled) of critical information.

--Rocky

From: Patrick East [mailto:patrick.e...@purestorage.com]
Sent: Monday, March 23, 2015 14:38
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: 
[cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

On Mon, Mar 23, 2015 at 12:59 PM, Stefano Maffulli 
mailto:stef...@openstack.org>> wrote:
On Mon, 2015-03-23 at 11:43 -0700, Mike Perez wrote:
> We've been talking about CI's for a year. We started talking about CI 
> deadlines
> in August. If you post a driver for Kilo, it was communicated that you're
> required to have a CI by the end of Kilo [1][2][3][4][5][6][7][8]. This
> should've been known by your engineers regardless of when you submitted your
> driver.

Let's work to fix the CI bits for Liberty and beyond. I have the feeling
that despite your best effort to communicate deadlines, some quite
visible failure has happened.

I think the only failure is on the side of any driver maintainers who did not 
make the deadlines. From my perspective (as one of the driver maintainers who 
did setup a CI system and a developer working on Cinder) this whole process has 
been a success. The test coverage has sky rocketed for Cinder, driver 
maintainers are forced to be a bit more active in the community, and the code 
base (in theory) no longer has volume drivers in tree that we don't know if 
they actually work or not. This is, in my opinion, a huge win for the project.

You've been clear about Cinder's deadlines, I've been trying to add them
also to the weekly newsletter, too.

To the people whose drivers don't have their CI completed in time: what
do you suggest should change so that you won't miss the deadlines in the
future? How should the processes and tool be different so you'll be
successful with your OpenStack-based products?

For anyone who struggled with getting a CI system operational there are 
numerous resources at your disposal (all of which have been advertised in 
Cinder meetings and the #openstack-cinder IRC channel). There are three 
meetings every week where you can get help setting them up [1]. There are a few 
different Cinder developers who have set up their own CI systems and shared 
code/instructions [2][3]. I have seen those same devs supporting them via IRC 
and have enabled several other companies to successfully use their tools. 
Between these resources I don't think anyone who has actually showed up at the 
meetings, asked for help, and make a good faith effort to keep everyone in the 
loop and show progress failed to get their system online and keep their driver 
in Cinder its not a coincidence.

There are also efforts to provide an easier to use CI system that is shared 
with the OpenStack infra team [4]. I would recommend anyone who wants to help 
ease this process for new drivers/maintainers to help contribute to this 
effort. I think this is going to be the best route forward to ensure people 
have the tools they need to setup and operate a stable third party ci system.


1 - 
https://wiki.openstack.org/wiki/Meetings/ThirdParty#Weekly_Third_Party_meetings
2 - https://github.com/rasselin/os-ext-testing
3 - https://github.com/j-griffith/sos-ci
4 - https://review.openstack.org/#/c/139745/


-Patrick

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next?

[Anita Kuno]

On 03/23/2015 05:21 PM, Stefano Maffulli wrote:
> On Mon, 2015-03-23 at 16:23 -0400, Anita Kuno wrote:
>> Some folks just will not respect other people's time. To pretend
>> otherwise is a huge dis-service to folks trying their hardest to
>> support those worthy of the support.
> 
> This may be true in general but I have yet to be convinced that this is
> the case.
That's fine. This isn't a fun space and many folks I have talked to have
purposely stayed out of it so as to not be involved. I think some folks
refer to it as a tar pit.

That's fine if you aren't convinced, I am long ago.

> My impression is that the situation is more complex than just
> lack of respect for other people's time.
Sure since that is what folks would like you to believe so they can
continue to not do the work.

> 
>> I'm really disappointed that there hasn't been more support for Mike
>> in this process.
> 
> I've been supporting Mike in this decision and have not seen any element
> that makes me believe that Mike's position is not supported. Where have
> you seen lack of support?
Well I'll tell you, having to deal with third party ci folks day after
day gets mighty lonely as noone else will touch it with a ten foot pole.
So that is nice to see you post to the mailing list that you support
Mike, thank you.

> 
> Going back to the point, in Paris I have been told that deadlines are
> hard to know. We've changed and this time you and Mike and cinder core
> and the whole Third-Party-CI team have been doing a lot more than in the
> past to keep people informed about deadlines, offering help,
> documentation, mentoring, etc. 
> 
> And still, that was not enough for some major members of OpenStack who
> failed to meet the deadlines.
I agree with you on this point, despite our best and continual effort
some folks fail to be able to meet expectations here. Is it possible
they don't feel they have to?

> That's why I am *not* asking you or Mike
> what you can do better. 
> 
> I'm asking the folks at NetApp, Huawei and others to tell us what they
> think should be done differently.
If you are asking what they think they should do differently, then I too
am keen to hear what they have to say for themselves.

Thanks stef,
Anita.
> 
> /stef
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Patrick East]

On Mon, Mar 23, 2015 at 12:59 PM, Stefano Maffulli 
wrote:

> On Mon, 2015-03-23 at 11:43 -0700, Mike Perez wrote:
> > We've been talking about CI's for a year. We started talking about CI
> deadlines
> > in August. If you post a driver for Kilo, it was communicated that you're
> > required to have a CI by the end of Kilo [1][2][3][4][5][6][7][8]. This
> > should've been known by your engineers regardless of when you submitted
> your
> > driver.
>
> Let's work to fix the CI bits for Liberty and beyond. I have the feeling
> that despite your best effort to communicate deadlines, some quite
> visible failure has happened.
>

I think the only failure is on the side of any driver maintainers who did
not make the deadlines. From my perspective (as one of the driver
maintainers who did setup a CI system and a developer working on Cinder)
this whole process has been a success. The test coverage has sky rocketed
for Cinder, driver maintainers are forced to be a bit more active in the
community, and the code base (in theory) no longer has volume drivers in
tree that we don't know if they actually work or not. This is, in my
opinion, a huge win for the project.


> You've been clear about Cinder's deadlines, I've been trying to add them
> also to the weekly newsletter, too.
>
> To the people whose drivers don't have their CI completed in time: what
> do you suggest should change so that you won't miss the deadlines in the
> future? How should the processes and tool be different so you'll be
> successful with your OpenStack-based products?
>
>
For anyone who struggled with getting a CI system operational there are
numerous resources at your disposal (all of which have been advertised in
Cinder meetings and the #openstack-cinder IRC channel). There are three
meetings every week where you can get help setting them up [1]. There are a
few different Cinder developers who have set up their own CI systems and
shared code/instructions [2][3]. I have seen those same devs supporting
them via IRC and have enabled several other companies to successfully use
their tools. Between these resources I don't think anyone who has actually
showed up at the meetings, asked for help, and make a good faith effort to
keep everyone in the loop and show progress failed to get their system
online and keep their driver in Cinder its not a coincidence.

There are also efforts to provide an easier to use CI system that is shared
with the OpenStack infra team [4]. I would recommend anyone who wants to
help ease this process for new drivers/maintainers to help contribute to
this effort. I think this is going to be the best route forward to ensure
people have the tools they need to setup and operate a stable third party
ci system.


1 -
https://wiki.openstack.org/wiki/Meetings/ThirdParty#Weekly_Third_Party_meetings
2 - https://github.com/rasselin/os-ext-testing
3 - https://github.com/j-griffith/sos-ci
4 - https://review.openstack.org/#/c/139745/


-Patrick
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next?

[Stefano Maffulli]

On Mon, 2015-03-23 at 16:23 -0400, Anita Kuno wrote:
> Some folks just will not respect other people's time. To pretend
> otherwise is a huge dis-service to folks trying their hardest to
> support those worthy of the support.

This may be true in general but I have yet to be convinced that this is
the case. My impression is that the situation is more complex than just
lack of respect for other people's time.

> I'm really disappointed that there hasn't been more support for Mike
> in this process.

I've been supporting Mike in this decision and have not seen any element
that makes me believe that Mike's position is not supported. Where have
you seen lack of support?

Going back to the point, in Paris I have been told that deadlines are
hard to know. We've changed and this time you and Mike and cinder core
and the whole Third-Party-CI team have been doing a lot more than in the
past to keep people informed about deadlines, offering help,
documentation, mentoring, etc. 

And still, that was not enough for some major members of OpenStack who
failed to meet the deadlines.  That's why I am *not* asking you or Mike
what you can do better. 

I'm asking the folks at NetApp, Huawei and others to tell us what they
think should be done differently.

/stef


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Walter A. Boring IV]

On 03/23/2015 01:50 PM, Mike Perez wrote:

On 12:59 Mon 23 Mar , Stefano Maffulli wrote:

On Mon, 2015-03-23 at 11:43 -0700, Mike Perez wrote:

We've been talking about CI's for a year. We started talking about CI deadlines
in August. If you post a driver for Kilo, it was communicated that you're
required to have a CI by the end of Kilo [1][2][3][4][5][6][7][8]. This
should've been known by your engineers regardless of when you submitted your
driver.

Let's work to fix the CI bits for Liberty and beyond. I have the feeling
that despite your best effort to communicate deadlines, some quite
visible failure has happened.

You've been clear about Cinder's deadlines, I've been trying to add them
also to the weekly newsletter, too.

To the people whose drivers don't have their CI completed in time: what
do you suggest should change so that you won't miss the deadlines in the
future? How should the processes and tool be different so you'll be
successful with your OpenStack-based products?

Just to be clear, here's all the communication attempts made to vendors:

1) Talks during the design summit and the meetup on Friday at the summit.

2) Discussions at the Cinder midcycle meetups in Fort Collins and Austin.

4) Individual emails to driver maintainers. This includes anyone else who has
worked on the driver file according to the git logs.

5) Reminders on the mailing list.

6) Reminders on IRC and Cinder IRC meetings every week.

7) If you submitted a new driver in Kilo, you had the annoying reminder from
reviewers that your driver needs to have a CI by Kilo.

And lastly I have made phone calls to companies that have shown zero responses
to my emails or giving me updates. This is very difficult with larger
companies because you're redirected from one person to another of who their
"OpenStack person" is.  I've left reminders on given voice mail extensions.

I've talked to folks at the OpenStack foundation to get feedback on my
communication, and was told this was good, and even better than previous
communication to controversial changes.

I expected nevertheless people to be angry with me and blame me regardless of
my attempts to help people be successful and move the community forward.


I completely agree here Mike.   The Cinder cores, PTL, and the rest of the
community have been talking about getting CI as a requirement for quite 
some time now.
It's really not the fault of the Cinder PTL, or core members, that your 
driver got pulled from the Kilo
release, because you had issues getting your CI up and stable in the 
required time frame.
Mike made every possible attempt to let folks know, up front, that the 
deadline was going to happen.


Getting CI in place is critical for the stability of Cinder in general. 
  We have already benefited from
having 3rd Party CI in place.  It wasn't but a few weeks ago that a 
change that was submitted actually
broke the HP drivers.   The CI we had in place discovered it, and 
brought it to the surface.   Without
having that CI in place for our drivers, we would be in a bad spot now. 
  In other words,  it should be a top
priority for vendors to get CI in place, if for the selfish reason of 
protecting their code!!!


That being said, I look forward to seeing folks submit their drivers 
back in the early L time
frame.   If my driver got pulled for K, It would be my top priority to 
get CI working NOW,

and the day L opens up, I have my driver patch up, with CI reporting.

Thanks Mike for all of your efforts on this,
Walt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Mike Perez]

On 12:59 Mon 23 Mar , Stefano Maffulli wrote:
> On Mon, 2015-03-23 at 11:43 -0700, Mike Perez wrote:
> > We've been talking about CI's for a year. We started talking about CI 
> > deadlines
> > in August. If you post a driver for Kilo, it was communicated that you're
> > required to have a CI by the end of Kilo [1][2][3][4][5][6][7][8]. This
> > should've been known by your engineers regardless of when you submitted your
> > driver.
> 
> Let's work to fix the CI bits for Liberty and beyond. I have the feeling
> that despite your best effort to communicate deadlines, some quite
> visible failure has happened. 
> 
> You've been clear about Cinder's deadlines, I've been trying to add them
> also to the weekly newsletter, too. 
> 
> To the people whose drivers don't have their CI completed in time: what
> do you suggest should change so that you won't miss the deadlines in the
> future? How should the processes and tool be different so you'll be
> successful with your OpenStack-based products?

Just to be clear, here's all the communication attempts made to vendors:

1) Talks during the design summit and the meetup on Friday at the summit.

2) Discussions at the Cinder midcycle meetups in Fort Collins and Austin.

4) Individual emails to driver maintainers. This includes anyone else who has
   worked on the driver file according to the git logs.

5) Reminders on the mailing list.

6) Reminders on IRC and Cinder IRC meetings every week.

7) If you submitted a new driver in Kilo, you had the annoying reminder from
   reviewers that your driver needs to have a CI by Kilo.

And lastly I have made phone calls to companies that have shown zero responses
to my emails or giving me updates. This is very difficult with larger
companies because you're redirected from one person to another of who their
"OpenStack person" is.  I've left reminders on given voice mail extensions.

I've talked to folks at the OpenStack foundation to get feedback on my
communication, and was told this was good, and even better than previous
communication to controversial changes.

I expected nevertheless people to be angry with me and blame me regardless of
my attempts to help people be successful and move the community forward.

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Anita Kuno]

On 03/23/2015 03:59 PM, Stefano Maffulli wrote:
> On Mon, 2015-03-23 at 11:43 -0700, Mike Perez wrote:
>> We've been talking about CI's for a year. We started talking about CI 
>> deadlines
>> in August. If you post a driver for Kilo, it was communicated that you're
>> required to have a CI by the end of Kilo [1][2][3][4][5][6][7][8]. This
>> should've been known by your engineers regardless of when you submitted your
>> driver.
> 
> Let's work to fix the CI bits for Liberty and beyond. I have the feeling
> that despite your best effort to communicate deadlines, some quite
> visible failure has happened.
The thing that quite visibly that happened is that some folks don't
recognize that their actions affect others and take responsibility for
those actions. I have learned much to my dismay that there is a certain
percentage of a group that will not follow the intent of the group
despite any amount of support given to them.
> 
> You've been clear about Cinder's deadlines, I've been trying to add them
> also to the weekly newsletter, too. 
> 
> To the people whose drivers don't have their CI completed in time: what
> do you suggest should change so that you won't miss the deadlines in the
> future?
Asking this question dismisses all the hard work that both Mike and
myself and many others including some very committed operators have put
into this space.

Some folks just will not respect other people's time. To pretend
otherwise is a huge dis-service to folks trying their hardest to support
those worthy of the support.

I'm really disappointed that there hasn't been more support for Mike in
this process. I can see how everyone thought I was the problem last year
when I had to endure this kind of treatment in Neutron, but I would
think after seeing the exact same kind of behaviour a second time folks
might be starting to see the pattern.

Anita.

> How should the processes and tool be different so you'll be
> successful with your OpenStack-based products?
> 
> Cheers,
> stef
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Cinder Third-Party CI: what next? (was Re: [cinder] Request exemption for removal of NetApp FC drivers (no voting CI))

[Stefano Maffulli]

On Mon, 2015-03-23 at 11:43 -0700, Mike Perez wrote:
> We've been talking about CI's for a year. We started talking about CI 
> deadlines
> in August. If you post a driver for Kilo, it was communicated that you're
> required to have a CI by the end of Kilo [1][2][3][4][5][6][7][8]. This
> should've been known by your engineers regardless of when you submitted your
> driver.

Let's work to fix the CI bits for Liberty and beyond. I have the feeling
that despite your best effort to communicate deadlines, some quite
visible failure has happened. 

You've been clear about Cinder's deadlines, I've been trying to add them
also to the weekly newsletter, too. 

To the people whose drivers don't have their CI completed in time: what
do you suggest should change so that you won't miss the deadlines in the
future? How should the processes and tool be different so you'll be
successful with your OpenStack-based products?

Cheers,
stef


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev