[openstack-dev] [OpenStack-Dev] [Nova][Neutron][Horizon][Cinder][Keystone][Glance][Ironic][Swift] Fault Classification Input Request

[Nematollah Bidokhti]

Hi,

Our [Fault-Genes WG] has been working on defining the fault classifications for 
key OpenStack projects in an effort to support OpenStack fault management & 
self-healing.
We have been using machine learning (unsupervised data) as a method to look 
into all bugs and issues submitted by the community and it has been very 
challenging to define the classification completely by the machine.
We have decided to go with supervised data set. In order to do this, we need to 
come up with our training data.

We need your help to generate the training data set. Basically, we only need 2 
or 3 unique fault classifications with a short description and the associated 
mitigations from each member who is familiar with OpenStack design & operation. 
This way we can build a focused library of faults & mitigations for each 
project.
Once this data is accumulated, we will develop our own specific algorithms that 
can be applied to all future OpenStack issues.
Thanks in advance for your support.
 No.

Project

Fault Classification

Description

Root Cause

Mitigation

1











2











3












Below are examples of what a couple of developers in Neutron have provided. I 
am sure there are other types of fault classifications in Neurton that have not 
been captured in this table.


Fault Classification


Root Cause


Mitigation


Network Connectivity Issues


Virtual interface in the VM admin down


Un-shut the virtual interface


Virtual interface does not have IP address via DHCP


Depends on lower level root cause


Virtual network does not have interface to the router


Add virtual network as one of the router interfaces


vNIC port of VM not active (stuck in build)


Depends on lower level root cause


Security group lock in traffic


Fix the security group to allow relevant traffic


Unable to Add Port to Bridge


Libvirtd in Apparmor is blocking


allow Libvirtd profile in Appamor


No Valid Host Found/insufficient hypervisor resources


Compute nodes do not have sufficient resources


free up required compute storage and memory resources on compute node


No Resource


Configuration issues


Change config setting


Authentication/permissions error


Configuration error such as port # or Password


Make sure end points are properly configured


Gateway access not reachable


Use custom keep-alive health-check


Design issue of OpenStack Network node


Out of band health checking mechanism


Security Group Mis-configuration


The security group


Change security rules/Programming the security group


DNS Attack


Implement CERT alerts updates


Network design issue


Network storm


Reduce L2 broadcast domain

Nemat



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Nematollah Bidokhti]

Hi,

I have missed the original email on this subject.
We [Fault Genes WG] have been doing some machine learning analysis on Nova 
bugs/issues from 3 different sources (Launchpad, Stackoverflow, 
ask.openstack.org). We have been able to take all the issues and bring them 
down to 15 clusters.
We have tried to find open source tools that can help us define the fault 
classifications, but have not been able to find any tool.

Therefore, our team have come to the conclusion that we need the support of 
some Nova experts to help define the classifications. I would like to have some 
discussions with Sean and others that have an interest in this area and compare 
notes and see how we can collaborate.

The goal of our WG is to apply the same technique to all key OpenStack projects.

Thanks,
Nemat 

-Original Message-
From: Emilien Macchi [mailto:emil...@redhat.com] 
Sent: Wednesday, July 05, 2017 12:24 PM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [nova] bug triage experimentation

On Fri, Jun 23, 2017 at 9:52 AM, Sean Dague  wrote:
> The Nova bug backlog is just over 800 open bugs, which while 
> historically not terrible, remains too large to be collectively usable 
> to figure out where things stand. We've had a few recent issues where 
> we just happened to discover upgrade bugs filed 4 months ago that 
> needed fixes and backports.
>
> Historically we've tried to just solve the bug backlog with volunteers.
> We've had many a brave person dive into here, and burn out after 4 - 6 
> months. And we're currently without a bug lead. Having done a big 
> giant purge in the past
> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046
> 517.html)
> I know how daunting this all can be.
>
> I don't think that people can currently solve the bug triage problem 
> at the current workload that it creates. We've got to reduce the smart 
> human part of that workload.
>
> But, I think that we can also learn some lessons from what active 
> github projects do.
>
> #1 Bot away bad states
>
> There are known bad states of bugs - In Progress with no open patch, 
> Assigned but not In Progress. We can just bot these away with scripts.
> Even better would be to react immediately on bugs like those, that 
> helps to train folks how to use our workflow. I've got some starter 
> scripts for this up at - https://github.com/sdague/nova-bug-tools
>
> #2 Use tag based workflow
>
> One lesson from github projects, is the github tracker has no workflow.
> Issues are openned or closed. Workflow has to be invented by every 
> team based on a set of tags. Sometimes that's annoying, but often 
> times it's super handy, because it allows the tracker to change 
> workflows and not try to change the meaning of things like "Confirmed 
> vs. Triaged" in your mind.
>
> We can probably tag for information we know we need at lot easier. I'm 
> considering something like
>
> * needs.system-version
> * needs.openstack-version
> * needs.logs
> * needs.subteam-feedback
> * has.system-version
> * has.openstack-version
> * has.reproduce
>
> Some of these a bot can process the text on and tell if that info was 
> provided, and comment how to provide the updated info. Some of this 
> would be human, but with official tags, it would probably help.
>
> #3 machine assisted functional tagging
>
> I'm playing around with some things that might be useful in mapping 
> new bugs into existing functional buckets like: libvirt, volumes, etc. 
> We'll see how useful it ends up being.
>
> #4 reporting on smaller slices
>
> Build some tooling to report on the status and change over time of 
> bugs under various tags. This will help visualize how we are doing
> (hopefully) and where the biggest piles of issues are.
>
> The intent is the normal unit of interaction would be one of these 
> smaller piles. Be they the 76 libvirt bugs, 61 volumes bugs, or 36 
> vmware bugs. It would also highlight the rates of change in these 
> piles, and what's getting attention and what is not.
>
>
> This is going to be kind of an ongoing experiment, but as we currently 
> have no one spear heading bug triage, it seemed like a good time to 
> try this out.
>
> Comments and other suggestions are welcomed. The tooling will have the 
> nova flow in mind, but I'm trying to make it so it takes a project 
> name as params on all the scripts, so anyone can use it. It's a little 
> hack and slash right now to discover what the right patterns are.

I also believe that some of the scripts could be transformed into native 
features of Storyboard where bugs could be auto-triaged periodically without 
human intervention.
Maybe it would convince more OpenStack projects to leave Launchpad and adopt 
Storyboard?
I would certainly one of those and propose such a change for TripleO & related 
projects.

Thanks,

> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> 

Re: [Openstack-operators] looking for feedback on proposals to improve logging

[Nematollah Bidokhti]

Hi,

I have been working on the concept of fault management blueprint to increase 
cloud resiliency. As part of this proposal, info such as logs, KPIs, health 
checks and so on are critical since we are engaging in deep data analysis and 
machine learning.
>> 
>> To ease the data analysis process there must be consistent logs. This comes 
>> with having IDs and severity properties.
>> 
>> Cloud resiliency in general requires fast fault detection, isolation and 
>> recovery. In addition, there are applications such as NFV that is sensitive 
>> to fast fault detection and recovery. One approach is to have meaningful 
>> logs where by parsing the data we can make real time fault management 
>> decisions.
>> 
>> Similar to interrupts, I would like us to have logging hierarchy which can 
>> help an automated fault management system to take accurate and appropriate 
>> actions. The format of the logs is important since it will ease the ML 
>> analysis later in the process.
>> 
>> Thanks,
>> Nemat

-Original Message-
From: Doug Hellmann [mailto:d...@doughellmann.com] 
Sent: Wednesday, April 26, 2017 7:28 AM
To: openstack-operators 
Subject: [Openstack-operators] looking for feedback on proposals to improve 
logging

I am looking for some feedback on two new proposals to add IDs to log messages. 
Please see the thread on openstack-dev, and comment there or on the specs 
referenced there.

http://lists.openstack.org/pipermail/openstack-dev/2017-April/115958.html

Thanks!
Doug

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Fault Genes WG Meeting Summary

[Nematollah Bidokhti]

Hi All,

Below is the link to the weekly meeting summary. There are a lot of exciting ML 
work being performed by the team.

https://etherpad.openstack.org/p/Fault-Genes

thanks,
Nemat



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Fault_Genes] Weekly Summary

[Nematollah Bidokhti]

Below is the link to our meeting summary and action items.

https://etherpad.openstack.org/p/Fault-Genes


Thanks,

Nemat Bidokhti

Chief Architect, Cloud Resiliency
IT Product Line, Computing Lab

Futurewei Technologies, Inc.
Huawei R USA
Tel:+1-408-330-4714
Cell:  +1-408-528-4909
Fax:   +1-408-330-5088
E-mail: http://nematollah.bidok...@huawei.com
2330 Central Expressway
Santa Clara, CA 95050
http://www.huawei.com/
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Fault Genes] WG Weekly Meeting

[Nematollah Bidokhti]

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft Exchange Server 2010
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Pacific Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T02
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ORGANIZER;CN=Nematollah Bidokhti:MAILTO:nematollah.bidok...@huawei.com
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='openstack
 -operat...@lists.openstack.org':MAILTO:openstack-operators@lists.openstack
 .org
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='user-comm
 it...@lists.openstack.org':MAILTO:user-commit...@lists.openstack.org
DESCRIPTION;LANGUAGE=en-US:\n\n\n  Meeting Conference Link: https://www
 .connectmeeting.att.com\n  Meeting Number: 8887160594\n  Code: 377
 3562\n  USA Toll-Free: 888-716-0594\n  USA Caller Paid: 215-861-61
 99\nFor Other Countries:Click Here to View Global Conference A
 ccess Numbers\n\n\n\nThe link to the wiki
  https://wiki.openstack.org/wiki/Fault_Genes_Working_Group\n\n\n\n
RRULE:FREQ=WEEKLY;INTERVAL=1;BYDAY=TH;WKST=SU
SUMMARY;LANGUAGE=en-US:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20160519T09
DTEND;TZID=Pacific Standard Time:20160519T10
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170302T184701Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:198
LOCATION;LANGUAGE=en-US:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:198
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
END:VCALENDAR
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Fault Genes] WG Weekly Meeting

[Nematollah Bidokhti]

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft Exchange Server 2010
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Pacific Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T02
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ORGANIZER;CN=Nematollah Bidokhti:MAILTO:nematollah.bidok...@huawei.com
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='openstack
 -operat...@lists.openstack.org':MAILTO:openstack-operators@lists.openstack
 .org
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='user-comm
 it...@lists.openstack.org':MAILTO:user-commit...@lists.openstack.org
DESCRIPTION;LANGUAGE=en-US:\n\n\n  Meeting Conference Link: https://www
 .connectmeeting.att.com\n  Meeting Number: 8887160594\n  Code: 377
 3562\n  USA Toll-Free: 888-716-0594\n  USA Caller Paid: 215-861-61
 99\nFor Other Countries:Click Here to View Global Conference A
 ccess Numbers\n\n\n\nThe link to the wiki
  https://wiki.openstack.org/wiki/Fault_Genes_Working_Group\n\n\n\n
RRULE:FREQ=WEEKLY;INTERVAL=1;BYDAY=TH;WKST=SU
EXDATE;TZID=Pacific Standard Time:20161124T08,20161208T08
SUMMARY;LANGUAGE=en-US:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20160519T08
DTEND;TZID=Pacific Standard Time:20160519T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170302T184338Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:114
LOCATION;LANGUAGE=en-US:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:114
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161103T08
DTEND;TZID=Pacific Standard Time:20161103T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161103T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170302T184338Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:114
LOCATION:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:114
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161110T08
DTEND;TZID=Pacific Standard Time:20161110T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161110T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170302T184338Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:114
LOCATION:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:114
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161201T08
DTEND;TZID=Pacific Standard Time:20161201T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161201T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170302T184338Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:114
LOCATION:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:114
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161222T08
DTEND;TZID=Pacific Standard Time:20161222T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161222T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170302T184338Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:114
LOCATION:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:114
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1

[Openstack-operators] [Fault Genes] WG Weekly Meeting

[Nematollah Bidokhti]

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft Exchange Server 2010
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Pacific Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T02
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ORGANIZER;CN=Nematollah Bidokhti:MAILTO:nematollah.bidok...@huawei.com
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='openstack
 -operat...@lists.openstack.org':MAILTO:openstack-operators@lists.openstack
 .org
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='user-comm
 it...@lists.openstack.org':MAILTO:user-commit...@lists.openstack.org
ATTENDEE;ROLE=OPT-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='Jiri Such
 omel':MAILTO:jiri.sucho...@suse.com
ATTENDEE;ROLE=OPT-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='Andrew Ve
 itch':MAILTO:andrew.vei...@netcracker.com
ATTENDEE;ROLE=OPT-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='Alan McNa
 mee':MAILTO:alan...@openet.com
ATTENDEE;ROLE=OPT-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='Calum Lou
 don':MAILTO:calum.lou...@metaswitch.com
ATTENDEE;ROLE=OPT-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='Christoph
 er Price':MAILTO:christopher.pr...@ericsson.com
DESCRIPTION;LANGUAGE=en-US:Standard Agenda Items:\n-   Launchpad data t
 ransformation to Fault Genes database\n-   Stackoverflow data capture\
 n-   User Interface for Operators\n-   Machine learning analysis p
 rocess\n-   Collaboration with other projects such as Congress\n- 
   Open items\n\n\n\n  Meeting Conference Link: https://www.connectmeet
 ing.att.com\n  Meeting Number: 8887160594\n  Code: 3773562\n  
 USA Toll-Free: 888-716-0594\n  USA Caller Paid: 215-861-6199\nFor Othe
 r Countries:Click Here to View Global Conference Access Number
 s\n\n\n\nThe link to the wiki https://wik
 i.openstack.org/wiki/Fault_Genes_Working_Group\n\n\n\n
SUMMARY;LANGUAGE=en-US:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20191212T08
DTEND;TZID=Pacific Standard Time:20191212T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20191212T08
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170302T184340Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:115
LOCATION;LANGUAGE=en-US:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:115
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:3
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
END:VCALENDAR
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Fault Genes] WG Weekly Meeting Summary

[Nematollah Bidokhti]

Hi All,

Following is our meeting summary:

*   Zainab presented her work on creating the dictionary.
*   Suli talked about his plan and development of Fault Genes web based 
database
*   Isaac discussed the logging research that he is doing.
*   The goal is for him to take more leadership in running the Logging WG.
*   There is a meeting with OSIC team regarding further collaboration on 
machine learning and Web based database.
*   Mike Turvey has been providing information and guidance on his 
development of the FI orchestration.
*   Since Nemat will be on China business trip, Isaac will run the meeting 
for the next two weeks.

Thanks,
Nemat


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Fault Genes] WG Weekly Meeting Summary

[Nematollah Bidokhti]

Hi All,

We had a great meeting today. Following were the items that we discussed:

*   Zainab presented her word frequency analysis of the Launchpad data and 
high level review of the source code
*   Michael provided some info on the types of data in Stackoverflow
*   Nemat reviewed the updated OpenStack Fault Management Blueprint 
architecture
*   Discussed the importance of having SMART logs in OpenStack
*   Isaac agreed to drive the SMART log sub project in Fault Genes
*   Ala will be driving the Collector component of the architecture
*   Zainab will update the chart with the time between failures
*   Michael will provide the Stackoverflow data in the appropriate format 
to the team
*   Zainab will run her word frequency script on the Stackoverflow data
*   Zainab will merge all word count data into one to be used for  
dictionary selection
*   Team will brainstorm on the potential fault classification to be used 
for the machine learning training data
*   Suli - provide update on the web based database and Fault Genes front 
end design

Thanks to Isaac for involving more people from his project team in Fault Genes 
WG.

Regards,
Nemat





___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Fault Genes] WG Weekly Meeting

[Nematollah Bidokhti]

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft Exchange Server 2010
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Pacific Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T02
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ORGANIZER;CN=Nematollah Bidokhti:MAILTO:nematollah.bidok...@huawei.com
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='openstack
 -operat...@lists.openstack.org':MAILTO:openstack-operators@lists.openstack
 .org
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='user-comm
 it...@lists.openstack.org':MAILTO:user-commit...@lists.openstack.org
DESCRIPTION;LANGUAGE=en-US:When: Occurs every Thursday effective 5/19/2016 
 from 8:00 AM to 9:00 AM (UTC-08:00) Pacific Time (US & Canada).\nWhere: Co
 nference Call\n\nNote: The GMT offset above does not reflect daylight savi
 ng time adjustments.\n\n*~*~*~*~*~*~*~*~*~*\n\nStandard Agenda Items:\n-  
  Launchpad data transformation to Fault Genes database\n-   Stacko
 verflow data capture\n-   User Interface for Operators\n-   Machin
 e learning analysis process\n-   Collaboration with other projects suc
 h as Congress\n-   Open items\n\n\n\n  Meeting Conference Link: ht
 tps://www.connectmeeting.att.com\n  Meeting Number: 8887160594\n  
 Code: 3773562\n  USA Toll-Free: 888-716-0594\n  USA Caller Paid: 2
 15-861-6199\nFor Other Countries:Click Here to View Global Con
 ference Access Numbers\n\n\n\nThe link to
  the wiki https://wiki.openstack.org/wiki/Fault_Genes_Working_Group\n\n\n\
 n
RRULE:FREQ=WEEKLY;INTERVAL=1;BYDAY=TH;WKST=SU
EXDATE;TZID=Pacific Standard Time:20161124T08,20161208T08
SUMMARY;LANGUAGE=en-US:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20160519T08
DTEND;TZID=Pacific Standard Time:20160519T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170113T180119Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:24
LOCATION;LANGUAGE=en-US:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:24
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161103T08
DTEND;TZID=Pacific Standard Time:20161103T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161103T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170113T180119Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:24
LOCATION:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:24
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161110T08
DTEND;TZID=Pacific Standard Time:20161110T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161110T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170113T180119Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:24
LOCATION:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:24
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161201T08
DTEND;TZID=Pacific Standard Time:20161201T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161201T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20170113T180119Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:24
LOCATION:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:24
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161222T08
DTEND;TZID=Pacific Standard Time:20161222T09

Re: [Openstack-operators] [Fault Genes] WG Weekly Meeting

[Nematollah Bidokhti]

Hi All,

Following are the meeting summaries:

*   We had a good participation
*   We have a new member (solution architect) from University of Alabama
*   Nemat presented the team machine learning approach to analyze the data 
and focused on data pre-processing
*   Team agreed that our initial focus will be on defining and analyzing 
the fault classifications
*   Isaac presented the activities that Intel & Rackspace are performing 
regarding the OpenStack resiliency and HA
*   Nemat presented the draft of OpenStack Fault Management Blueprint white 
paper & team provided some feedback.
o   The plan is to have internal reviews first and then send it to a larger 
audience
o   Also adding additional use cases

Action Items:

*   Michael will obtain & combine the latest set of data from Launchpad, 
stack overflow and provide them to the team for analysis
*   Zainab, Suli & Michael will perform the 1st part of the data 
pre-processing for the next meeting.
o   The activity is to come up with word frequency of use from the raw data
o   This is the 1st step for us to start the machine learning data modeling 
which will help us to create the dictionary
*   Isaac will provide all the data and links to the Intel/Rackspace fault 
insertion testing data
*   Nemat submit an abstract for OpenStack summit in Boston regarding the 
OpenStack fault management blueprint

Thanks,
Nemat


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Fault Genes] WG Weekly Meeting Summary

[Nematollah Bidokhti]

General:
-   We had two new members (Ahmed & Zainab) joining our WG
-   Still plan to generate the white paper. Tentative target date is before 
the Operator mid cycle meeting.

Standard Agenda Items:
-   Launchpad data transformation to Fault Genes database
o   Michael: will send out the latest dataset to team members
o   Suli: will has create draft of the web based database and will complete 
after next week
-   Stack-overflow data capture
o   Michael: will send the output of the Stack-overflow script to the team 
for review
-   User Interface for Operators
o   Suli: will create the UI
o   Team discussed the format today
o   Need to finalize the interface
-   Machine learning analysis process
o   Jinzhong, Suli, Zainab & Michael: collaborate on the data analysis
-   Collaboration with other projects
o   Nemat: will continue to work with Eric Kao from Congress project
-   Open items
o   No open items discussed




___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] [Fault Genes] WG Meeting Summary

[Nematollah Bidokhti]

​I'm very interested to know if there is data from stackoverflow which other 
groups could use, is there a place where this data is being collected or easily 
queried.  Specifically, I'm wondering if there is data pertaining the DevOpps 
or CloudApp communities which might be sought?​

We will bring in the data from Stackoverflow and other sources to our database 
where you will be able to run queries. The Stackoverflow data is not as crisp 
as you might find in Launchpad. We are going to run some scripts that will grab 
info based on keywords.

BTW, we would like to have the opportunity to work with Operators where we can 
receive OpenStack bugs and mitigations data.

In addition, we are interested in the OpenStack tribal knowledge where they did 
not become bugs in Launchpad, but annoying enough that needs to be addressed.

​Interesting, what would be the audience/intended topic of the whitepaper?
The goal is to make it interesting enough so developers and operators can 
benefit from the concept.

Nemat


From: David F Flanders [mailto:fland...@openstack.org]
Sent: Wednesday, November 23, 2016 2:02 PM
To: Nematollah Bidokhti
Cc: openstack-operators@lists.openstack.org; user-commit...@lists.openstack.org
Subject: Re: [User-committee] [Fault Genes] WG Meeting Summary

Great to see the summary on-list here Nemat (nice work modelling good behaviour 
for other WG chairs), especially for folk on this side of the planet who are 
unable to attend the meeting given timezones.
Comments/questions inline below:

On Thu, Nov 24, 2016 at 5:41 AM, Nematollah Bidokhti 
<nematollah.bidok...@huawei.com<mailto:nematollah.bidok...@huawei.com>> wrote:
Hi All,

Following is the meeting summary:

-  Launchpad data transformation to Fault Genes database

o   Discussed the design of a new web based database where we can upload info 
and query the database

o   Suli: will develop the database and he will be sending out the plans so all 
team members can review and provide their feedback

o   Michael: will port the latest image of the Launchpad database. He will 
ensure that we bring in all the related fields.

o   Jinzhang mentioned that for the data analysis is best to have the data in 
.csv format

-  Stackoverflow data capture

o   Michael is still working on extracting the applicable data from 
Stackoverflow site that could be used in conjunction with our Launchpad or 
other sources of data

o   He will use the input provided by Zsolt and create the environment on his 
MAC

​I'm very interested to know if there is data from stackoverflow which other 
groups could use, is there a place where this data is being collected or easily 
queried.  Specifically, I'm wondering if there is data pertaining the DevOpps 
or CloudApp communities which might be sought?​


-  User Interface for Operators

o   Suli will create the UI as part of his website design

o   We need to brainstorm on what is the best format to communicate with the 
users

-  Machine learning analysis process

o   Jinzhang provided valuable insights on how we can approach the machine 
learning in our application

o   He mentioned that this is a challenging machine learning project since it 
needs to be applied to natural language instead of some numbers or metrics

o   He suggested to create a training data set which will require OpenStack 
experts

o   For the 1st version of the data we will focus on extracting the fault 
classification first

o   Then share this info with the community and get feedback from appropriate 
experts

-  Collaboration with other projects such as Congress

o   Nemat is communicating with the Congress PTL on future collaboration and 
how Fault Genes data and output can support other projects

-  White Paper

o   Anni Lai suggested to write a white paper to help the community become more 
familiar on the concept that we working towards

o   Team agreed to support this effort

o   Nemat will put together the template and start the paper

o   The team will each contribute based on their areas of expertise
​Interesting, what would be the audience/intended topic of the whitepaper?
/
​dff​

Thanks,

Nemat Bidokhti
Chief Architect, Cloud Resiliency
IT Product Line, Computing Lab
Futurewei Technologies, Inc.
HUAWEI R USA
Tel: +1-408-330-4714<tel:%2B1-408-330-4714>
Cell:   +1-408-528-4909<tel:%2B1-408-528-4909>
Fax:+1-408-330-5088<tel:%2B1-408-330-5088>
E-mail: nematollah.bidok...@huawei.com<mailto:nematollah.bidok...@huawei.com>
2330 Central Expressway
Santa Clara, CA 95050
http://www.huawei.com<http://www.huawei.com/>


___
User-committee mailing list
user-commit...@lists.openstack.org<mailto:user-commit...@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee



--
Flanders | OpenStack Foundation | Community Manager (Cloud Application 
Communities)
http:/

[Openstack-operators] [Fault Genes] WG Meeting Summary

[Nematollah Bidokhti]

Hi All,

Following is the meeting summary:

-  Launchpad data transformation to Fault Genes database

o   Discussed the design of a new web based database where we can upload info 
and query the database

o   Suli: will develop the database and he will be sending out the plans so all 
team members can review and provide their feedback

o   Michael: will port the latest image of the Launchpad database. He will 
ensure that we bring in all the related fields.

o   Jinzhang mentioned that for the data analysis is best to have the data in 
.csv format

-  Stackoverflow data capture

o   Michael is still working on extracting the applicable data from 
Stackoverflow site that could be used in conjunction with our Launchpad or 
other sources of data

o   He will use the input provided by Zsolt and create the environment on his 
MAC

-  User Interface for Operators

o   Suli will create the UI as part of his website design

o   We need to brainstorm on what is the best format to communicate with the 
users

-  Machine learning analysis process

o   Jinzhang provided valuable insights on how we can approach the machine 
learning in our application

o   He mentioned that this is a challenging machine learning project since it 
needs to be applied to natural language instead of some numbers or metrics

o   He suggested to create a training data set which will require OpenStack 
experts

o   For the 1st version of the data we will focus on extracting the fault 
classification first

o   Then share this info with the community and get feedback from appropriate 
experts

-  Collaboration with other projects such as Congress

o   Nemat is communicating with the Congress PTL on future collaboration and 
how Fault Genes data and output can support other projects

-  White Paper

o   Anni Lai suggested to write a white paper to help the community become more 
familiar on the concept that we working towards

o   Team agreed to support this effort

o   Nemat will put together the template and start the paper

o   The team will each contribute based on their areas of expertise

Thanks,

Nemat Bidokhti
Chief Architect, Cloud Resiliency
IT Product Line, Computing Lab
Futurewei Technologies, Inc.
HUAWEI R USA
Tel: +1-408-330-4714
Cell:   +1-408-528-4909
Fax:+1-408-330-5088
E-mail: nematollah.bidok...@huawei.com
2330 Central Expressway
Santa Clara, CA 95050
http://www.huawei.com

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Canceled: [Fault Genes] WG Weekly Meeting

[Nematollah Bidokhti]

BEGIN:VCALENDAR
METHOD:CANCEL
PRODID:Microsoft Exchange Server 2010
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Pacific Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T02
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ORGANIZER;CN=Nematollah Bidokhti:MAILTO:nematollah.bidok...@huawei.com
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='openstack
 -operat...@lists.openstack.org':MAILTO:openstack-operators@lists.openstack
 .org
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='user-comm
 it...@lists.openstack.org':MAILTO:user-commit...@lists.openstack.org
DESCRIPTION;LANGUAGE=en-US:When: Thursday\, November 24\, 2016 8:00 AM-9:00
  AM (UTC-08:00) Pacific Time (US & Canada).\nWhere: Conference Call\n\nNot
 e: The GMT offset above does not reflect daylight saving time adjustments.
 \n\n*~*~*~*~*~*~*~*~*~*\n\nStandard Agenda Items:\n-   Launchpad data 
 transformation to Fault Genes database\n-   Stackoverflow data capture
 \n-   User Interface for Operators\n-   Machine learning analysis 
 process\n-   Collaboration with other projects such as Congress\n-
Open items\n\n\n\n  Meeting Conference Link: https://www.connectmee
 ting.att.com\n  Meeting Number: 8887160594\n  Code: 3773562\n 
  USA Toll-Free: 888-716-0594\n  USA Caller Paid: 215-861-6199\nFor Oth
 er Countries:Click Here to View Global Conference Access Numbe
 rs\n\n\n\nThe link to the wiki https://wi
 ki.openstack.org/wiki/Fault_Genes_Working_Group\n\n\n\n
SUMMARY;LANGUAGE=en-US:Canceled: [Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161124T08
DTEND;TZID=Pacific Standard Time:20161124T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161124T08
CLASS:PUBLIC
PRIORITY:1
DTSTAMP:20161121T230508Z
TRANSP:OPAQUE
STATUS:CANCELLED
SEQUENCE:22
LOCATION;LANGUAGE=en-US:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:22
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:FREE
X-MICROSOFT-CDO-INTENDEDSTATUS:FREE
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:2
X-MICROSOFT-CDO-INSTTYPE:3
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
END:VCALENDAR
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Canceled: [Fault Genes] WG Weekly Meeting

[Nematollah Bidokhti]

BEGIN:VCALENDAR
METHOD:CANCEL
PRODID:Microsoft Exchange Server 2010
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Pacific Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T02
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ORGANIZER;CN=Nematollah Bidokhti:MAILTO:nematollah.bidok...@huawei.com
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='openstack
 -operat...@lists.openstack.org':MAILTO:openstack-operators@lists.openstack
 .org
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='user-comm
 it...@lists.openstack.org':MAILTO:user-commit...@lists.openstack.org
DESCRIPTION;LANGUAGE=en-US:When: Thursday\, November 24\, 2016 8:00 AM-9:00
  AM (UTC-08:00) Pacific Time (US & Canada).\nWhere: Conference Call\n\nNot
 e: The GMT offset above does not reflect daylight saving time adjustments.
 \n\n*~*~*~*~*~*~*~*~*~*\n\n\n\n\n  Meeting Conference Link: https://ww
 w.connectmeeting.att.com\n  Meeting Number: 8887160594\n  Code: 37
 73562\n  USA Toll-Free: 888-716-0594\n  USA Caller Paid: 215-861-6
 199\nFor Other Countries:Click Here to View Global Conference 
 Access Numbers\n\n\n\nThe link to the wik
 i https://wiki.openstack.org/wiki/Fault_Genes_Working_Group\n\n\n\n
SUMMARY;LANGUAGE=en-US:Canceled: [Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161124T08
DTEND;TZID=Pacific Standard Time:20161124T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161124T08
CLASS:PUBLIC
PRIORITY:1
DTSTAMP:20161121T230040Z
TRANSP:OPAQUE
STATUS:CANCELLED
SEQUENCE:21
LOCATION;LANGUAGE=en-US:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:21
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:FREE
X-MICROSOFT-CDO-INTENDEDSTATUS:FREE
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:2
X-MICROSOFT-CDO-INSTTYPE:3
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
END:VCALENDAR
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Fault Genes] WG Weekly Meeting

[Nematollah Bidokhti]

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft Exchange Server 2010
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Pacific Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T02
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ORGANIZER;CN=Nematollah Bidokhti:MAILTO:nematollah.bidok...@huawei.com
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='openstack
 -operat...@lists.openstack.org':MAILTO:openstack-operators@lists.openstack
 .org
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN='user-comm
 it...@lists.openstack.org':MAILTO:user-commit...@lists.openstack.org
DESCRIPTION;LANGUAGE=en-US:When: Occurs every Thursday effective 5/19/2016 
 from 8:00 AM to 9:00 AM (UTC-08:00) Pacific Time (US & Canada).\nWhere: Co
 nference Call\n\nNote: The GMT offset above does not reflect daylight savi
 ng time adjustments.\n\n*~*~*~*~*~*~*~*~*~*\n\n\n\n\n  Meeting Confere
 nce Link: https://www.connectmeeting.att.com\n  Meeting Number: 888716
 0594\n  Code: 3773562\n  USA Toll-Free: 888-716-0594\n  USA Ca
 ller Paid: 215-861-6199\nFor Other Countries:Click Here to Vie
 w Global Conference Access Numbers\n\n\n\
 nThe link to the wiki https://wiki.openstack.org/wiki/Fault_Genes_Working_
 Group\n\n\n\n
RRULE:FREQ=WEEKLY;INTERVAL=1;BYDAY=TH;WKST=SU
EXDATE;TZID=Pacific Standard Time:20161124T08
SUMMARY;LANGUAGE=en-US:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20160519T08
DTEND;TZID=Pacific Standard Time:20160519T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20161121T230039Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:21
LOCATION;LANGUAGE=en-US:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:21
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161103T08
DTEND;TZID=Pacific Standard Time:20161103T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161103T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20161121T230039Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:21
LOCATION:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:21
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
BEGIN:VEVENT
SUMMARY:[Fault Genes] WG Weekly Meeting
DTSTART;TZID=Pacific Standard Time:20161110T08
DTEND;TZID=Pacific Standard Time:20161110T09
UID:04008200E00074C5B7101A82E00850B72A929FAAD101000
 0100028E111C3B604D445920643DCDC32B7B2
RECURRENCE-ID;TZID=Pacific Standard Time:20161110T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20161121T230039Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:21
LOCATION:Conference Call
X-MICROSOFT-CDO-APPT-SEQUENCE:21
X-MICROSOFT-CDO-OWNERAPPTID:1315723232
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:1
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
END:VCALENDAR
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Audit Logging - Interested? What's missing?

[Nematollah Bidokhti]

Hi Tom,

It would be great if the logs were formatted as such that could convey the 
following:
- Fault classification/types
- Potential root Causes
- Latest state before failure/crash

The above can help with automation and self healing. This is part of our Fault 
Genes WG mission to get to consistent log structure for our fault management 
policies.

Thanks,
Nemat


-Original Message-
From: Tom Fifield [mailto:t...@openstack.org] 
Sent: Wednesday, November 16, 2016 2:29 PM
To: OpenStack Operators
Subject: [Openstack-operators] Audit Logging - Interested? What's missing?

Hi Ops,

Was chatting with Department of Defense in Australia the other day, and one of 
their pain points is Audit Logging. Some bits of OpenStack just don't leave 
enough information for proper audit. So, thought it might be a good idea to 
gather people who are interested to brainstorm how to get it to a good level 
for all :)

Does your cloud need good audit logging? What do you wish was there at the 
moment, but isn't?


Regards,


Tom

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Fault Genes] WG Weekly Meeting Summary

[Nematollah Bidokhti]

Hi All,

We had a great meeting today. Following were discussed and action items:
*   Started reviewing the Launchpad critical bugs
o   Classifying the bugs (fault class & fault types), recommended mitigation
*   Decided to focus on the high bug rate projects
*   Added two new members to the team
o   Zsolt Nagy (T-Systems)
o   Michael Zhouzzy (UCSD)
*   Asked the team members to contribute to the creation of fault classes & 
types list for their respective areas
o   Input from Matt Greene
*   Fault Class: VM not accessible
*   Fault Type: network connectivity, insufficient hypervisor resources, VM 
guest mis-configuration, authentication/permissions error
*   Michael: will update the Google sheet to include the dropdowns
*   Zsolt: provide the source code to grab data from StackOverflow
*   Michael: Use Zsolt's source code to provided the latest updates & 
determine if there are open APIs to export the data to our worksheets for 
further analysis
*   Zsolt: invite more people from the T-Systems Operation team
*   Nemat: invite Ching and Farhad to join the team as Neutron experts

Thanks,
Nemat




___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] New working group: [fault genes]. Recap from Austin from "Taxonomy of Failure" Ops session and plans going forward as a working group

[Nematollah Bidokhti]

Hi,

This email is a recap from our OpenStack summit meeting "Taxonomy of Failure" 
in Austin. The purpose of this email is to provide a summary of the meeting and 
future plans.

We had between 55-60 people participating in our session and received a number 
of comments and suggestions. Basically all comments were positive and felt we 
are going in a right direction.

The goal is to look at OpenStack resiliency in holistic fashion by identifying 
all possible failure modes (either experienced to date or based on design 
implementation), classifying them, defining the ideal mitigation strategy, how 
should they be reported and how they can be re-created with the OpenStack 
version in mind. The results of this effort will be used throughout OpenStack 
lifecycle (design, development, test, deployment).

After our meeting I met with a lot of companies in the market place and 
received lots of encouragement to complete the effort that we have started. 
There were 20 companies that I met with and all expressed their interest to 
support this activity. As a result, we have decided to start a working group 
"Fault Genes" to focus on all OpenStack failure modes.

The plan is to start with email communications and filling out our Google Sheet 
template 
(https://docs.google.com/spreadsheets/d/1sekKLp7C8lsTh-niPHNa2QLk5kzEC_2w_UsG6ifC-Pw/edit#gid=2142834673)
 that we have set up, start out with a weekly meeting, adjusting as the group 
sees fit and in 3 months have a check point on what we have accomplished. Then, 
we should have a picture of what we have accomplished, where this will go and 
have information to present at OpenStack in Barcelona. Below is the link to the 
etherpad:

https://etherpad.openstack.org/p/AUS-ops-Taxonomy-of-Failures

For those who were in the meeting or discussed this at the summit, and  you 
understand spreadsheet, please take time and fill in the spreadsheet with the 
failure modes that you have experienced so far and related attributes for each 
failure mode.

I'll schedule a meeting to get those who weren't at the summit informed of the 
process and how to use the spreadsheet.

Suggestions of meeting times, or further discussion here is appreciated and 
appropriate.

My availability for meetings is:  1600-2359 UTC

Please use this link http://doodle.com/poll/8ymwuqva7itv84p8 to provide your 
suggested time.

Thanks,

Nemat Bidokhti
Chief Reliability Architect
IT Product Line, Computing 
Lab
Futurewei Technologies, Inc.
HUAWEI R USA
Tel: +1-408-330-4714
Cell:   +1-408-528-4909
Fax:+1-408-330-5088
E-mail: 
nematollah.bidok...@huawei.com
2330 Central Expressway 
Santa Clara, CA 95050
http://www.huawei.com
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators