Re: [openstack-dev] [Openstack-operators] [nova] Default scheduler filters survey

2018-04-30 Thread Mikhail Medvedev 
On Mon, Apr 30, 2018 at 8:46 AM, Jay Pipes <jaypi...@gmail.com> wrote:
> On 04/30/2018 09:18 AM, Mikhail Medvedev wrote:
>>
>> On Sun, Apr 29, 2018 at 4:29 PM, Ed Leafe <e...@leafe.com> wrote:
>>>
>>>
>>> Another data point that might be illuminating is: how many sites use a
>>> custom (i.e., not in-tree) filter or weigher? One of the original design
>>> tenets of the scheduler was that we did not want to artificially limit what
>>> people could use to control their deployments, but inside of Nova there is a
>>> lot of confusion as to whether anyone is using anything but the included
>>> filters.
>>>
>>> So - does anyone out there rely on a filter and/or weigher that they
>>> wrote themselves, and maintain outside of OpenStack?
>>>
>>
>> Internal cloud that is used for Power KVM CI single use VMs:
>>
>> AvailabilityZoneFilter
>> AggregateMultiTenancyIsolation
>> RetryFilter
>> RamFilter
>> ComputeFilter
>> ComputeCapabilitiesFilter
>> ImagePropertiesFilter
>> CoreFilter
>> NumInstancesFilter *
>> NUMATopologyFilter
>>
>> NumInstancesFilter is a custom weigher I have added that returns
>> negative number of instances on a host. Using it this way gives an
>> even spread of instances over the compute nodes up to a point the
>> compute cores are filled up evenly, then it overflows to the compute
>> nodes with more CPU cores. Maybe it is possible to achieve the same
>> with existing filters, at the time I did not see how.

Correction: above describes custom weigher I've added, not the in-tree
NumInstancesFilter.

>
>
> Hi Mikhail,
>
> Did you mean to say you created a new *weigher*, not filter?

Jay, thanks for spotting this, been awhile since I've done it.
NumInstancesFilter is a standard filter, so I obviously did not write
it.

I've added a custom weigher that I have created
(scheduler_weight_classes=pkvmci-os.nova.scheduler.weights.instance.InstanceWeigher)
and maintain locally.

>
> Best,
> -jay
>
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova] Default scheduler filters survey

2018-04-30 Thread Mikhail Medvedev 
On Sun, Apr 29, 2018 at 4:29 PM, Ed Leafe <e...@leafe.com> wrote:
>
> Another data point that might be illuminating is: how many sites use a custom 
> (i.e., not in-tree) filter or weigher? One of the original design tenets of 
> the scheduler was that we did not want to artificially limit what people 
> could use to control their deployments, but inside of Nova there is a lot of 
> confusion as to whether anyone is using anything but the included filters.
>
> So - does anyone out there rely on a filter and/or weigher that they wrote 
> themselves, and maintain outside of OpenStack?
>

Internal cloud that is used for Power KVM CI single use VMs:

AvailabilityZoneFilter
AggregateMultiTenancyIsolation
RetryFilter
RamFilter
ComputeFilter
ComputeCapabilitiesFilter
ImagePropertiesFilter
CoreFilter
NumInstancesFilter *
NUMATopologyFilter

NumInstancesFilter is a custom weigher I have added that returns
negative number of instances on a host. Using it this way gives an
even spread of instances over the compute nodes up to a point the
compute cores are filled up evenly, then it overflows to the compute
nodes with more CPU cores. Maybe it is possible to achieve the same
with existing filters, at the time I did not see how.

---
Mikhail Medvedev
IBM

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-06-16 Thread Mikhail Medvedev 
On Fri, Jun 16, 2017 at 6:01 AM, Sean Dague <s...@dague.net> wrote:
> On 06/15/2017 10:06 PM, Tony Breeds wrote:
>> Hi All,
>>   I just push a review [1] to bump the minimum etcd version to
>> 3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
>> cycle to be making changes like this but releasing pike with a dependacy
>> on 3.1.x make it harder for users on ppc64le (not many but a few :D)
>>
>> Yours Tony.
>>
>> [1] https://review.openstack.org/474825
>
> It should be fine, no one is really using these much at this point.
> However it looks like mirroring is not happening automatically? The
> patch fails on not existing in the infra mirror.
>
> -Sean
>

It appears so. Also, IIRC, infra mirror would only host x86 binaries.
Right now PowerKVM CI works by patching devstack-gate to override
infra etcd download url. The fix [2] still needs to get merged to make
it a bit easier to use d-g with your own etcd mirror.

[2] https://review.openstack.org/#/c/467437/

---
Mikhail Medvedev
IBM OpenStack CI for KVM on Power

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [intel experimental ci] Is it actually checking anything?

2017-04-17 Thread Mikhail Medvedev 
On Mon, Apr 17, 2017 at 12:31 PM, Jay Pipes <jaypi...@gmail.com> wrote:
> Please see the below output from the Intel Experimental CI (from
> https://review.openstack.org/414769):
>
> On 04/17/2017 01:28 PM, Intel Experimental CI (Code Review) wrote:
>>
>> Intel Experimental CI has posted comments on this change.
>>
>> Change subject: placement: SRIOV PF devices as child providers
>> ..
>>
>>
>> Patch Set 17:
>>
>> Build succeeded (check pipeline).
>>
>> - tempest-dsvm-full-nfv-xenial
>> http://intel-openstack-ci-logs.ovh/portland/2017-04-17/414769/17/check/tempest-dsvm-full-nfv-xenial/1bcdb64
>> : FAILURE in 38m 34s (non-voting)
>> - tempest-dsvm-intel-nfv-xenial
>> http://intel-openstack-ci-logs.ovh/portland/2017-04-17/414769/17/check/tempest-dsvm-intel-nfv-xenial/a21d879
>> : FAILURE in 40m 00s (non-voting)
>> - tempest-dsvm-multinode-ovsdpdk-nfv-networking-xenial
>> http://intel-openstack-ci-logs.ovh/portland/2017-04-17/414769/17/check/tempest-dsvm-multinode-ovsdpdk-nfv-networking-xenial/837e59d
>> : FAILURE in 47m 45s (non-voting)
>
>
> As you can see, it says the build succeeded, but all three jobs in the
> pipeline failed.

This would happen when CI is voting but all the jobs in a check are
non-voting. Zuul ignores non-voting job result, and as there isn't a
single voting job, it reports 'build succeeded'. Maybe it should be a
zuul bug?

>
> Is someone actively looking into this particular 3rd party CI system?

I do not see anything wrong with that CI (apart from misleading
comment due to zuul issue I mentioned above).

>
> Best,
> -jay
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

---
Mikhail Medvedev
IBM

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [dib] diskimage-builder v2 RC1 release; request for test

2017-03-20 Thread Mikhail Medvedev 
On Fri, Mar 17, 2017 at 3:23 PM, Andre Florath  wrote:
> Hello!
>
> Thanks for the bug report. Can you please file this as a bug?

Hi Andre,
Submitted the bug https://bugs.launchpad.net/diskimage-builder/+bug/1674402

>
> There is a very high probability that I introduced a change that
> leads to the failure [1] - even if this is fixed now there is a
> high probability that it will fail again when [2] is merged.
>
> The reason is, that there are no test cases because there is no
> nodepool CI job running on PPC. (Or do I miss something here?)

Correct, there isn't a ppc CI running on diskimage-builder patches.

>
> We are only a very few people at diskimage-builder with limited
> resources and had to concentrate on the 'main-line' (i.e.: that
> what can be tested by us). A discussion about what to support
> or test was already started some time ago [3].
>
> Looks that you are from IBM: would it be possible to provide
> PPC hardware for testing and the man-power to integrate
> this into the CI?
> This would really help finding such problems during development
> phase and would put me into the situation to be able to fix your
> problem.
>

Agreed, there is little can be done without being able to test the failure case.

Would adding a third-party CI job help? I can put together a
functional job on ppc64. I assume we want a job based on
gate-dib-dsvm-functests-*?

> Kind regards
>
> Andre
>
> [1] https://review.openstack.org/#/c/375261/
> [2] https://review.openstack.org/#/c/444586/
> [3] https://review.openstack.org/#/c/418204/
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [dib] diskimage-builder v2 RC1 release; request for test

2017-03-17 Thread Mikhail Medvedev 
On Thu, Feb 9, 2017 at 12:22 AM, Ian Wienand <iwien...@redhat.com> wrote:
> Hi
>
> diskimage-builder has been working on a feature/v2 branch
> incorporating some largely internal changes to the way it finds and
> calls elements, enhancements to partitioning and removal of some
> long-deprecated elements.  We have just tagged 2.0.0rc1 and are
> requesting testing by interested parties.
>

I am too late, but it is a good place to mention that 2.0.0 introduced
a regression that ends up breaking image build on ppc64 platform
somehwere within bootloader element.

Error trace:

2017-03-17 15:54:37,317 INFO nodepool.image.build.devstack-xenial: +
[[ ppc64el =~ ppc ]]
2017-03-17 15:54:37,317 INFO nodepool.image.build.devstack-xenial: +
/usr/sbin/grub-install --modules=part_msdos --force /dev/loop0
--no-nvram
2017-03-17 15:54:37,369 INFO nodepool.image.build.devstack-xenial:
Installing for powerpc-ieee1275 platform.
2017-03-17 15:54:45,912 INFO nodepool.image.build.devstack-xenial:
/usr/sbin/grub-install: warning: unknown device type loop0p1
2017-03-17 15:54:45,913 INFO nodepool.image.build.devstack-xenial: .
2017-03-17 15:54:45,987 INFO nodepool.image.build.devstack-xenial:
/usr/sbin/grub-install: error: the chosen partition is not a PReP
partition.

On 1.2.8, the cmdline was ` /usr/sbin/grub-install
--modules=part_msdos --force /dev/loop0p1 --no-nvram`. I do not know
yet why grub-install thinks it should still use loop0p1 on 2.1.0.

I have not dug too deep into it, ended up downgrading
diskimage-builder to 1.2.8. Simple downgrade would not work, because
1.2.8 would now fail with dib-run-parts binary missing. So I also had
to copy dib-run-parts from 2.1.0. Maybe installing from source and
pinning 1.2.8 would have been a better strategy.

---
Mikhail Medvedev (mmedvede)
IBM

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party][ci] Transition your devstack-gate based CI to local.conf

2017-02-28 Thread Mikhail Medvedev 
>
> I am usually on freenode during US daytime, #openstack-third-party-ci
> and #openstack-infra are good places to ask questions.
>

Forgot to mention my IRC nick is mmedvede

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party][ci] Transition your devstack-gate based CI to local.conf

2017-02-28 Thread Mikhail Medvedev 
On Mon, Feb 27, 2017 at 11:12 PM, Nilesh Savant
 wrote:
> Hello Mikhail,
>
>
>
> We are operating CI for Veritas HyperScale, Due to this changes our CI
> started failing,
>
>
>
> Could you please help us by providing the sample local.conf file as per new
> changes.
>
>
> It will be great if you can point us to any of the working CI with your
> changes for reference.
>

You can find working local.conf files in the logs of OpenStack Infra
CI, e.g. [1]. I am maintaining IBM PowerKVM CI, our current conf is
[2]. For the sake of ml (the logs would expire) a local.conf might
looks like this:

[[local|localrc]]
Q_USE_DEBUG_COMMAND=True
NETWORK_GATEWAY=10.1.0.1
USE_SCREEN=False
DEST=/opt/stack/new
DATA_DIR=/opt/stack/data
ACTIVE_TIMEOUT=90
BOOT_TIMEOUT=90
ASSOCIATE_TIMEOUT=60
TERMINATE_TIMEOUT=60
[[test-config|/opt/stack/new/tempest/etc/tempest.conf]]
[compute-feature-enabled]
interface_attach=False
[compute]
volume_device_name=sdb

To fix your CI you need to stop using localrc, and use
[[local|localrc]] section of local.conf instead. There are a few
things you can do, e.g. use DEVSTACK_LOCAL_CONFIG as suggested by
Clark. What it means is you can try setting it to contain your old
localrc settings: e.g. DEVSTACK_LOCAL_CONFIG=$(< old_localrc_file).
This needs to be done before devstack-gate starts, you can not use it
from pre_test_hook. You should also avoid writing directly to
local.conf. I would not want to give you changes that I made to our CI
because I likely am not using a stable interface.

Please also check the writeup by Sean Dague [3].

I am usually on freenode during US daytime, #openstack-third-party-ci
and #openstack-infra are good places to ask questions.

[1] 
http://logs.openstack.org/50/438750/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d917c7c/logs/local.conf.txt.gz
[2] 
http://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/50/438750/1/check/tempest-dsvm-full-xenial/ee1c6b8/local.conf.txt.gz
[3] http://lists.openstack.org/pipermail/openstack-dev/2017-February/112872.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party][ci] openstack CI VM template

2017-02-28 Thread Mikhail Medvedev 
On Tue, Feb 28, 2017 at 2:52 AM, Guo, Ruijing  wrote:
> Hi, CI Team,
>
>
>
> I’d like to know if openstack CI VM support nested virtualization.
>

OpenStack CI infrastructure is using nested visualization inside of
devstack VMs to perform tempest testing. But at the moment accel=tcg
is used (emulation) for second level virt. IIRC it is done because
some of the provider clouds had problems with KVM acceleration.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][swg] per-project "Business only" moderated mailing lists

2017-02-27 Thread Mikhail Medvedev 
On Mon, Feb 27, 2017 at 11:54 AM, Shamail  wrote:
>
>
> The single openstack-dev list works fine for me.  I am not using anything to 
> filter either beyond what my default mail clients offer.
>
> Could we modify the question slightly to also ask how people are dealing with 
> reading through the single list?  That might help us write a page that 
> newcomers could read to get tips on how to best handle the mailing list 
> traffic (which seems to be the origin of the proposed change).
>

+1, collecting workflows people use for dealing with ml would be good.
I am new to such a high-volume list, and I have tried several things.
Now I have a few filters in gmail that mark threads I am not
interested in, but they all still go into my main inbox. I then mute
irrelevant threads. Once you did the first round of mutes, it becomes
easier. I spend 10-20 minutes daily to get through all of the threads.

I see a very marginal benefit in adding more lists.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [third-party][ci] Transition your devstack-gate based CI to local.conf

2017-02-22 Thread Mikhail Medvedev 
Hi third-party CI operators,

This is for those who use devstack-gate with localrc. You might have
started seeing failures since [1] landed. This is due to a few
devstack settings now being ignored. When devstack sees that you have
both localrc and [[local|localrc]] section in local.conf, it only uses
localrc file.

A solution is to move your localrc settings into [[local|localrc]]
section of local.conf. For a temporary fix you can also pin
devstack-gate to a version before [1], until your configuration is
migrated.

I hope this saves some of you some debugging time. I see at least one
CI that is currently affected (nova Virtuozzo Storage CI).

[1] 
https://review.openstack.org/#q,I111a6df3adc07ed426ac714cb9e64667a63e0e5a,n,z

Happy testing,

---
Mikhail Medvedev
IBM

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] [gate] [all] openstack services footprint lead to oom-kill in the gate

2017-02-02 Thread Mikhail Medvedev 
On Thu, Feb 2, 2017 at 12:28 PM, Jeremy Stanley <fu...@yuggoth.org> wrote:
> On 2017-02-02 04:27:51 + (+), Dolph Mathews wrote:
>> What made most services jump +20% between mitaka and newton? Maybe there is
>> a common cause that we can tackle.
> [...]
>
> Almost hesitant to suggest this one but since we primarily use
> Ubuntu 14.04 LTS for stable/mitaka jobs and 16.04 LTS for later
> branches, could bloat in a newer release of the Python 2.7
> interpreter there (or something even lower-level still like glibc)
> be a contributing factor?

In our third-party CI (IBM KVM on Power) we run both stable/mitaka and
master on Ubuntu Xenial. I went ahead and plotted dstat graphs, see
http://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/dstat20170202/
. It does look like there is some difference in overall memory use -
mitaka uses a bit less. This is anecdotal, but still is an extra data
point. Also note that we have 12G of ram, and we do not see oom kills.

> I agree it's more likely bloat in some
> commonly-used module (possibly even one developed outside our
> community), but potential system-level overhead probably should also
> get some investigation.
> --
> Jeremy Stanley
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Mikhail Medvedev
IBM

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [third-party][ci] Cancel Tue 17:00 UTC biweekly third-party CI working group meeting

2016-11-23 Thread Mikhail Medvedev 
Hi All,

Recently there was not much attendance and activity during biweekly third-party
CI working group meetings on Tuesdays 17:00 UTC [1]. During the last meeting
[2] it was discussed that it is a good time to free up the meeting slot in case
other teams want to use it. I have created the patch [3] to do so.

When we have something to discuss, I suggest we attend another third-party
meeting slot on Mondays 15:00 UTC (also [1]). That meeting did not see much
attendance lately either, so I think it should be ok to share the slot. There
is still an #openstack-third-party-ci channel that is being logged and we can
use for ad-hoc discussions.

[1] https://wiki.openstack.org/wiki/Meetings/ThirdParty
[2] 
http://eavesdrop.openstack.org/meetings/third_party/2016/third_party.2016-11-15-17.00.log.html
[3] 
https://review.openstack.org/#q,I70a5e7a0f5267d2a4e69d5a10ffb00d8c95ebc6d,n,z

---
Mikhail Medvedev (mmedvede)
IBM

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Infra] Announcing Gertty 1.4.0

2016-08-01 Thread Mikhail Medvedev 
On Mon, Aug 1, 2016 at 4:00 PM, James E. Blair  wrote:
> Masayuki Igawa  writes:
>
>> Hi!
>>
>> On Wed, Jul 27, 2016 at 11:50 PM, James E. Blair  wrote:
>>> Michał Dulko  writes:
>>>
 Just wondering - were there tries to implement syntax highlighting in
 diff view? I think that's the only thing that keeps me from switching to
 Gertty.
>>>
>>> I don't know of anyone working on that, but I suspect it could be done
>>> using the pygments library.
>>
>> Oh, it's an interesting feature to me :) I'll try to investigate and
>> implement in next couple of days :)
>
> As I think about this, one challenge in particular comes to mind: Gerrit
> uses background color (green and pink) to distinguish old and new
> text when displaying diffs.  In Gertty, I avoided that and used
> foreground colors instead because text with green and red backgrounds is
> difficult to read on a terminal.
>
> We essentially have two channels of information that we want to
> represent with color -- the diff, and the syntax.  They can sometimes
> overlap.
>
> Perhaps we could use a 256 color (or even RGB) terminal for this
> feature.  Then we may be able to get just the right shade of background
> color for the diff channel, and use the foreground colors for syntax
> highlighting.
>
> At any rate, it may be worth trying to solve *this* problem first with a
> mockup to see if there is any way of doing this without making our eyes
> bleed before working on the code to implement it.

In theory it is possible split diff and syntax spatially, so there
would be no need to mix diff and syntax colors. Mockup
http://i.imgur.com/gAD9x9v.png

>
> -Jim
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] NOT_REGISTERED to be or not to be

2016-08-01 Thread Mikhail Medvedev 
Thanks starting the discussion, Lenny. I thought about a way to
accomplish that, and came up with few options:

 - Pre-register all possible jenkins jobs with gearman. This only
avoids not registered errors due to gearman server restart, not if you
misconfigured your system;

 - Add option to zuul to treat builds as effectively canceled if there
is no corresponding gearman worker, this would avoid the
not_registered comment in all cases. This can be done in one line of
code if you just hack it [1] (assuming my patch is correct).

 - Automatically register all jobs that zuul tries to start with
gearman. That is, check if  exists on gearman, and register
dummy function if not, for each attempted build. This would avoid
missing any patches.

[1] 
https://review.openstack.org/#q,Ie6d5ea35c6eeed465168f24921b04442df8f5744,n,z


On Mon, Aug 1, 2016 at 10:53 AM, Lenny Verkhovsky  wrote:
> Hi,
>
>
>
> Currently in some cases[1] CI sets a comment on the patch set as
> NOT_REGISTERED.
>
>
>
> Those comments are very hard to monitor for CI operators and have noisy
> value for the developers.
>
>
>
> Maybe a better solution is not commenting in such cases at all as discussed
> in [2].
>
>
>
> If a developer is missing some important CI comments, it can rechecked later
> or sent an email to CI owner.
>
>
>
> [1] No valid slaves
>
>Not all jobs are registered due to zuul restart for instance
>
>
>
> [2]
> http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/%23openstack-meeting.2016-08-01.log.html
>
>
>
>
>
> Thanks.
>
> Lenny
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Dell recheck command issue (was: UFCG OneView CI comments missing recheck command)

2016-07-08 Thread Mikhail Medvedev 
On Fri, Jul 8, 2016 at 10:10 AM, Dmitry Tantsur <dtant...@redhat.com> wrote:
> I've noticed that Dell CI has the same problem: it uses "recheck Dell"
> causing the whole check pipeline to rerun.

There is a doc patch that added selective recheck syntax in addition
to standard recheck [1]: it asks third-party CIs to support both
"recheck" and ": recheck". There is also [2] to revert that
patch.

Whether or not the patch would get reverted, it would be nice if there
is a recommended way to only recheck a particular third-party CI. Can
selective recheck be abused to beat an unstable CI into submission?
Yes. And it also has valid use cases, e.g. rechecking a CI failing due
an obvious one-time fluke.

[1] https://review.openstack.org/#/c/327065/
[2] https://review.openstack.org/#/c/329049/

> On 06/29/2016 02:23 AM, Villalovos, John L wrote:
>>
>>
>>
>>> -Original Message-
>>> From: Thiago Paiva [mailto:thia...@lsd.ufcg.edu.br]
>>> Sent: Tuesday, June 28, 2016 17:00
>>> To: OpenStack Development Mailing List (not for usage questions)
>>> Cc: ufcg-oneview...@lsd.ufcg.edu.br
>>> Subject: Re: [openstack-dev] [ironic] UFCG OneView CI comments missing
>>> recheck command
>>>
>>> Hi Jay,
>>>
>>> Sorry about that. The comment should be "recheck oneview" to test again.
>>> I'll patch the failure message with instructions, thanks for the warning.
>>
>>
>> I'm not sure "recheck oneview" is a good command because it will kick off
>> the master recheck. I would suggest something that will not trigger the
>> normal jobs to recheck.
>>
>> "retest oneview"??  Hopefully there is some standard for 3rd Party CI to
>> use.
>>
>> And yes, please do put in the message the command to do a job recheck.
>>
>> John
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Mikhail Medvedev
IBM

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] [infra] [qa] Graphs of how long jobs take?

2016-06-08 Thread Mikhail Medvedev 
Hi Jay,

On Wed, Jun 8, 2016 at 5:56 PM, Jay Faulkner <j...@jvf.cc> wrote:

> Hey all,
>
> As you may recall, recently Ironic was changed to use iPXE and TinyIPA in
> the jobs, as part of an attempt to get the jobs to use less ram and perhaps
> even run more quickly in the short run. However, when I tried to make a
> graph at graphite.openstack.org showing the duration of the jobs, it
> doesn't look like that metric was available
> (stats.zuul.pipeline.check.job.check-tempest-dsvm-ironic-pxe_ssh.* appears
> to only track the job result).
>

I did find two metrics that seem to be what you are looking for:

stats.timers.nodepool.job.gate-tempest-dsvm-ironic-pxe_ssh.master.ubuntu-trusty.runtime.mean
stats.timers.zuul.pipeline.check.job.gate-tempest-dsvm-ironic-pxe_ssh.*.mean



> Is there a common or documented way or tool to graph duration of jobs so I
> can see the real impact of this change?
>
>
> Thanks a bunch,
>
> Jay Faulkner
>
> OSIC
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Mikhail Medvedev
IBM
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Infra][cinder] How to configure the third party CI to be triggered only when jenkins +1

2016-02-23 Thread Mikhail Medvedev 
On Mon, Feb 22, 2016 at 7:32 PM, liuxinguo  wrote:
> Hi,
>
>
>
> There is no need to trigger third party CI if a patch does not pass Jenkins
> Verify.
>
> I think there is a way to reach this but I’m not sure how.
>
>
>
> So is there any reference or suggestion to configure the third party CI to
> be triggered only when jenkins +1?
>
>

If you are using zuul, then you should look into 'approval' setting in
layout. E.g. check the current layout that Infra uses for gate
pipeline: 
https://github.com/openstack-infra/project-config/blob/6b71e8cac676e04141839eeecce3462df3a04848/zuul/layout.yaml#L41-L46.

>
> Thanks for any input!
>
>
>
> Regards,
>
> Wilson Liu
>
>
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] nova cli commands fail with 404. devstack installation from today

2016-01-21 Thread Mikhail Medvedev 
On Thu, Jan 21, 2016 at 1:48 PM, Bob Hansen  wrote:

> Found it. The contents of the admin file (e.g.
> ../devstack/accrc/admin/admin) that I sourced for the admin credentials do
> not work with the nova cli. This combination of OS_* variables produced the
> error.
>
> export OS_PROJECT_NAME="admin"
> export OS_AUTH_URL="http://127.0.0.1:35357;
> export OS_CACERT=""
> export OS_AUTH_TYPE=v2password
> export OS_PASSWORD="secretadmin"
> export OS_USER_DOMAIN_ID=default
> export OS_PROJECT_DOMAIN_ID=default
>
> this combination works with nova, glance and neutron.
>
> export OS_PROJECT_NAME=admin
> export OS_PASSWORD=secretadmin
> export OS_AUTH_URL=http://127.0.0.1:35357
> export OS_USERNAME=admin
> export OS_TENANT_NAME=admin
> export OS_CACERT=
>
> To be honest, I've seen so many examples of the 'correct' set of
> environment variables with different AUTH_TYPES, it's very hard to tell
> which variable 'set' is appropriate for which AUTH_TYPE and version of the
> keystone API.
>
> A pointer to this sort of information is appreciated.
>
AFAIK, OpenStackClient (OSC) is making it a bit saner. See the docs
http://docs.openstack.org/developer/python-openstackclient/, and
"authentication" section in particular.

>
> Bob Hansen
> z/VM OpenStack Enablement
>
>
> [image: Inactive hide details for Bob Hansen---01/21/2016 10:46:15
> AM---Yes, it is image-list not image list. I don't seem to be able t]Bob
> Hansen---01/21/2016 10:46:15 AM---Yes, it is image-list not image list. I
> don't seem to be able to find any other hints in any of the
>
> From: Bob Hansen/Endicott/IBM@IBMUS
> To: "OpenStack Development Mailing List \(not for usage questions\)" <
> openstack-dev@lists.openstack.org>
> Date: 01/21/2016 10:46 AM
>
> Subject: Re: [openstack-dev] nova cli commands fail with 404. devstack
> installation from today
> --
>
>
>
> Yes, it is image-list not image list. I don't seem to be able to find any
> other hints in any of the nova logs.
>
> nova --debug image-list shows this:
>
> DEBUG (extension:157) found extension EntryPoint.parse('token =
> keystoneauth1.loading._plugins.identity.generic:Token')
> DEBUG (extension:157) found extension EntryPoint.parse('v3token =
> keystoneauth1.loading._plugins.identity.v3:Token')
> DEBUG (extension:157) found extension EntryPoint.parse('password =
> keystoneauth1.loading._plugins.identity.generic:Password')
> DEBUG (v2:62) Making authentication request to
> *http://127.0.0.1:35357/tokens* 
> INFO (connectionpool:207) Starting new HTTP connection (1): 127.0.0.1
> DEBUG (connectionpool:387) "POST /tokens HTTP/1.1" 404 93
> DEBUG (session:439) Request returned failure status: 404
> DEBUG (shell:894) The resource could not be found. (HTTP 404)
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/novaclient/shell.py", line
> 892, in main
> OpenStackComputeShell().main(argv)
> File "/usr/local/lib/python2.7/dist-packages/novaclient/shell.py", line
> 726, in main
> api_version = api_versions.discover_version(self.cs, api_version)
> File "/usr/local/lib/python2.7/dist-packages/novaclient/api_versions.py",
> line 267, in discover_version
> client)
> File "/usr/local/lib/python2.7/dist-packages/novaclient/api_versions.py",
> line 248, in _get_server_version_range
> version = client.versions.get_current()
> File "/usr/local/lib/python2.7/dist-packages/novaclient/v2/versions.py",
> line 83, in get_current
> return self._get_current()
> File "/usr/local/lib/python2.7/dist-packages/novaclient/v2/versions.py",
> line 56, in _get_current
> url = "%s" % self.api.client.get_endpoint()
> File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py",
> line 132, in get_endpoint
> return self.session.get_endpoint(auth or self.auth, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py",
> line 634, in get_endpoint
> return auth.get_endpoint(self, **kwargs)
> File
> "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py",
> line 209, in get_endpoint
> service_catalog = self.get_access(session).service_catalog
> File
> "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py",
> line 135, in get_access
> self.auth_ref = self.get_auth_ref(session)
> File
> "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line
> 64, in get_auth_ref
> authenticated=False, log=False)
> File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py",
> line 545, in post
> return self.request(url, 'POST', **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/_utils.py",
> line 180, in inner
> return func(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py",
> line 440, in request
> raise exceptions.from_response(resp, method, url)
> NotFound: The resource could not be found. (HTTP 404)
>
>
>
> Bob Hansen
> z/VM OpenStack Enablement
>
>
> [image: Inactive hide details

Re: [openstack-dev] [Third-Party][CI] Add accouts to CI group

2015-12-25 Thread Mikhail Medvedev 
Hi Watanabe,

Fujitsu ETERNUS FC CI account has been added to the gerrit group.


On Thu, Dec 24, 2015 at 10:16 PM, Watanabe, Isao <
watanabe_i...@jp.fujitsu.com> wrote:

> Hello, Third-Party Coordinators
>
> Sorry for sending the wrong information.
> Please ignore the previous mail.
>
> Could you add the following account to CI mail list group, please?
> Name: Fujitsu ETERNUS FC CI
> Mail: lsoft-openstac...@ml.css.fujitsu.com
>
> Best regards,
> Watanabe.isao
>
> > -Original Message-
> > From: Watanabe, Isao [mailto:watanabe_i...@jp.fujitsu.com]
> > Sent: Friday, December 25, 2015 12:36 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [Third-Party][CI] Add accouts to CI group
> >
> > Hello, Third-Party Coordinators
> >
> > Could you add the following account to CI mail list group, please?
> > Name: Fujitsu C-Fabric CI
> > Mail: lsoft-cfabri...@ml.css.fujitsu.com
> >
> > Best regards,
> > Watanabe.isao
> >
> >
> >
> > > -Original Message-
> > > From: Watanabe, Isao [mailto:watanabe_i...@jp.fujitsu.com]
> > > Sent: Friday, October 16, 2015 8:52 PM
> > > To: openstack-dev@lists.openstack.org
> > > Subject: [openstack-dev] [Third-Party][CI] Add accouts to CI group
> > >
> > > Hello, Third-Party Coordinators
> > > Cc：Ramy
> > >
> > > Could you add the following 2 accounts to CI group, please?
> > > Also, we will continually update the wiki.
> > > [1]
> > > Name: Fujitsu C-Fabric CI
> > > Mail: lsoft-cfabri...@ml.css.fujitsu.com
> > > Wiki:
> > > https://wiki.openstack.org/wiki/ThirdPartySystems/Fujitsu_C-Fabric_CI
> > >
> > > [2]
> > > Name: Fujitsu IRMC CI
> > > Mail: lsoft-irm...@ml.css.fujitsu.com
> > > Wiki:
> https://wiki.openstack.org/wiki/ThirdPartySystems/Fujitsu_IRMC_CI
> > >
> > >
> > > Best regards,
> > > Watanabe.isao
> > >
> > >
> > >
> > >
> > >
> > 
> > > __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ThirdParty][CI] [patch] Status page at http://ci-watch.tintri.com/project

2015-12-21 Thread Mikhail Medvedev 
On Dec 21, 2015 10:19 AM, "Asselin, Ramy"  wrote:
>
> Hi Phillip,
>
> Yes, please offer a patch to that repo Anita suggested. There's a small
group of us actively working on improving the code and eventually getting
ci-watch deployed in openstack.org. Your patch and help would be very much
appreciated.

+1

>
> We also meet bi-weekly in the 3rd party ci working group:
https://wiki.openstack.org/wiki/Meetings/ThirdParty
> We also discuss some issues in #openstack-third-party-ci
>
> Thanks!
> Ramy
>
> -Original Message-
> From: Anita Kuno [mailto:ante...@anteaya.info]
> Sent: Monday, December 21, 2015 6:52 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [ThirdParty][CI] [patch] Status page at
http://ci-watch.tintri.com/project
>
> On 12/21/2015 09:20 AM, Philipp Marek wrote:
> > Hi all,
> >
> > I quite like the page at http://ci-watch.tintri.com/project - it gives
> > a very quick overview about the failures one should look into, and
> > which to ignore ;)
> >
> >
> > Please let me state before anything else that I don't know any of the
> > restrictions that may have led into the current design - it's very
> > likely that I'm just missing a few points, and that some or all of my
> > comments below are invalid anyway. As always, take enough salt!
> >
> >
> > One thing about that page that is bothering me is the performance...
> > my
> > (current) Firefox asks me several times whether I'd like to stop the
> > JS, or whether it should be allowed to continue.
> >
> > With this patch (and a local exported copy of the page) I don't get
> > asked about that any more; it seems to give me a speedup of ~200, as
> > no intermediate lists need to be built and filtered any more:
> >
> > $ diff -u verified.js.orig verified.js
> > --- verified.js.orig2015-12-21 15:03:45.614529924 +0100
> > +++ verified.js 2015-12-21 15:03:36.114432601 +0100
> > @@ -33,9 +33,9 @@
> >  $(document).ready(function () {
> >$("colgroup").each(function (i, elem) {
> >  if ($(elem).hasClass("verified-1")) {
> > -  $("#results").find("td").filter(":nth-child(" + (i + 1) +
")").addClass("verified-1");
> > +  $("#results td:nth-child(" + (i + 1) +
> > + ")").addClass("verified-1");
> >  } else if ($(elem).hasClass("verified1")) {
> > -  $("#results").find("td").filter(":nth-child(" + (i + 1) +
")").addClass("verified1");
> > +  $("#results td:nth-child(" + (i + 1) +
> > + ")").addClass("verified1");
> >  }
> >});
> >$("#verified1-button").on("click", toggle_verified_plus);
> >
> >
> > Furthermore, I'm wondering whether
> >
> > 
> > 
> > 
> > 
> > 
> >
> > couldn't be simplified to
> >
> > 
> > 
> > 
> > 
> >
> > with the rest being done via CSS? Perhaps a  would be needed
> > within the  to get the vertical size right, but everything else
> > should be possible via CSS, I believe.
> >
> > This change should reduce the size of the generated HTML big some 50%
> > or so, too.
> >
> >
> >
> > Thanks for listening - if you disagree, please ignore and continue
> > working on something else ;)
> >
> >
> > Regards,
> >
> > Phil
> >
> >
> > __
> >  OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> The repo is here if you would like to offer your patch via Gerrit.
> http://git.openstack.org/cgit/openstack-infra/ciwatch/
>
> Thanks Philipp,
> Anita.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Learning to Debug the Gate

2015-11-18 Thread Mikhail Medvedev 
We had a mini tutorial today in #openstack-infra, were Clark Boylan
explained how one can bring up an environment to debug logstash-2.0.
This is tangential to "debugging the gate", but still could be useful to
better understand logstash/es pipeline. 

I did condense the conversation into the doc, excuse any grammar /
punctuation that I missed:

http://paste.openstack.org/show/479346/

-- 
Mikhail Medvedev

On Tue, Nov 10, 2015, at 10:21, Sean Dague wrote:
> There is also a neutron issue which is causing a 35% failure rate in
> neutron jobs -
> http://tinyurl.com/ne3ex4v
> 
> That one still needs resolution.
> 
> On 11/10/2015 10:54 AM, Davanum Srinivas wrote:
> > Took about 35 mins or so :)
> > 
> > -- Dims
> > 
> > On Tue, Nov 10, 2015 at 10:45 AM, Matt Riedemann
> > <mrie...@linux.vnet.ibm.com> wrote:
> >>
> >>
> >> On 11/9/2015 3:54 PM, Anita Kuno wrote:
> >>>
> >>> On 11/05/2015 07:45 PM, Anita Kuno wrote:
> >>>>
> >>>> On 11/03/2015 05:30 PM, Anita Kuno wrote:
> >>>>>
> >>>>> On 11/02/2015 12:39 PM, Anita Kuno wrote:
> >>>>>>
> >>>>>> On 10/29/2015 10:42 PM, Anita Kuno wrote:
> >>>>>>>
> >>>>>>> On 10/29/2015 08:27 AM, Anita Kuno wrote:
> >>>>>>>>
> >>>>>>>> On 10/28/2015 12:14 AM, Matt Riedemann wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 10/27/2015 4:08 AM, Anita Kuno wrote:
> >>>>>>>>>>
> >>>>>>>>>> Learning how to debug the gate was identified as a theme at the
> >>>>>>>>>> "Establish Key Themes for the Mitaka Cycle" cross-project session:
> >>>>>>>>>> https://etherpad.openstack.org/p/mitaka-crossproject-themes
> >>>>>>>>>>
> >>>>>>>>>> I agreed to take on this item and facilitate the process.
> >>>>>>>>>>
> >>>>>>>>>> Part one of the conversation includes referencing this video
> >>>>>>>>>> created by
> >>>>>>>>>> Sean Dague and Dan Smith:
> >>>>>>>>>> https://www.youtube.com/watch?v=fowBDdLGBlU
> >>>>>>>>>>
> >>>>>>>>>> Please consume this as you are able.
> >>>>>>>>>>
> >>>>>>>>>> Other suggestions for how to build on this resource were mentioned
> >>>>>>>>>> and
> >>>>>>>>>> will be coming in the future but this was an easy, actionable first
> >>>>>>>>>> step.
> >>>>>>>>>>
> >>>>>>>>>> Thank you,
> >>>>>>>>>> Anita.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> __
> >>>>>>>>>>
> >>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>>>>>>> Unsubscribe:
> >>>>>>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/tales-from-the-gate-how-debugging-the-gate-helps-your-enterprise
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> The source for the definition of "the gate":
> >>>>>>>>
> >>>>>>>> http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n34
> >>>>>>>>
> >>>>>>>> Thanks for following along,
> >>>>>>>> Anita.
> >>>>>>>>
> >>>>>>>
> >>>>>>> This is the status page showing the status of our running jobs,
>