[OpenDaylight Infrastructure] [opendaylight.org #33382] Jenkins queue at 500+ entries

[Anil Belur via RT]

On Mon Nov 28 11:39:58 2016, lorand.jakab+...@gmail.com wrote:
> Well, we had a few jobs fail with "java.util.concurrent.ExecutionException:
> org.jclouds.rest.AuthorizationException: {"forbidden": {"message": "Quota
> exceeded for instances: Requested 1, but already used 250 of 250
> instances", "code": 403}}". Not sure about lost jobs though...
> 
> On Mon, Nov 28, 2016 at 6:31 PM, Jamo Luhrsen  wrote:
> 
> > Lori,
> >
> > Somehow it seems to be better now.  I just looked for the first time today
> > and the queue is only ~40
> >
> > I wonder if we lost a bunch of jobs somehow?
> >
> > JamO
> >
> > On 11/28/2016 02:41 AM, Lori Jakab wrote:
> > > Hi,
> > >
> > > Just an FYI, the Jenkins queue is currently at over 500 entries, so jobs
> > may take a while, can someone please take a look?
> > >
> > > Thanks,
> > > -Lori
> > >
> > >
> > > ___
> > > dev mailing list
> > > d...@lists.opendaylight.org
> > > https://lists.opendaylight.org/mailman/listinfo/dev
> > >
> >

Hello, 

We had to restart Jenkins yesterday, since the stale jobs (running for ~23 
hours) were not getting cleaned up. 
My apologies for the delay in updating the ticket. 

Thanks,
Anil
___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure

[OpenDaylight Infrastructure] [opendaylight.org #33382] [linuxfoundation.org #33382] Re: [opendaylight-dev] Jenkins queue at 500+ entries

[Lori Jakab via RT]

Well, we had a few jobs fail with "java.util.concurrent.ExecutionException:
org.jclouds.rest.AuthorizationException: {"forbidden": {"message": "Quota
exceeded for instances: Requested 1, but already used 250 of 250
instances", "code": 403}}". Not sure about lost jobs though...

On Mon, Nov 28, 2016 at 6:31 PM, Jamo Luhrsen  wrote:

> Lori,
>
> Somehow it seems to be better now.  I just looked for the first time today
> and the queue is only ~40
>
> I wonder if we lost a bunch of jobs somehow?
>
> JamO
>
> On 11/28/2016 02:41 AM, Lori Jakab wrote:
> > Hi,
> >
> > Just an FYI, the Jenkins queue is currently at over 500 entries, so jobs
> may take a while, can someone please take a look?
> >
> > Thanks,
> > -Lori
> >
> >
> > ___
> > dev mailing list
> > d...@lists.opendaylight.org
> > https://lists.opendaylight.org/mailman/listinfo/dev
> >
>

___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure

Re: [OpenDaylight Infrastructure] [opendaylight-dev] Jenkins queue at 500+ entries

[Lori Jakab]

Well, we had a few jobs fail with "java.util.concurrent.ExecutionException:
org.jclouds.rest.AuthorizationException: {"forbidden": {"message": "Quota
exceeded for instances: Requested 1, but already used 250 of 250
instances", "code": 403}}". Not sure about lost jobs though...

On Mon, Nov 28, 2016 at 6:31 PM, Jamo Luhrsen  wrote:

> Lori,
>
> Somehow it seems to be better now.  I just looked for the first time today
> and the queue is only ~40
>
> I wonder if we lost a bunch of jobs somehow?
>
> JamO
>
> On 11/28/2016 02:41 AM, Lori Jakab wrote:
> > Hi,
> >
> > Just an FYI, the Jenkins queue is currently at over 500 entries, so jobs
> may take a while, can someone please take a look?
> >
> > Thanks,
> > -Lori
> >
> >
> > ___
> > dev mailing list
> > d...@lists.opendaylight.org
> > https://lists.opendaylight.org/mailman/listinfo/dev
> >
>
___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure

[OpenDaylight Infrastructure] Jenkins sandbox jobs not working - quota full

[Kochba, Alon]

Hi,

It seems that the Jenkins sandbox hasn't recovered from the maintenance, 
possibly something needs to be cleared.
It looks like there are no executors in use but when trying to boot we get an 
exception due to Quota exceeded [1], as if 250 instances are already in use.

Please update once resolved, feel free to retry [1] to verify.

09:04:14 Queuing cloud instance: #0 1, ODLPUB centos7-java-builder-2c-4g
09:04:14 Queuing cloud instance: #0 3, ODLPUB centos7-devstack-2c-8g
09:04:14 Queuing cloud instance: #1 3, ODLPUB centos7-devstack-2c-8g
09:04:14 Queuing cloud instance: #2 3, ODLPUB centos7-devstack-2c-8g
09:04:19 ERROR: Exception creating a node: 
org.jclouds.compute.RunNodesException: error running 1 node 
group(centos7-devstack-2c-8g) location(DFW) 
image(926a98a2-0b89-4a8c-8aff-95463f3dcb62) size(general1-8) 
options({scriptPresent=true, userMetadata={Name=centos7-devstack-2c-8g}, 
networks=[cac67a72-aefc-48f8-ae55-9affa3540dd0], autoAssignFloatingIp=false, 
keyPairName=sandbox, userData=[B@71cee215, configDrive=false})
09:04:19 Execution failures:
09:04:19
09:04:19 1) ExecutionException on centos7-devstack-2c-8g-5f0:
09:04:19 java.util.concurrent.ExecutionException: 
org.jclouds.rest.AuthorizationException: {"forbidden": {"message": "Quota 
exceeded for instances: Requested 1, but already used 250 of 250 instances", 
"code": 403}}

[1] 
https://jenkins.opendaylight.org/sandbox/job/netvirt-csit-1node-openstack-mitaka-alonko-upstream-transparent-boron/2/consoleFull

Thanks,
--alon

___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure

Re: [OpenDaylight Infrastructure] [opendaylight-dev] Jenkins queue at 500+ entries

[Jamo Luhrsen]

Lori,

Somehow it seems to be better now.  I just looked for the first time today and 
the queue is only ~40

I wonder if we lost a bunch of jobs somehow?

JamO

On 11/28/2016 02:41 AM, Lori Jakab wrote:
> Hi,
> 
> Just an FYI, the Jenkins queue is currently at over 500 entries, so jobs may 
> take a while, can someone please take a look?
> 
> Thanks,
> -Lori
> 
> 
> ___
> dev mailing list
> d...@lists.opendaylight.org
> https://lists.opendaylight.org/mailman/listinfo/dev
> 
___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure

[OpenDaylight Infrastructure] [opendaylight.org #33382] [linuxfoundation.org #33382] Re: [opendaylight-dev] Jenkins queue at 500+ entries

[James Luhrsen via RT]

Lori,

Somehow it seems to be better now.  I just looked for the first time today and 
the queue is only ~40

I wonder if we lost a bunch of jobs somehow?

JamO

On 11/28/2016 02:41 AM, Lori Jakab wrote:
> Hi,
> 
> Just an FYI, the Jenkins queue is currently at over 500 entries, so jobs may 
> take a while, can someone please take a look?
> 
> Thanks,
> -Lori
> 
> 
> ___
> dev mailing list
> d...@lists.opendaylight.org
> https://lists.opendaylight.org/mailman/listinfo/dev
> 

___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure

[OpenDaylight Infrastructure] Jenkins queue at 500+ entries

[Lori Jakab]

Hi,

Just an FYI, the Jenkins queue is currently at over 500 entries, so jobs
may take a while, can someone please take a look?

Thanks,
-Lori
___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure

Re: [OpenDaylight Infrastructure] Opentack- access to VMs console

[Jose Corbacho Gil]

Hi,

I am going to auto answer to my original mail because I was wrong. The issue is 
not due to the ovs-dpdk, it is related to an DHCP feature that we are 
internally added to our ODL flavor (something already ported to the community) 
and for such reason we are disabling the neutron dhcp service.

I have heard that in Boron it has been introduced DHCP services for IPv6. Are 
you still using the neutron DHCP service?

Thanks,
Jose

From: Jose Corbacho Gil
Sent: miércoles, 23 de noviembre de 2016 18:06
To: infrastructure@lists.opendaylight.org; 
integration-...@lists.opendaylight.org
Cc: David Lopez Muñoz ; Satish Kumar 

Subject: Opentack- access to VMs console

Hi,

We are working with a openstack release using OVS-DPDK as the non DPDK option 
does not provide the required performance for Data plane VNFs (close to 
10Gbps). Due to the above OVS version, the neutron dhcpd service is not running 
and it is not possible to have IP connectivity to nova VMs by using the qdhcpd 
namespaces (i.e. sudo ip netns exec qdhcp-${net_id} ping -c 3 ${vm_ip}20s).

Based on the above limitation, we are using a serial text based console (no the 
VNC console that you can get by executing "nova get-vnc-console  
novnc"), however we need to configure our nova VMs for having kernel 
redirection to a second serial port, as the first serial console is being used 
by nova console-log service. See the following serial section of a dumpxml nova 
VM (using default configuration):


  
  


  


  
  



These are the instructions to enable the kernel redirection to a second serial 
port:

Ubuntu

1.   sudo -i

2.   cp /etc/init/ttyS0.conf  /etc/init/ttyS1.conf

3.   vi /etc/init/ttyS1.conf

a.   Change the line "exec /sbin/getty -L ttyS0 115200 vt102"

b.   By "exec /sbin/getty -L ttyS1 115200 vt102"

4.   Reboot VM to ensure the changes

5.   desde el compute: virsh reboot 

6.   cuando la VM levante: virsh console  serial1

SLES

-  In /etc/inittab add a second serial port by adding the following line

o   ttyS1::respawn:/sbin/getty -L 115200 ttyS1 vt100 # GENERIC_SERIAL

-  Reload inititab

o   "kill -HUP 1"


After that, we can get a serial text based VM console by executing "virsh 
console  serial1 --force" and using an expect script we are parsing the 
output of several commands at VM level.

Note: We cannot use the serial0 (default option when not specifying the serial 
option) as the nova console-log service is using it.


Example of virsh console connection using default serial0 port:

root@node-26:~# virsh console 53
Connected to domain instance-027a
Escape character is ^]
error: internal error: character device  is not using a PTY

Example of virsh console connection using serial1 port:

root@node-26:~# virsh console 53 serial1
Connected to domain instance-027a
Escape character is ^]

Ubuntu 14.04.5 LTS mtc ttyS1

mtc login:



We have checked that in the library OpenStackOperations.robot you are using the 
nova console-log service to be sure that the VM has got the right IP address, 
so serial0 is already used in your setup and tests.

Verify VMs Received DHCP Lease
[Arguments]@{vm_list}
[Documentation]Using nova console-log on the provided ${vm_name} to 
search for the string "obtained" which
...correlates to the instance receiving it's IP address via DHCP. This 
should provide a good indication
...that the instance is fully up and ready.


What is your VM connectivity model suggested for an environment using OVS-DPDK?
Base on the previous lines, for us we should go for the "virsho console 
serial1" option.
I have seen other approaches using "nova get-serial-console" but it appears 
that is not compatible with nova console-log service:

-  
http://blog.oddbit.com/2014/12/22/accessing-the-serial-console-of-your-nova-servers/

-  http://docs.openstack.org/developer/nova/testing/serial-console.html


Thanks,
Jose




[Ericsson]
JOSE CORBACHO GIL
Services SDN  - Test Envrionment Responsible
BICP DUAC SDN PDG M2 SDN ODL Design & CI
BU IT & Cloud Products

Ericsson
Via de los Poblados, 13
28033 - Madrid, Spain
Phone +34 91 339 40 75
Mobile +34 650 86 85 61
jose.corbacho@ericsson.com
www.ericsson.com


[http://www.ericsson.com/current_campaign]

Legal entity: ERICSSON ESPAÑA S.A., registered office in Spain. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer

___
infrastructure mailing list
infrastructure@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/infrastructure