Re: [OpenStack-Infra] git review -d without check out

[Tony Breeds]

On Tue, Mar 27, 2018 at 02:16:32PM +0300, Ram Rachum wrote:
> Hi,
> 
> Is there a way to do `git review -d` without having it do a checkout? i.e.
> I just want to have these commits in my Git database so I could cherrypick
> them on some other branch. We've got tons of submodules so checking out
> often creates problems.
> 
> I tried `git fetch gerrit my_commit_hash:temporary_branch_name` but that's
> insanely slow for some reason.

Thanks more or less what I do.

This is my ~bin/git-os-change
---
#!/usr/bin/env bash

review=$1
revision=${2:-1}

if [ -z "$review" ] ; then
echo Need an OpenStack gerrit review number >&2
exit 1
fi

ref=$(printf "refs/changes/%d/%d/%d" "${review: -2}" "${review}" "${revision}")
git fetch gerrit "${ref}:${ref}"
---

It's about the same speed as git review.

---
[tony@thor ~]$ cd tmp
/home/tony/tmp
[tony@thor tmp]$ cp -a ~/projects/openstack/openstack-dev/devstack devstack-1
[tony@thor tmp]$ cp -a ~/projects/openstack/openstack-dev/devstack devstack-2
[tony@thor tmp]$ cd devstack-1
/home/tony/tmp/devstack-1
[tony@thor devstack-1]$ time git review -d 554977
Downloading refs/changes/77/554977/3 from gerrit
Switched to branch "review/eric_berglund/etcd_version"

real0m3.892s
user0m0.508s
sys 0m0.225s
[tony@thor devstack-1]$ cd ../devstack-2
[tony@thor devstack-2]$ time git os-change 554977 3
remote: Counting objects: 7, done
remote: Finding sources: 100% (4/4)
remote: Total 4 (delta 3), reused 3 (delta 3)
Unpacking objects: 100% (4/4), done.
From ssh://review.openstack.org:29418/openstack-dev/devstack
 * [new ref]   refs/changes/77/554977/3 -> refs/changes/77/554977/3

real0m2.813s
user0m0.182s
sys 0m0.152s
---

Yours Tony.


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Options for logstash of ansible tasks

[James E. Blair]

Ian Wienand  writes:

> The closest other thing I could find was "aggregate" [1]; but this
> relies on having a unique task-id to group things together with.
> Ansible doesn't give us this in the logs and AFAIK doesn't have a
> concept of a uuid for tasks.

We control the log output format in Zuul (both job-output.txt and
job-output.json).  So we could include a unique ID for tasks if we
wished.  However, we should not put that on every line, so that still
would require some handling in the log processor.

As soon as I say that, it makes me think that the solution to this
really should be in the log processor.  Whether it's a grok filter, or
just us parsing the lines looking for task start/stop -- that's where we
can associate the extra data with every line from a task.  We can even
generate a uuid right there in the log processor.

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] Options for logstash of ansible tasks

[Ian Wienand]

I wanted to query for a failing ansible task; specifically what would
appear in the console log as

 2018-03-27 15:07:49.294630 | 
 2018-03-27 15:07:49.295143 | TASK [configure-unbound : Check for IPv6]
 2018-03-27 15:07:49.368062 | primary | skipping: Conditional result was False
 2018-03-27 15:07:49.400755 | 

While I can do

 message:"configure-unbound : Check for IPv6"

I want to correlate that with a result, looking also for the matching

 skipping: Conditional result was False

as the result of the task.  AFAICT, there is no way in kibana to
enforce a match on consecutive lines like this (as it has no concept
they are consecutive).

I considered a few things.  We could conceivably group everything
between "TASK" and a blank " | " into a single entry with a multiline
filter.  It was pointed out that this would make, for example, the
entire devstack log as a single entry, however.

The closest other thing I could find was "aggregate" [1]; but this
relies on having a unique task-id to group things together with.
Ansible doesn't give us this in the logs and AFAIK doesn't have a
concept of a uuid for tasks.

So I'm at a bit of a loss as to how we could effectively index ansible
tasks so we can determine the intermediate values or results of
individual tasks?  Any ideas?

-i

[1] 
https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Problems setting up my own OpenStack Infrastructure

[Bernd Bausch]

Resending this message because it was too large for the distribution list.

---

Clark,

My first test uses this local.pp. It's copied verbatim from [1]:

# local.pp
class { 'openstack_project::etherpad':
  ssl_cert_file_contents  => hiera('etherpad_ssl_cert_file_contents'),
  ssl_key_file_contents   => hiera('etherpad_ssl_key_file_contents'),
  ssl_chain_file_contents => hiera('etherpad_ssl_chain_file_contents'),
  mysql_host  => hiera('etherpad_db_host', 'localhost'),
  mysql_user  => hiera('etherpad_db_user', 'etherpad'),
  mysql_password  => hiera('etherpad_db_password','etherpad'),
}

The commands I run are also verbatim from the same page:

# ./install_puppet.sh
# ./install_modules.sh
# puppet apply -l /tmp/manifest.log --modulepath=modules:/etc/puppet/modules 
manifests/local.pp


My second test closely follows [2]. Here, I take the puppetmaster's original 
site.pp, adapt the domain "openstack.org" to my domain at home and remove all 
node definitions except puppetmaster and etherpad. My file is at the end of 
this message[4].

The commands:

# ./install_puppet.sh
# ./install_modules.sh
# vi site.pp  # see [4]
# puppet 
apply --modulepath='/opt/system-config/production/modules:/etc/puppet/modules' 
 -e 'include openstack_project::puppetmaster'


> Generally though hiera is used for anything that will be secret or very site 
> specific. So in this case the expectation is that you will set up a hiera 
> file with the info specific for your deployment (because you shouldn't have 
> the ssl cert private data for our deployment and we shouldn't have yours).
> This is likely a missing set of info for our docs. We should add something 
> with general hiera setup to get people going.

Yes. The documentation seems to treat the hiera as a given; it just exists, 
and there doesn't seem to be any information about its content or even whether 
it's really required.
Once I know the issues and technology better (steep learning curve), I'd be 
happy to write documentation from the perspective of a newbie.
For now, let me do more testing with hardcoded values rather than hiera. I 
certainly learn a lot doing this.

> Unfortunately I don't remember off the top of my head how to set up a hiera 
> so I will have to dig into docs (or maybe someone else can chime in with 
> that info).

In principle, I can do that (for Puppet 4 at least), but the question is what 
goes into the OpenStack CI production hiera. I see a directory 
/opt/system-config/production/hiera [3] - is that it? It doesn't contain 
anything about Etherpad, though. I also did a codesearch for 
"etherpad_ssl_cert_file_contents", no result (except for the site.pp).

Thanks much, Clark!

Bernd
---
[1] 
https://docs.openstack.org/infra/system-config/sysadmin.html#making-a-change-in-puppet
[2] https://docs.openstack.org/infra/system-config/puppet.html
[3] https://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera
[4] My site.pp:

#
# Top-level variables
#
# There must not be any whitespace between this comment and the variables or
# in between any two variables in order for them to be correctly parsed and
# passed around in test.sh
#
$elasticsearch_nodes = hiera_array('elasticsearch_nodes')

#
# Default: should at least behave like an openstack server
#
node default {
  class { 'openstack_project::server':
sysadmins => hiera('sysadmins', []),
  }
}

# Node-OS: trusty
# (I try this with Centos 7 first)
node 'puppetmaster.home' {
  class { 'openstack_project::server':
iptables_public_tcp_ports => [8140],
sysadmins => hiera('sysadmins', []),
pin_puppet=> '3.6.',
  }
  class { 'openstack_project::puppetmaster':
root_rsa_key   => 
hiera('puppetmaster_root_rsa_key'),
puppetmaster_clouds=> 
hiera('puppetmaster_clouds'),
enable_mqtt=> true,
mqtt_password  => 
hiera('mqtt_service_user_password'),
mqtt_ca_cert_contents  => 
hiera('mosquitto_tls_ca_file'),
  }
  file { '/etc/openstack/infracloud_vanilla_cacert.pem':
ensure  => present,
owner   => 'root',
group   => 'root',
mode=> '0444',
content => hiera('infracloud_vanilla_ssl_cert_file_contents'),
require => Class['::openstack_project::puppetmaster'],
  }
  file { '/etc/openstack/infracloud_chocolate_cacert.pem':
ensure  => present,
owner   => 'root',
group   => 'root',
mode=> '0444',
content => hiera('infracloud_chocolate_ssl_cert_file_contents'),
require => Class['::openstack_project::puppetmaster'],
  }
  file { '/etc/openstack/limestone_cacert.pem':
ensure  => present,
owner   => 'root',
group   => 'root',
mode=> '0444',
content => hiera('limestone_ssl_cert_file_contents'),
require => 

Re: [OpenStack-Infra] Adding new etcd binaries to tarballs.o.o

[Jeremy Stanley]

On 2018-03-28 08:22:53 +1100 (+1100), Ian Wienand wrote:
[...]
> Anyway, we got there in the end :) I'll add to my todo list to clear
> them from tarballs.o.o once this settles out.
[...]

Thanks! Excellent news indeed.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Adding new etcd binaries to tarballs.o.o

[Ian Wienand]

On 03/28/2018 01:04 AM, Jeremy Stanley wrote:

I would be remiss if I failed to remind people that the *manually*
installed etcd release there was supposed to be a one-time stop-gap,
and we were promised it would be followed shortly with some sort of
job which made updating it not-manual. We're coming up on a year and
it looks like people have given in and manually added newer etcd
releases at least once since. If this file were important to
testing, I'd have expected someone to find time to take care of it
so that we don't have to. If that effort has been abandoned by the
people who originally convinced us to implement this "temporary"
workaround, we should remove it until it can be supported properly.


In reality we did fix it, as described with the
use-from-cache-or-download changes in the prior mail.  I even just
realised I submitted and forgot about [1] which never got reviewed to
remove the tarballs.o.o pointer -- that setting then got copied into
the new devstack zuulv3 jobs [2].

Anyway, we got there in the end :) I'll add to my todo list to clear
them from tarballs.o.o once this settles out.

-i

[1] https://review.openstack.org/#/c/508022/
[2] https://review.openstack.org/#/c/554977/

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Adding new etcd binaries to tarballs.o.o

[Jeremy Stanley]

On 2018-03-27 10:39:35 +1100 (+1100), Tony Breeds wrote:
[...]
> Except something sets ETCD_DOWNLOAD_URL to tarballs.o.o
[...]

I would be remiss if I failed to remind people that the *manually*
installed etcd release there was supposed to be a one-time stop-gap,
and we were promised it would be followed shortly with some sort of
job which made updating it not-manual. We're coming up on a year and
it looks like people have given in and manually added newer etcd
releases at least once since. If this file were important to
testing, I'd have expected someone to find time to take care of it
so that we don't have to. If that effort has been abandoned by the
people who originally convinced us to implement this "temporary"
workaround, we should remove it until it can be supported properly.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] git review -d without check out

[Ram Rachum]

Hi,

Is there a way to do `git review -d` without having it do a checkout? i.e.
I just want to have these commits in my Git database so I could cherrypick
them on some other branch. We've got tons of submodules so checking out
often creates problems.

I tried `git fetch gerrit my_commit_hash:temporary_branch_name` but that's
insanely slow for some reason.


Thanks,
Ram.
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra