not enough disk on git.fedorahosted.org

[Miroslav Suchý]

Please delete something or buy another disk :)

We got there email hook, which probobly do not have enough disk space on 
some mount.


[msu...@dri/~/rhn/spacewalk.pub/schema/spacewalk]$ git push --tags
Counting objects: 1, done.
Writing objects: 100% (1/1), 229 bytes, done.
Total 1 (delta 0), reused 0 (delta 0)
Traceback (most recent call last):
   File /usr/bin/send-unicode-email.py, line 32, in ?
 smtp.sendmail(sender, recipients, msg.as_string())
   File /usr/lib/python2.4/smtplib.py, line 680, in sendmail
 raise SMTPSenderRefused(code, resp, from_addr)
smtplib.SMTPSenderRefused: (452, '4.3.1 Insufficient system storage', 
'=?utf-8?q?Miroslav_Such=C3=BD?= msu...@fedoraproject.org')
To ssh://msu...@git.fedorahosted.org/git/spacewalk.git/
  * [new tag] spacewalk-schema-0.9.4-1 - spacewalk-schema-0.9.4-1


-- 
Miroslav Suchy
Red Hat Satellite Engineering
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

www and git of fedoraproject.org do not respond

[Miroslav Suchý]

www and git service of fedoraproject.org time-outs.
Host responds to ping.
-- 
Miroslav Suchy
Red Hat Satellite Engineering
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: www and git of fedorahosted.org do not respond

[Miroslav Suchý]

On 11/09/2010 09:13 AM, Ricky Zhou wrote:
 wget -SO/dev/nullhttp://fedoraproject.org/

Did I say fedoraproject? I'm dumb, sorry. I meant fedorahosted.org.

$ wget -SO/dev/null https://fedorahosted.org/
--2010-11-09 09:24:06--  https://fedorahosted.org/
Resolving fedorahosted.org... 66.135.52.17
Connecting to fedorahosted.org|66.135.52.17|:443...

 and waits


-- 
Miroslav Suchy
Red Hat Satellite Engineering
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Summary/Minutes from today's Fedora Infrastructure meeting (2013-09-12)

[Miroslav Suchý]

On 09/12/2013 09:51 PM, Kevin Fenzi wrote:

19:39:45 pingou I think that's the idea
19:39:51 pingou make a first release of copr
19:39:55 pingou have people start to use it
19:40:01 pingou integrate it into koji later on


Yes this is true.
I am using current code and polishing it for public roll out.
After it will go public, I will start next phase when I will try to merge it 
with koji code. But that is far future.

My goal is to go public at the end of this month or begging of next one. We 
will see.

I want to join your meetings, I will start with next one.
Hopefully I will manage to prepare RFR by that time.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Summary/Minutes from today's Fedora Infrastructure meeting (2013-09-12)

[Miroslav Suchý]

On 09/13/2013 06:40 PM, Kevin Fenzi wrote:

I think we can pretty much look at opening it up where it is, then see
how things are resource wise before announcing it more widely?


*nod*
Only thing what worries me is disk space.
Currently I have from Fedora Cloud 200 GB. According to my calculations this 
will last 20 days.
So I'm looking for more disk space now. I know I can get little bit more from Fedora Cloud, but I'm looking for 
something in order of terabytes. I have some negotiations running right now, we will see...

But if you know about few terabytes which are laying around, then let me know :)

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Migrating to our own bugzilla instance.

[Miroslav Suchý]

On 09/17/2013 12:37 PM, Jóhann B. Guðmundsson wrote:

since my frustration level with RH bugzilla has grown to an all time high due 
to frequent collision with internal RH
administrative policy's that nobody in the community knows exactly which are,


Can you please elaborate which Red Hat policy collide with Fedora needs? I did not have such experience, so I'm really 
curious.



frequent RH employement mistakes in bug
handling between Fedora and RHEL


That is because those people work on RHEL and Fedora. And they will continue on that even if you split BZ into two 
instances. It will be still those same humans and they will be making same mistakes. I doubt that having two instances 
will help here.



as well as several other issue we are faced with it in the QA community and the
hindrance it serves to the growth to our community and


Can you be specific here, please?


the fact we cant hack in it directly to make ours as well as
other processes work smoothly which makes everybody's life easier.


But it give fedora infrastructure team more free time, which you can spend on some other projects (and we have plenty of 
them).

If you want to hack BZ, you can hack it in upstream:
  http://www.bugzilla.org/
all changes done there will land in bugzilla.redhat.com sooner or later.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Summary/Minutes from today's Fedora Infrastructure meeting (2013-09-12)

[Miroslav Suchý]

On 09/16/2013 04:58 PM, Kevin Fenzi wrote:

When we first setup our cloud, we setup the storage for volumes on just
the head node. The other 5 nodes in the mix also have storage. Over
time we added another nodes storage to it, but haven't done anything
with the other 4. Thats a bit spread out however, volumes can only be
on one of them, so while it allows for more volumes, it doesn't allow
for one really large volume.


Ideal setup for GlusterFS, I would say.


Our cloud setup is isolated from the rest of the network in the
datacenter it's at. So, for example it couldn't talk to any of our
netapp storage or the like.


:(

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Migrating to our own bugzilla instance.

[Miroslav Suchý]

On 09/18/2013 04:20 PM, Jóhann B. Guðmundsson wrote:

On 09/17/2013 12:37 PM, Jóhann B. Guðmundsson wrote:

since my frustration level with RH bugzilla has grown to an all time high due 
to frequent collision with internal RH
administrative policy's that nobody in the community knows exactly which are,


Can you please elaborate which Red Hat policy collide with Fedora needs?


Provide me that policy list and I will point them out.


:)
Red Hat have policies only for Red Hat products. Red Hat have no policy for 
Fedora.
So I'm really curious what happened to you (or somebody else) that you are saying you are frustrated. There must be some 
story behind, right?



frequent RH employement mistakes in bug
handling between Fedora and RHEL


That is because those people work on RHEL and Fedora. And they will continue on 
that even if you split BZ into two
instances. It will be still those same humans and they will be making same 
mistakes. I doubt that having two instances
will help here.


Yes it will and RHEL != Fedora so stop acting like it does.


I did not said that they are equal! I said that a lot of Red Hat people spend a 
lot of their time working on Fedora.
And if developer maintain some package in some Red Hat product, in Fedora and in EPEL (which is part of Fedora) - I can 
imagine that if you have same BZ opened to all three products and I you want to flip BZ to different state, developer 
can make error and make it for different product (but same component) than you intended. This can happen. And having 
this scenario on my mind I do not know how different instance of BZ would help this.
But maybe you have different scenario on your mind. So can you elaborate on frequent RH employement mistakes in bug 
please?


--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Introduction

[Miroslav Suchý]

Hi,
I've been on this list for a while, but mostly passive. I want to change that. 
So let me formally introduce myself.

My name is Miroslav Suchý (sometime I use short name Mirek).
I'm using Linux from previous millennium. I'm Red Hat employee since 2006.
I worked on RHN, Spacewalk and Katello in past. Since this summer I have been 
working on Copr.

I would like to attend meetings and submit RFR for Copr when it becomes ready.

You can catch me on #fedora-buildsys during day (of European TZ).
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: How we handle attacks?

[Miroslav Suchý]

On 10/03/2013 02:55 PM, Jhoanir Torres wrote:

Is highly recommended use 'Fail2Ban' in victim servers.


And do we already use it? Because git grep in ansible.git returns zero to me.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: How we handle attacks?

[Miroslav Suchý]

On 10/07/2013 05:23 AM, Anshu Prateek wrote:

Most of these logins are automated bot attempts. On my personal servers, one 
easy way I have found is changing the
default port to something else and that cuts down my lastb by almost 99%!


Yes, I do that for my personal servers as well (and it works really good). But I do not think this is good approach in 
organization when people fluctuate quite often (think about apprentice group).


fail2ban looks good, I'm trying it right now. Unless somebody will object I 
will add it to ./tasks/cloud_setup_basic.yml
so all cloud images will use it.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Arm in Fedora cloud?

[Miroslav Suchý]

Hi,
Is is possible to get arm VM from Fedora Cloud. If not native, then at least 
emulated on x86 machine?
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

How I create new AMI?

[Miroslav Suchý]

Hi,
can somebody point me to documentation how to create new AMI in Fedora Cloud, 
please?

I know how to create it in OpenStack dashboard via WebUI, but we do not have 
dashboard, right?
How can I create it in Fedora Cloud?
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: How I create new AMI?

[Miroslav Suchý]

On 10/14/2013 11:07 PM, Kevin Fenzi wrote:

Basically we just make a .qcow2 image and then someone who's got access
uploads it into the cloud, then it becomes available (provided it's
marked public).


And what is the command?

Mirek-in-learning-mode

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Copr documentation

[Miroslav Suchý]

Hi,
I created
  http://infrastructure.fedoraproject.org/infra/docs/copr.txt
which have some informations, which is stored neither in copr wiki, nor in 
ansible playbooks.
And which may be usefull if somebody have to act on copr machines without 
knowledge of copr.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Some questions around coprs

[Miroslav Suchý]

On 12/04/2013 09:20 PM, Kevin Fenzi wrote:

1. Do we even want to persue this?


Not my priority. But if somebody will be willing to do it, then you are welcome.


2. If so, do we have any ideas how signing copr packages could work?


I did not investigated it yet (again not priority right now) but probably
obs-sign:
http://en.opensuse.org/openSUSE:Build_Service_Signer
https://github.com/openSUSE/obs-sign
or sigul:
https://fedoraproject.org/wiki/User:Mitr


3. Mirroring doesn't seem like it would be that hard, just rsync off
the repos and push them out in our regular mirroring system. Could be a
fair bit of churn tho, and there's no set schedule, so we would have to
decide on frequency, etc.


Copr is just starting. Not so much users right now. I do not think we *need* 
mirroring right now.
I would put this on back burner and revisit this question in ~9 months.
But again - if somebody is willing to configure it, then he is welcome.


4. If coprs moves to being inside koji, could we at that point have a
better time with these needs?


I think, that it does not matter.


5. Perhaps we could propose some kind if pergatory type setup between
coprs (experemental, just builds, may set your house on fire, may
update incompatibly every day) and fedora repository packages (with all
the updates guidelines, reviews, etc).


Whoa! That is completly Fedora.next hidden in this sentence :)

We are preparing something like this for SCL right now:
https://www-dev.softwarecollections.org/en/directory/new/
Note: ^ this may or not work, this is dev instance under heavy development.
It is focused on SCL only.
This will import SCL from Copr and allow to go through some kind of review. And reviewed collections will get some kind 
of publicity.
This is sooo fresh that I hesitate to anticipate anything. But if this will succeed, we can do something similar in 
higher scale with all projects on Copr.



Possibly related to this: I wonder if copr could grow a 'meta repo'
that has all the repodata of all existing coprs. Then you could just
enable one thing and be able to install any coprs?


Yes. I have in plan to provide such thing. Unfortunately according to yesterday FesCO meeting this could not be shipped 
in Fedora itself. At least not yet.



--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Ansible question

[Miroslav Suchý]

On 12/07/2013 10:28 AM, Michael Scherer wrote:

Le vendredi 06 décembre 2013 à 18:01 +0100, Miroslav Suchy a écrit :

Working on Copr, I want to replace/add one line in file. I spent more
then hour trying various things, but I'm out of ideas.

What I'm trying to do is:

self.conn.module_name = lineinfile
self.conn.module_args = dest=/etc/mock/%s.cfg
line=\config_opts['chroot_setup_cmd'] = 'install @build %s'\
regexp=\^.*chroot_setup_cmd.*$\ % (self.chroot, self.buildroot_pkgs)

Which in yaml language should be (with placeholders expanded):

- name: putting scl-utils-build into minimal buildroot of fedora-19-i386
lineinfile:
  dest=/etc/mock/fedora-19-i386.cfg
  line=config_opts['chroot_setup_cmd'] = 'install @build
scl-utils-build'
  regexp=^.*chroot_setup_cmd.*$

I tried several things - among all:
- change regexp
- do not use regexp at all as that should put $line at the end of
file, which would work as well
- use command module with sed, but there is too much of escaping and
it is unreadable

Can somebody advise me what should be correct form to replace or add
that line to mock config please?

I tested the following playbook, and it work. So I think we may need
more information on what you try and how.


I found it.
For the record: It was permission problem. The connection was made as copr user and not as root user. And copr user 
obviously can't modify /etc/mock/* files.



--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

How tickets are resolved

[Miroslav Suchý]

Hi,
I have suggestion.
Can we please put into tickets how they have been resolved?
I mean something else then Fixed.

Something like:
  Fixed - puppet.git commit abc123
or
  Fixed - I run command rm foo.bar

This way people (and apprentice group especially) can learn how infra set up 
works.
And if ticket need to be reopened (or audited) later, you can easily what was 
really done.

I can understand exceptions in complicated tickets, but usually it is just few commits and one or few more commands. 
Isn't it?


--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: January status update for Fedora Infrastructure Apprentices

[Miroslav Suchý]

On 01/06/2014 08:45 PM, Kevin Fenzi wrote:
 0. Whats your fedora account system login?

msuchy

 1. Have you logged in and used your fi-apprentice membership to look at
 our machines/setup in the last month? Do you plan to?

No. I was eating candies and packing and unpacking gifts.

 2. Has it helped you decide any area you wish to focus on or contribute
 to more?

-

 3. Have you looked at or been able to work on any of the fi-apprentice
 'easyfix' tickets?
 https://fedorahosted.org/fedora-infrastructure/report/14

No.


 4. Do you still wish to be a member of the group? If not (for whatever
 reason) could you provide any hints to help others down the road?

yes.

 5. Is there any help or communication or ideas you have that would help
 you do any of the above?

No.

 6. What do you find to be the hardest part of getting involved?
 Finding things to work on? Getting attention from others to help you?
 Finding tickets in your interest area? 

Hardest part is to find free time :(


 7. Have you been able to make any weekly irc meetings? Do you find them
 helpful or interesting? 

Yes.


 8. What are you most looking forward to in the world of computing for
 2014?

Hopefully more machines with Arm64 will be GA.


-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: state of the infra ansible, cron job and roadmap

[Miroslav Suchý]

On 01/08/2014 09:10 PM, Kevin Fenzi wrote:
 a) run a --check --diff once a day and yell about unreachable or
 changed0
 (I could commit this now)

+1 but allow to set exceptions. For example I expect that copr-fe-dev and 
copr-be-dev differ from ansible config,
because I'm breaking it on purpose during development very often.
On the other hand I would love to get warnings about productions copr machines.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Is copr ready for primetime?

[Miroslav Suchý]

On 02/08/2014 07:37 PM, Kevin Fenzi wrote:

On Sat, 08 Feb 2014 21:29:49 +1100
Graham Williamson gra...@williamsonsinc.id.au wrote:


I've created a ticket to add some missing web apps to apps.fp.o.

https://fedorahosted.org/fedora-infrastructure/ticket/4224

The question (as discussed in irc just now) is should copr be added at
this stage.  So, does wide beta equate to production enough to make
it onto apps.fp.o?


Nearly. :) more below


For me to think of it as production ready I'd like:

* nagios monitoring


This weekend I learned more about fedmsg from Ralph, so I want to create alerts 
based on fedmsg (probably today or tomorrow)


* Our cloud more updated/stable.


Not sure about this. Yes, sometimes we run out of space in Cloud and it cause 
problems. Not sure what to do with this.
Beside requesting money for more HW for next fiscal year, what I already done.


If you want to migrate it to apps.fp.o then I'm more then happy. But there are 
some problems to resolve.

1) You can migrate copr-fe.cloud.fedoraproject.org (that is that user-facing copr.fedoraproject.org). You can try to set 
it up in apps.stg.fp.o. This one should be quite easy as it is just python (wsgi) application with postgresql db. We 
will see how it will work and I can then migrate data.


2) But we have copr-be.cloud.fedoraproject.org. This one spin up VM from Fedora Cloud. I'm not sure if you can do that 
outside of Fedora Cloud. And it have attached storage with all builded rpms. It currently have 800 GB with 60 GB 
consumed. For those reason I have *no* idea how to migrate copr-be to apps.fp.o.


But from Copr POV it can be completly fine to have copr-fe in apps and copr-be 
in cloud.


--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Problems with recent ansible

[Miroslav Suchý]

FYI:
https://groups.google.com/forum/#!topic/ansible-project/Mj6vmhqMED8

just beware before you do yum upgrade.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Problems with recent ansible

[Miroslav Suchý]

On 02/27/2014 12:21 AM, Kevin Fenzi wrote:

I'd be interested in short reproducers here... the changes between
1.4.3 and 1.4.5 are really minor:


That is really strange. The problem disappear after I downgraded. I was not able to reproduce it on another machine. And 
today it start happen on copr production machine again (with old ansible). And then suddenly disappear. And it suddenly 
work even with new version and I swear I did not change anything relevant.

Nothing to see here, go home :) Well unless I will be able to reproduce it.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Problems with recent ansible

[Miroslav Suchý]

It happen again.But now I have more traces and hints.

This morning (9:33 UTC) I get Nagios alert that:
WARN:  datanommer has not seen a copr message in 6 hours, 10 minutes, 39 seconds
which means that sometime between 3:30 UTC and 4:30 UTC something happen.

I logged to copr-be and to my surprise:
ansible-playbook - -c ssh /home/copr/provision/builderpb.yml
ERROR: debug is not a legal parameter in an Ansible task or handler
without changing anything over night.

To my surprise I find that:
rpm -V ansible
...
missing /usr/share/ansible/utilities
missing /usr/share/ansible/utilities/accelerate
missing /usr/share/ansible/utilities/debug
missing /usr/share/ansible/utilities/fail
missing /usr/share/ansible/utilities/include_vars
missing /usr/share/ansible/utilities/pause
missing /usr/share/ansible/utilities/set_fact
missing /usr/share/ansible/utilities/wait_for

I.e. Whole content of /usr/share/ansible/utilities is missing.
I quickly reinstall ansible package and everything started working again.

Now I have to find the cause otherwise I expect that it happen again this night.

I checked syslog and only relevant informations are:
1)
Feb 28 03:46:22 dhcp-client03 systemd[1]: Got automount request for 
/proc/sys/fs/binfmt_misc, triggered by 24347 (find)
Feb 28 03:46:22 dhcp-client03 systemd[1]: Mounting Arbitrary Executable File 
Formats File System...
Feb 28 03:46:22 dhcp-client03 systemd[1]: Mounted Arbitrary Executable File 
Formats File System.

2)
Feb 28 04:04:05 dhcp-client03 systemd-logind[291]: New session 24 of user root.
Feb 28 04:04:05 dhcp-client03 ansible-yum: Invoked with CHECKMODE=True name=cloud-utils list=None 
disable_gpg_check=False conf_file=None state=present disablerepo=None enablerepo=None

Feb 28 04:04:05 dhcp-client03 systemd-logind[291]: Removed session 24.
Feb 28 04:04:05 dhcp-client03 systemd-logind[291]: New session 25 of user root.
Feb 28 04:04:05 dhcp-client03 ansible-command: Invoked with executable=None shell=False args=growpart /dev/vda 2 
removes=None creates=None chdir=None

Feb 28 04:04:06 dhcp-client03 systemd-logind[291]: Removed session 25.
Feb 28 04:04:06 dhcp-client03 systemd-logind[291]: New session 26 of user root.
Feb 28 04:04:06 dhcp-client03 ansible-setup: Invoked with CHECKMODE=True 
filter=* fact_path=/etc/ansible/facts.d
Feb 28 04:04:06 dhcp-client03 systemd-logind[291]: Removed session 26.
Feb 28 04:04:07 dhcp-client03 systemd-logind[291]: New session 27 of user root.
Feb 28 04:04:07 dhcp-client03 ansible-yum: Invoked with CHECKMODE=True name=fedmsg,libsemanage-python,python-psutil 
list=None disable_gpg_check=False conf_file=None state=installed disablerepo=None 
pkg=fedmsg,libsemanage-python,python-psutil enablerepo=None

Feb 28 04:04:42 dhcp-client03 systemd-logind[291]: Removed session 27.

I am not sure about the first one.

The second one is some ansible playbook (can it be that nirik check of 
differences?)
But I'm really clueless how it can remove /usr/share/ansible/utilities/*
Does somebody have some idea?

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Problems with recent ansible

[Miroslav Suchý]

On 02/28/2014 06:52 PM, Kevin Fenzi wrote:

Just ping me on irc when you are available to watch the copr-be end.


Will do.

In the meantime I 'solved' it by chattr +i on those files.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Fedora Cloud cleanup

[Miroslav Suchý]

According to my searches this images are not used (I searched both ansible.git 
and puppet.git).

not used at all:
f17_qcow_id: ami-0001
f16-64: ami-0002
Can I remove it? Unless you stop me, I will remove it after one week.

used by inventory/host_vars/209.132.184.166 (jenkins-f18) and 
playbooks/fedora_temp_instance.yml
vars/global.yml:f18_qcow_id: ami-0016
Can it be removed? Or migrated to more recent Fedora?


--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [ansible] Try out this conditional restart stuff.

[Miroslav Suchý]

On 03/14/2014 04:30 PM, Ralph Bean wrote:

+- name: restart fedmsg-gateway
+  command: /usr/local/bin/conditional-restart.sh fedmsg-gateway fedmsg-gateway


Ralph,
I tried to run copr-backend playbook and this notified and therefore executed, 
but failed, because

NOTIFIED: [restart fedmsg-gateway] 
failed: [209.132.184.142] = {cmd: [/usr/local/bin/conditional-restart.sh, fedmsg-gateway, fedmsg-gateway], 
failed: true, item: , rc: 2} 


msg: [Errno 2] No such file or directory

and:
yum whatprovides /usr/local/bin/conditional-restart.sh
No matches found

So where I can get that file please?


--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [ansible] Try out this conditional restart stuff.

[Miroslav Suchý]

On 03/17/2014 02:43 PM, Ralph Bean wrote:

Did I incorrectly assume
that we are including that in every playbook?


Obviously :)
I fixed that already.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Scheduling an IRC meeting on our private cloud setup

[Miroslav Suchý]

On 03/25/2014 08:12 PM, Ralph Bean wrote:

Meeting started by nirik at 18:00:03 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-classroom/2014-03-25/infrastructure-private-cloud-class.2014-03-25-18.00.log.html


I promised to ask OpenStack guys two guestions. Here are the answers:

1) Can we put Arm64 into OpenStack? No. Not yet supported.

2) Can be TripleO scripted? Yes, there already exist puppet modules - he was 
aware of:
  https://github.com/agroup/tripleo-puppet-elements
I asked uncle Google and said that ansible exists as well:
  https://github.com/redhat-openstack/khaleesi
but OS guy said this is for devel setups and for production we should rather 
use:
  https://github.com/agroup/instack
  https://github.com/agroup/instack-undercloud
which is intended for production - although that is not ansible.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

New fedmsg certs

[Miroslav Suchý]

Today I was notified that #fedora-fedmsg say:
 [09:43] fedmsg-bot copr.build.end (invalid signature!) -- ...

I had to re-run playbook because that it seem that fedmsg got new certs and 
revoked old.

I still see at least:
[09:56] fedmsg-bot buildsys.build.state.change (invalid signature!) -- karsten's libccp4-6.3.1-4.fc20 failed to build 
(ppc) http://ppc.koji.fedoraproject.org/koji/buildinfo?buildID=186452
[09:43] fedmsg-bot fedbadges.badge.award (invalid signature!) -- albertone has been awarded the Paranoid Panda badge 
https://badges.fedoraproject.org/user/albertone
[09:44] fedmsg-bot fedbadges.person.rank.advance (invalid signature!) -- albertone moved to position 6003 on the 
badges leaderboard
[09:46] fedmsg-bot bodhi.update.request.testing (invalid signature!) -- jgrulich submitted 
kde-plasma-networkmanagement-0.9.0.11-1.fc19 to testing 
https://admin.fedoraproject.org/updates/kde-plasma-networkmanagement-0.9.0.11-1.fc19
[10:00] fedmsg-bot fedocal.meeting.reminder (invalid signature!) -- Friendly reminder!  The ROS/RPM IRC workshop 
meeting from the ambassadors calendar starts in 11 hours https://apps.fedoraproject.org/calendar/meeting/365/


So at least those (and maybe others) need to redeploy fedmsg certs.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Openstack ansible manifest

[Miroslav Suchý]

Following yesterday meeting:
This is may work-in-progress of openstack ansible manifest:
https://github.com/xsuchy/openstack-ansible-install

I decided to skip all this undercloud etc. and just follow what main wiki 
suggest:
http://docs.openstack.org/trunk/install-guide/install/yum/content/index.html

Far from being finished, but comments and collaborators are welcome.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Vacation Thu,Fri *2

[Miroslav Suchý]

I will be on vacation on 1st, 2nd and 8th, 9th May. Just FYI.
-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Plan of work for Copr signing

[Miroslav Suchý]

FYI - this is my schedule of work needed to sign packages in Copr:

Hardware:
=
Next visit in PHX is planned on June/July. Next one is January of 2015.

Ideal (and most paranoid) setup would require one physical machine for Signing server and one for copr-backend and one 
wire between them. With no remote access to signing server.


But we have not HW for this.

What we can have is have signing machine in VM with restrictive SW defined network. If that VM can be only one VM on 
host, then it would be great.


To set up VM and networking and create ansible manifest, can take up to one 
week.


Software:
=
I would go the obs-sign way.
It would require to get one patch into GPG2. Patch is made by SuSe, but does 
not live in upstream.
TMraz (RH packager) preliminary approved this patch, but have few comments, which would need to be address (name of cmd 
option, no man page...). Then I will try to get it in upstream, but there is risc of rejecting. But TMraz is willing to 
accept it as patch into Fedora and RH package. This is backup plan. (1.5 week to work on patch, 1 w for communitation 
with upstream or tmraz)

JStribrny promised to re-package obs-sign. (0.5w)
We should enhance documentation of obs-sign and likely write HOWTO for 
deployment. (0.75w)
We need to deploy and configure obs-sign on VM. (0.75w)
Mutatis mutandis of Copr (1w).
Sum it up (5.5 week)

Total = 6.5 weeks


--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Plan of work for Copr signing

[Miroslav Suchý]

On 05/23/2014 05:45 PM, Kevin Fenzi wrote:

* a key per user


key per user


When are things intended to be signed? At the end of successfull build?


At the end of successful build.


If signing fails, will that fail the build?


Should it? Likely yes. I will think about it.


Can obs-signd handle multiple incoming connections? Or can it only sign
one thing at a time? Would things block waiting to sign?


It is single threaded application and process packages one by one.
But I do not expect that it become bottleneck as the signing should be really 
fast (no testing thou yet).

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: July status update for Fedora Infrastructure Apprentices

[Miroslav Suchý]

On 07/04/2014 09:00 PM, Kevin Fenzi wrote:

0. Whats your fedora account system login?


msuchy


1. Have you logged in and used your fi-apprentice membership to look at
our machines/setup in the last month? Do you plan to?


No. I'm mostly relying on presence in systadmin-cloud membership.


2. Has it helped you decide any area you wish to focus on or contribute
to more?


Cloud.


3. Have you looked at or been able to work on any of the fi-apprentice
'easyfix' tickets?
https://fedorahosted.org/fedora-infrastructure/report/14


No.


4. Do you still wish to be a member of the group? If not (for whatever
reason) could you provide any hints to help others down the road?


I'm considering to leave and focus on sysadmin-cloud.


5. Is there any help or communication or ideas you have that would help
you do any of the above?


no.


6. What do you find to be the hardest part of getting involved?
Finding things to work on? Getting attention from others to help you?
Finding tickets in your interest area?


To find free time :)


7. Have you been able to make any weekly irc meetings? Do you find them
helpful or interesting?


Yes to both.


8. What is your favorite movie of all time?


At ziji duchove and if fact most of movies from this director:
http://www.imdb.com/name/nm0513792/



--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Transifex has become proprietary

[Miroslav Suchý]

On 07/03/2014 04:20 PM, Dimitris Glezos wrote:

It's a good thing that this came up, it'd be nice to have a clear decision from 
the Fedora part. I explained in detail
the log  reasoning behind the decision to stop maintaining the open-source 
branch in the GitHub issue Rahul provided.


Dimitris,
I understood that it costed you a lot of time to provide sources, which works for everybody and which were rarely used. 
So you stopped releasing it.


But can you release the code, which works just for you? The code for your main instance? And if somebody want to run his 
own instance, let him maintain the differences.
This will have the benefit, that people will be able to  browse the code and contribute with fixes/RFE (yet without 
testing), but you can finalize it yourself as you did with my XLIFF contribution.
And you will be open-source company (which I believe you are in heart). And Fedora will be able to continue Transifex. 
Otherwise - I'm afraid - the force to use open-source solution will be too strong and migration to other system will be 
inevitable. And that would be shame, because I still think that Transifex has superior features.


Please reconsider this in your team.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

New OpenStack instance

[Miroslav Suchý]

This is mostly for Kevin and Stephen.

Stephen provisioned fed-cloud09.cloud.fedoraproject.org to RHEL7.

I installed there new OpenStack instance using [1].
I make git-clone and files are in:
  /root/openstack-ansible-install
I modified var/* to match IP and generated new passwords.
It would be nice if somebody can put content of
  var/os.yml
in private.git, but I would rather not put it in
  {{ private }}/vars.yml
so we do not get collisions, and rather to put it in:
  {{ private }}/vars_os.yml
or similar.

Additionally I removed VG vg_guest and created VG 'cinder-volumes', because 
that is what packstack expect.
Is setup of fed-cloud09 already in some playbook? If yes, it need to be changed.

Dashboard is available at:
http://209.132.184.9/dashboard/

Right now it is just http, If we want https (would be nice) then I would need pem encode cert and key (without 
passphrase). Modify

CONFIG_HORIZON_SSL=n
CONFIG_SSL_CERT=
CONFIG_SSL_KEY=
in
  /root/openstack-ansible-install/files/packstack-controller-answers.txt
and rerun playbook:
  ansible-playbook -i localhost, controller.yml

I will try to add to controller.yml other tasks to re-create flavours, common 
images (F20,19 etc..).
If you want to add there something, you are welcome. It is still git, so please 
commit before you finish.
At the end I want to copy resulting playbook to lockbox.

BTW: If you add some task (eg. those flavour etc.) I recommend to temporary 
comment out line:
  - command: packstack --answer-file=/root/packstack-controller-answers.txt
as this step take huge amount of time (up to 10-20 minutes).


[1] https://github.com/xsuchy/openstack-ansible-install/tree/packstack

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

OpenStack Ansible modules

[Miroslav Suchý]

FYI:
Adam packaged openstack-ansible-modules for Fedora (and Epel7).
  https://bugzilla.redhat.com/show_bug.cgi?id=1134377
I expect that it would simplify our new OS playbook.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: September status update for Fedora Infrastructure Apprentices

[Miroslav Suchý]

On 09/03/2014 12:15 AM, Kevin Fenzi wrote:

Greetings.

You are getting this email because you are in the 'fi-apprentice' group
in the fedora account system (or are reading this on the
infrastructure list).

Feel free to reply just directly to me, or cc the infrastructure list
for everyone to see and comment on.

https://fedoraproject.org/wiki/Infrastructure_Apprentice

At the first of every month(or so), I am going to be sending out an
email like this one. I would like feedback on how things are going for
you.

I'd like to ask for everyone to send me a quick reply with the
following data or anything related you can think of that might help us
make the apprentice program more useful.

0. Whats your fedora account system login?

1. Have you logged in and used your fi-apprentice membership to look at
our machines/setup in the last month? Do you plan to?


yes


2. Has it helped you decide any area you wish to focus on or contribute
to more?


clouds



3. Have you looked at or been able to work on any of the fi-apprentice
'easyfix' tickets?
https://fedorahosted.org/fedora-infrastructure/report/14


no


4. Do you still wish to be a member of the group? If not (for whatever
reason) could you provide any hints to help others down the road?


yes


5. Is there any help or communication or ideas you have that would help
you do any of the above?


no


7. Have you been able to make any weekly irc meetings? Do you find them
helpful or interesting?


yes


8. What is your favorite book of all time? :)


Heaven Has No Favorites (German: Der Himmel kennt keine Günstlinge) - a novel 
by the German writer Erich Maria Remarque.


--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Something is polluting lockbox01 /

[Miroslav Suchý]

On 09/17/2014 05:18 PM, Kevin Fenzi wrote:

Yes, ansible makes these anytime a playbook has failed hosts. The idea
is that you can then pass this retry to it on the next run and it will
only run on those hosts that failed.;)

There shouldn't be any in / they should be in/root/


I guess ('cos of fed-cloud09.cloud.fedoraproject.org.retry) that rbac likely change $HOME and it land in / instead of 
home or /tmp.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Copr to use primary Fedora download location

[Miroslav Suchý]

Hi,
right now Copr is using stock mock, with its default configuration.
Which means that Copr builders are downloading packages from Fedora mirrors.
I find this sub-optimal, because:
 * sometimes is mirror little bit off-sync and occasionally this result in 
failed builds.
 * while mirrors are generally good thing, primary Fedora servers are AFAIK just few racks away from Copr. In term of 
measurable data it is 100 ms of ping vs. 4 ms of ping.


Therefore I plan to replace:
  
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releaseverarch=$basearch
in mock config with:
  
baseurl=http://dl.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/

and similary for epel, rawhide and F21.
But before I proceed, would like to ask if this is ok? Or should I rather not 
use dl.f.o for some reason?

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Copr to use primary Fedora download location

[Miroslav Suchý]

Additionally I would like to do the same for Centos.
Before I ask CentOS guys... do we have somewhere in our datacenter copy of 
CentoOS repo?

If not, I'm not sure if I would like to rsync everything. Maybe rather just 
setup squid.

--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

change in conditional-restart.sh

[Miroslav Suchý]

Hi,
today I run groups/copr-backend.yml playbook and fedmsg/base notified restart 
httpd. Which failed because
httpd is there installed, but not enabled (it is there just as requirement of 
webalizer).
So I'm thinking about change (after freeze):
diff --git a/roles/base/files/common-scripts/conditional-restart.sh 
b/roles/base/files/common-scripts/conditional-restart.sh
index f95ef74..f4ac932 100644
--- a/roles/base/files/common-scripts/conditional-restart.sh
+++ b/roles/base/files/common-scripts/conditional-restart.sh
@@ -10,9 +10,13 @@ rpm -q $PACKAGE
 INSTALLED=$?

 if [ $INSTALLED -eq 0 ]; then
-echo Package $PACKAGE installed.  Attempting restart of $SERVICE.
-/sbin/service $SERVICE restart
-exit $?  # Exit with the /sbin/service status code
+if chkconfig $PACKAGE; then
+echo Package $PACKAGE installed.  Attempting restart of $SERVICE.
+/sbin/service $SERVICE restart
+exit $?  # Exit with the /sbin/service status code
+else
+echo Package $PACKAGE not enabled.  Skipping restart of $SERVICE.
+fi
 fi

 # If the package wasn't installed, then pretend everything is fine.

This works for httpd from el6 to Fedora21. But I tested it only for httpd.
But this is used for all services.
Please raise your voice if you are aware of some service which we want to 
restart, but we do not enable it.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: route between fed-cloud10 and fed-cloud09

[Miroslav Suchý]

On 02/04/2015 02:32 PM, Kevin Fenzi wrote:
 On Tue, 03 Feb 2015 17:54:27 +0100
 Miroslav Suchy msu...@redhat.com wrote:
 
 [root@fed-cloud10 etc(keystone_admin)]# telnet 209.132.184.9 443
 Trying 209.132.184.9...
 telnet: connect to address 209.132.184.9: No route to host

 I am able to connect using 172.24.0.0 network, but it would be nice
 (and required to have new OS) to be able to communicate even over
 public IP.

 Can someone (nirik, ssmoogen) fix this please? And I would need to do 
 that for fed-cloud11 too. It seems that without this fixed I am
 unable to add compute node to OS.
 
 It seems the interfaces on 10 are backwards from the ones on 09... 
 but I am not sure how it even works that we can access 10. ;)
 
 (ie, it's trying to use eth1 to talk to the external on 09 instead of
 eth0). 
 
 Will try and figure out whats going on...
 
 kevin
 
 
 
 ___
 infrastructure mailing list
 infrastructure@lists.fedoraproject.org
 https://admin.fedoraproject.org/mailman/listinfo/infrastructure
 

I could not get to it now so I can not validate, but...
I have in roles/cloud_compute/tasks/main.yml
- lineinfile: dest=/etc/sysconfig/network-scripts/ifcfg-eth1 regexp=^ONBOOT= 
line=ONBOOT=yes
  notify:
- restart network
- lineinfile: dest=/etc/sysconfig/network-scripts/ifcfg-eth1 regexp=^NETMASK= 
line=NETMASK=255.255.255.0
  notify:
- restart network
- lineinfile: dest=/etc/sysconfig/network-scripts/ifcfg-eth1 regexp=^IPADDR= 
line=IPADDR={{compute_private_ip}}
  notify:
- restart network
- lineinfile: dest=/etc/sysconfig/network-scripts/ifcfg-eth1 
regexp=BOOTPROTO= line=BOOTPROTO=none
  notify:
- restart network
Can it be that there is no MAC binding in udev so eth1 can become eth0 and vice 
versa?
Or something else?

BTW it does not block me any more. I managed to change endpoints to use 
internal IP. But still nice to have.



-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Proper SSL cert for fed-cloud09?

[Miroslav Suchý]

When I do:
[root@fed-cloud09 ~(keystone_admin)]# cinder type-list
ERROR: Unable to establish connection: [Errno 1] _ssl.c:504: error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Which just transit to:
[root@fed-cloud09 ~(keystone_admin)]# curl -i 
https://fed-cloud09.cloud.fedoraproject.org/
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.

Is it time to get SSL cert signed by some CA?
However I would swear I had not this problems yesterday. But it behaves this 
way even if I revert my work.

Comments are welcome.
-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: How to open port?

[Miroslav Suchý]

On 02/02/2015 04:10 PM, Kevin Fenzi wrote:
 Just copy paste the iptables section from base role and adjust the path
 to the iptables templates

KISS - I will try this approach. Thanks

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

How to open port?

[Miroslav Suchý]

How do we open ports in ansible today?

I want to open port 5672 for 172.24.0.10/24. Currently it is open only to:

[root@fed-cloud09 ~]# iptables-save |grep 5672
-A INPUT -s 209.132.184.9/32 -p tcp -m multiport --dports 5671,5672 -m comment 
--comment 001 amqp incoming
amqp_209.132.184.9 -j ACCEPT

So I done this change:

diff --git a/inventory/host_vars/fed-cloud09.cloud.fedoraproject.org
b/inventory/host_vars/fed-cloud09.cloud.fedoraproject.org
index 2559de1..4a96e81 100644
--- a/inventory/host_vars/fed-cloud09.cloud.fedoraproject.org
+++ b/inventory/host_vars/fed-cloud09.cloud.fedoraproject.org
@@ -1,2 +1,3 @@
 ---
 root_auth_users:  msuchy
+tcp_ports: [ 80, 443, 5672 ]

But it have no effect (yes, I run the playbook again).
What is our best practice now and where I made mistake?

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Proper SSL cert for fed-cloud09?

[Miroslav Suchý]

On 02/05/2015 01:13 AM, Kevin Fenzi wrote:
 Could we instead call it 'openstack.cloud.fedoraproject.org' or
 'controller.cloud.fedoraproject.org' or something? Not sure if that
 needs us to rename/reinstall the node, or can just be done in the
 cert... 

It can be just cname + name in cert. Reinstall is quite fast with ansible, that 
is no problem.
I automated all but one workaround (there is still usually need one reboot).

 Along those same lines, how about we move the existing host playbooks
 to a group/openstack-controller.yml (currently just fed-cloud09, but
 I'd like to see if we can allocate one machine moving forward to be
 our test for the 'next' openstack) and group/openstack-compute.yml
 (fed-cloud10/11, but some more will be installed next week) to make them
 more generic and ready for more nodes? 

Compute node is already in roles/cloud_compute/tasks/main.yml so migration to 
groups should be easy (not my priority thou).
I see no benefits in migrating controller playbook to group or roles. It is 
only one.
I +1 to controller-next instance, because upgrade of OpenStack is not 
supported. However those playbook will be quite
different and it does not have sense to have them in one playbook with when 
directives.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Route to Dell EquaLogic

[Miroslav Suchý]

On 02/05/2015 11:40 AM, Miroslav Suchý wrote:
 172.24.0.0  0.0.0.0 255.255.255.0   U 0  00 br-tun

Hmm, I rebooted the machine and this ^^^ line disappeared from route and 
172.24.0.100 is now reachable.
I wish I knew what is going on.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Route to Dell EquaLogic

[Miroslav Suchý]

[root@fed-cloud09 ~(keystone_admin)]# ssh grpadmin@172.24.0.100
ssh: connect to host 172.24.0.100 port 22: No route to host

Nirik can this be result of your (?) change in routes that
[root@fed-cloud09 ~(keystone_admin)]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric RefUse Iface
...
172.24.0.0  0.0.0.0 255.255.255.0   U 0  00 eth1
172.24.0.0  0.0.0.0 255.255.255.0   U 0  00 br-tun

So EquaLogic is actually not routable?

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

New OpenStack instance - status

[Miroslav Suchý]

Since I'm leaving for one week vacation, I think I may write down current 
status of our new OpenStack instance and write
down TODO list. Just in case someone is desperate enough to do some fixes.

I updated docs.git/cloud.txt - mainly which playbooks we use right now and 
where to write down IP, when you add new
compute node.

Controller - should be OK. At least I see no problems there right now. Network 
is stable. I can log to EqualLogic
(credentials are at bottom of cinder.conf). Volumes are created correctly. I 
can reach compute nodes. AMQP works and is
reachable from Compute nodes (do not try to play with SSLRabbitMQ it will 
never work on RHEL7). Horizon works (over https).

Compute nodes - it looks good until you try to start VM. :) I fixed several 
problems, but new ones still pop ups.

If you want to debug it, just go to dashboard and start new VM (note that 
m1.tiny is too small for Fedora image) and on
controller do:
  tail -f /var/log/nova/nova-scheduler.log
And look for something like:
  Choosing host WeighedHost [host: fed-cloud13.cloud.fedoraproject.org, weight: 
1.0] for instance
75f1b5ca-88d5-4e57-8c18-8d6554e1f2bc

then log to that instance (right now root@fed-cloud09 can ssh directly as 
root@fed-cloudXX) and
  tail -f /var/log/nova/nova-compute.log /var/log/neutron/openvswitch-agent.log
When spin up of VM fail, then controller try 2 next machines before giving up.

Right now there is some error:
  TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'\n
which is new to me and which I will not manage to fix before I will leave today.
It may be last one problem or they may be dozen other still waiting in queue. 
It's hard to tell.

Smaller fixes to do:
* playbook hosts/fed-cloud09.cloud.fedoraproject.org.yml can be enhanced that 
after packstack execution the machine
should be restarted. Right now I am waiting for first error after packstack and 
then I restart the machine manualy and
re-run playbook again. This is last manual workaround. Everything else was 
already automated.
* routing between compute nodes and controller using public IP does not work. 
Not fatal right now, but nice to have.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Ansible question

[Miroslav Suchý]

I have this ansible snippet:
  - name: Create users
keystone_user:
  login_user=admin login_password={{ ADMIN_PASS }} 
login_tenant_name=admin
  user={{ item.name }}
  email={{ item.email }}
  tenant={{ item.tenant }}
  password={{ item.password }}
  state=present
with_items:
  - { name: kevin, email: 'ke...@fedoraproject.org', tenant: 
infrastructure, password: {{kevin_password}} }
  - { name: laxathom, email: 'laxat...@fedoraproject.org', tenant: 
infrastructure, password: {{laxathom_password}} }


But when I run it it produce:
TASK: [Create users] **
changed: [fed-cloud09.cloud.fedoraproject.org] = (item={'password': u'', 
'name': 'kevin', 'tenant':
'infrastructure', 'email': 'ke...@fedoraproject.org'})

changed: [fed-cloud09.cloud.fedoraproject.org] = (item={'password': u'', 
'name': 'laxathom', 'tenant':
'infrastructure', 'email': 'laxat...@fedoraproject.org'})



Is there way to mask the output (using -name or something) so the password is 
not print to console?
-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Can not use mirrorlist with RHEL $releasever (bz#1175566)

[Miroslav Suchý]

On 01/08/2015 11:13 PM, Ian Wienand wrote:
 Hi,
 
 I'd like to try and find the/a person who could help out with [1].
 
 EPEL version updates are a fairly constant annoyance that causes
 issues with CI systems in upstream openstack when the version updates.
 
 As described in the bug, I'd really like to just setup a .repo file
 with
 
   
 http://mirrors.fedoraproject.org/mirrorlist?repo=epel-$releaseverarch=$basearch
 
 to install epel-release and things should just work to always grab
 the latest release.  However [6|7]Server, as given in $releasever by
 RHEL/Centos, don't work as a path.
 
 Any suggestions on how we can get this fixed?

I have been facing this in Copr and mock where you was unable to set additional 
repos for project because:
  
https://copr-be.cloud.fedoraproject.org/results/foo/bar/epel-$releasever-$basearch/
was expanded to {6,7}Server as you stated.

The only solution is to create your own maping and pass it to yum using 
--releasever
In mock I done it that e.g. /etc/mock/epel-6-x86_64.cfg has
  config_opts['releasever'] = '6'
and mock have this defined for every chroot config and pass this value to 
--releasever of yum/dnf.

This way you can actually pass to mockchain
--addrepo='https://copr-be.cloud.fedoraproject.org/results/foo/bar/epel-$releasever-$basearch/'
and it will work as expected.

I'm not sure if this will help you in your specific case thou.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Routing between tenants networks

[Miroslav Suchý]

Quick note for those interested in new OpenStack instance:

Routing between two tenants is apparently not possible. Or to be precise 
I did not discovered how to do that (and even Larsks did not know). 
However ... we can mark same network as shared. This means that those 
networks are visible for all tenants and tenants can assign IP from this 
network to their VMs. So you can route two VM of two different tenants, 
but they must be with the same network.


So I had two option hows to set up Copr network:
1) put copr-be in copr-net network, but copr-be will be owned by 
infrastructure tenant or
2) we can give copr-be two NICs. One with IP from infrastructure network 
(with floatingIP mapped to this IP) and second NIC with IP from copr 
network. This way copr-be will be able to route builders using private 
IP. And we keep others VM (e.g. signer) quite isolated.


The option 2 seems much better to me, therefore I'm going this way.
I already tested it and it really works.
So conclusion is that copr-net and coprdev-net will be visible to 
all tenants. And while you technically can put machines in that network, 
you should not do that as those networks are reserved for production and 
staging instances of Copr builders.


Mirek
--
,,,
   (o o)
  =oOO==(_)==OOo===
 )  mailto:miros...@suchy.cz  tel:+420-603-775737
(   One picture is worth 128K words.
 )Oooo.
 .oooO   (   )
 (   )) /
  \ ((_/
   \_)

___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New OpenStack instance - status

[Miroslav Suchý]

On 03/07/2015 06:59 PM, Kevin Fenzi wrote:
 * We will need to adapt to not giving every instance a floating ip. For
   copr, I think this would be fine, as you don't care that they have

*nod* I was not sure how VM behave when does not have public IP so I tested it.
It is basicaly behind NAT and all internet is accessible.
Therefore yes, Copr builders do not need floating ip.

However this instance of OpenStack behave differently from the old one. When 
you start up VM, you do not get public IP
automatically.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New OpenStack instance - status

[Miroslav Suchý]

On 03/09/2015 01:00 PM, Kevin Fenzi wrote:
 nova commands worked fine from here, but I didn't really try and do
 anything fancy. We could see if the euca stuff will just keep working
 for us for now. 

It works fine. It is just that if you miss some functionality (and I miss a 
lot) and file RFE, it will be likely
rejected that you should now use openstack command.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New OpenStack instance - status

[Miroslav Suchý]

On 03/07/2015 06:59 PM, Kevin Fenzi wrote:
 * Can we adjust the default tennat quotas in the playbooks? They seem a
   bit low to me given the amount of resources we have. 

I put (and tested) the quota for Copr (it is on bottom of playbook).
Can you please write quotas for other tenants (or you can post it to me). I 
have no idea what are needs of those tenants.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New OpenStack instance - status

[Miroslav Suchý]

On 03/09/2015 10:29 AM, Miroslav Suchý wrote:
 On 03/07/2015 07:29 PM, Kevin Fenzi wrote:
  * I see that the tenants have the same internal 172.16.0.0 net right
now, can we make sure we seperate them from each other? ie, I don't
want a infrastructure instance being able to talk to a copr builder
if we can avoid it. 
 Are you sure?
 From: playbooks/hosts/fed-cloud09.cloud.fedoraproject.org.yml
   # 172.16.0.1/12 -- 172.21.0.1/12 - Free to take
   # 172.23.0.1/12 - free (but used by old cloud)
   # 172.24.0.1/12 - RESERVED it is used internally for OS
   # 172.25.0.1/12 - Cloudintern
   # 172.26.0.1/12 - infrastructure
   # 172.27.0.1/12 - persistent
   # 172.28.0.1/12 - transient
   # 172.29.0.1/12 - scratch
   # 172.30.0.1/12 - copr
   # 172.31.0.1/12 - Free to take
 And checking dashboard I see infra in .26 network and copr in .16. Hmm that 
 is different one, but copr should have .30.
 Playbook seems to be correct. Strange.

Ah. Of course /12 is mistake. There should be /16.
However when I see that with /16 we have only 7 free subnets. I would rather 
use /20 subnets, which would give us 4094
IPs per one subnet. That should be enough and it gives us plenty of subnets for 
use.


So it would be:
  # 172.16.0.1/16 -- 172.21.0.1/20- Free to take
  # 172.23.0.1/16 - free (but used by old cloud)
  # 172.24.0.1/24 - RESERVED it is used internally for OS
  # 172.25.0.1/20  - Cloudintern (172.25.0.1 - 172.25.15.254)
  # 172.25.16.1/20 - infrastructure (172.25.16.1 - 172.25.31.254)
  # 172.25.32.1/20 - persistent (172.25.32.1 - 172.25.47.254)
  # 172.25.48.1/20 - transient (172.25.48.1 - 172.25.63.254)
  # 172.25.64.1/20 - scratch (172.25.64.1 - 172.25.79.254)
  # 172.25.80.1/20 - copr (172.25.80.1 - 172.25.95.254)
  # 172.25.96.1/20 -- 172.25.240.1/20 - free
  # 172.26.0.1/16 -- 172.31.0.1/16 - free


Comments?


-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New OpenStack instance - status

[Miroslav Suchý]

All services are using SSL but novncproxy, which does not worked for me and 
according some random notes on internet does
not work over SSL due some bugs.
But novncproxy does not work for me even over plain http. And I do not know 
why. If somebody else can check it, it would
be great.
Strange thing is that
  telnet fed-cloud09.cloud.fedoraproject.org 6080
from my workstation is rejected, while on fed-cloud09 it pass. And iptable 
allows port 6080. Strange.


I tried to automatize adding of SSH keys using this:

TASK: [shell source /root/keystonerc_admin  F=$(mktemp)  {{ lookup('pipe',
'/srv/web/infra/ansible/scripts/auth-keys-from-fas msuchy') }} $F  nova 
--os-username msuchy --os-password
{{msuchy_password}} --os-tenant-name copr keypair-list | ( grep msuchy || nova 
--os-username msuchy --os-password
{{msuchy_password}} --os-tenant-name copr keypair-add --pub_key $F msuchy ); 
rm -f $F] ***

which does not work. While executing this from shell:

source /root/keystonerc_admin  F=$(mktemp)  cat id_rsa.pub  $F  nova 
--os-username msuchy --os-password
$PASSWORD --os-tenant-name copr keypair-list | ( grep msuchy || nova 
--os-username msuchy --os-password $PASSWORD
--os-tenant-name copr keypair-add --pub_key $F msuchy ); rm -f $F

works. So probably problem is in that lookup() and again I do not know why.


Anyway, I am able (again) to start VM and log to those VM.

My plan for next week is to migrate dev instance to new OpenStack (before it 
will be re-provisioned) and see what needs
to be changed.


-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New OpenStack instance - status

[Miroslav Suchý]

On 03/06/2015 04:02 PM, Miroslav Suchý wrote:
 I tried to automatize adding of SSH keys using this:
 
 TASK: [shell source /root/keystonerc_admin  F=$(mktemp)  {{ lookup('pipe',
 '/srv/web/infra/ansible/scripts/auth-keys-from-fas msuchy') }} $F  nova 
 --os-username msuchy --os-password
 {{msuchy_password}} --os-tenant-name copr keypair-list | ( grep msuchy || 
 nova --os-username msuchy --os-password
 {{msuchy_password}} --os-tenant-name copr keypair-add --pub_key $F msuchy 
 ); rm -f $F] ***
 
 which does not work. While executing this from shell:
 
 source /root/keystonerc_admin  F=$(mktemp)  cat id_rsa.pub  $F  nova 
 --os-username msuchy --os-password
 $PASSWORD --os-tenant-name copr keypair-list | ( grep msuchy || nova 
 --os-username msuchy --os-password $PASSWORD
 --os-tenant-name copr keypair-add --pub_key $F msuchy ); rm -f $F
 
 works. So probably problem is in that lookup() and again I do not know why.

Ok, I just find that there is ansible module for that. And it works fine.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

users belonging to tenant in FedoraCloud

[Miroslav Suchý]

In new OpenStack instances users belong to this tenants:

  - { name: kevin, email: 'ke...@fedoraproject.org', tenant: 
infrastructure, password: {{kevin_password}} }
  - { name: laxathom, email: 'laxat...@fedoraproject.org', tenant: 
infrastructure, password: {{laxathom_password}} }
  - { name: samkottler, email: 'samkott...@fedoraproject.org', tenant: 
infrastructure, password:
{{samkottler_password}} }
  - { name: puiterwijk, email: 'puiterw...@fedoraproject.org', tenant: 
infrastructure, password:
{{puiterwijk_password}} }
  - { name: mattdm, email: 'mat...@fedoraproject.org', tenant: 
infrastructure, password: {{mattdm_password}} }
  - { name: tflink, email: 'tfl...@fedoraproject.org', tenant: qa, 
password: {{tflink_password}} }
  - { name: copr, email: 'ad...@fedoraproject.org', tenant: copr, password: 
{{copr_password}} }
  - { name: twisted, email: 'build...@twistedmatrix.com', tenant: 
pythonbots, password: {{twisted_password}} }
  - { name: ausil, email: 'den...@ausil.us', tenant: infrastructure, 
password: {{ausil_password}} }
  - { name: anthomas, email: 'antho...@redhat.com', tenant: cloudintern, 
password: {{anthomas_password}} }
  - { name: jskladan, email: 'jskla...@redhat.com', tenant: qa, password: 
{{jskladan_password}} }
  - { name: gholms, email: 'gho...@fedoraproject.org', tenant: cloudintern, 
password: {{gholms_password}} }
  - { name: cockpit, email: 'walt...@redhat.com', tenant: scratch, 
password: {{cockpit_password}} }
  - { name: nb, email: 'n...@fedoraproject.org', tenant: infrastructure, 
password: {{nb_password}} }
  - { name: pingou, email: 'pin...@pingoured.fr', tenant: infrastructure, 
password: {{pingou_password}} }
  - { name: codeblock, email: 'codebl...@elrod.me', tenant: infrastructure, 
password: {{codeblock_password}} }
  - { name: msuchy, email: 'msu...@redhat.com', tenant: copr, password: 
{{msuchy_password}} }
  - { name: red, email: 'r...@fedoraproject.org', tenant: infrastructure, 
password: {{red_password}} }

This is list of available tenants:
  - { name: persistent, desc: persistent instances }
  - { name: qa, desc: developmnet and test-day applications of QA }
  - { name: transient, desc: 'transient instances' }
  - { name: infrastructure, desc: one off instances for infrastructure 
folks to test or check something
(proof-of-concept) }
  - { name: cloudintern, desc: 'project for the cloudintern under mattdm' }
  - { name: cloudsig, desc: 'Fedora cloud sig folks.' }
  - { name: copr, desc: 'Space for Copr builders' }
  - { name: coprdev, desc: 'Development version of Copr' }
  - { name: pythonbots, desc: 'project for python build bot users - 
twisted, etc' }
  - { name: scratch, desc: 'scratch and short term instances' }

If you want to have access to additional tenants, please reply to this email 
(publicly so others can review) and I will
grant you access to those tenants.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: users belonging to tenant in FedoraCloud

[Miroslav Suchý]

On 03/12/2015 04:26 PM, Kevin Fenzi wrote:
 I think it might be good to have you, me and patrick at least in all
 teanants as we often need to look at and diagnose issues other people
 have. Of course we could just login as admin, but perhaps we should
 discourage that... 

Done in commit:
* e8bdfb2  add msuchy, patrick and kevin to all tenants

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

SSL certificate for new FedoraCloud for user's command line tools

[Miroslav Suchý]

The new FedoraCloud (FC) is still not in final state, but if you work with it. 
Or you will work with it in future - here
is quick HOWTO regarding certificates.

The cerficate can be found at:
 https://fed-cloud09.cloud.fedoraproject.org/pub/fed-cloud09.pem
Your RC file is at:
 
https://fed-cloud09.cloud.fedoraproject.org/dashboard/project/access_and_security/

FC use self signed cert. When you use command line tools, e.g:
  nova list
you have to either use --insecure option:
  nova --insecure list
or (and this is preferred way):
  * download the PEM file (see above) and put it in
/etc/pki/ca-trust/source/anchors/
  * run:
/usr/bin/update-ca-trust
Then the certificate become trusted and you can use
  nova list
without errors.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New OpenStack instance - status

[Miroslav Suchý]

On 03/07/2015 07:29 PM, Kevin Fenzi wrote:
 * I see that the tenants have the same internal 172.16.0.0 net right
   now, can we make sure we seperate them from each other? ie, I don't
   want a infrastructure instance being able to talk to a copr builder
   if we can avoid it. 

Are you sure?
From: playbooks/hosts/fed-cloud09.cloud.fedoraproject.org.yml
  # 172.16.0.1/12 -- 172.21.0.1/12 - Free to take
  # 172.23.0.1/12 - free (but used by old cloud)
  # 172.24.0.1/12 - RESERVED it is used internally for OS
  # 172.25.0.1/12 - Cloudintern
  # 172.26.0.1/12 - infrastructure
  # 172.27.0.1/12 - persistent
  # 172.28.0.1/12 - transient
  # 172.29.0.1/12 - scratch
  # 172.30.0.1/12 - copr
  # 172.31.0.1/12 - Free to take
And checking dashboard I see infra in .26 network and copr in .16. Hmm that is 
different one, but copr should have .30.
Playbook seems to be correct. Strange.

 * Do we want to also revisit flavors available? Perhaps drop the
   builder one and just use m1.large for it? we should have resources to
   use more cpus/mem and should make copr builds faster/better. 

80GB is too much, and 4 VCPU too. I think having extra flavor for builder is 
nice as we can change it any time without
affecting other instances/tenants.

 * Is there any way to see how much space is available on the equalogics
   aside from just logging into it via ssh?

Unfortunately no.
I reported it as RFE some time ago.
  https://bugs.launchpad.net/cinder/+bug/1380555
You can only amount of used space using cinder list  cinder show volume-id


-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New OpenStack instance - status

[Miroslav Suchý]

On 03/07/2015 06:59 PM, Kevin Fenzi wrote:
 All thats set and I can see console in the web dash again just fine for
 any of the instances I tried, and they are all https using only. 

Works for me too. Nice. Thanks.

  I tried to automatize adding of SSH keys using this:
 I wonder if we shouldn't have something to update/upload everyones ssh
 keys. Might be handy but of course it's not a blocker/that important. 
 We could even look at just tieing into our existing fedmsg listener
 (when someone with a cloud account changes ssh key, update the cloud). 

Done. Search for upload SSH keys for users action.
However it work only initially. Once user alter his password it will fail.
I ignore those cases with ignore_errors: yes though.
I have pending RFE for OpenStack so admin is able to upload ssh keys to user.

I skipped (commented out) users:
  * twisted
  * cockpit
as I do not know which ssh keys they use. Can somebody put there right values?

  Anyway, I am able (again) to start VM and log to those VM.
 Me too. I uploaded the F22 Alpha cloud image and it worked fine.
 (aside cloud-init taking about 35 seconds to run. It seemed to be
 timing out on some metadata ?)
 
 We should look at hooking our cloud image upload service into this soon
 so we can get images as soon as they are done.

I will leave this one for somebody else.

  My plan for next week is to migrate dev instance to new OpenStack
  (before it will be re-provisioned) and see what needs to be changed.
 Sounds good!
 
 I think: 
 
 * Might be a good time to look at moving copr to f21? and builders also
   to be f21? (they should come up faster and in general be better than
   the el6 ones currently used, IMHO)

I will start by moving builder to F21 (this really limit us) and once it will 
be finished I move backend and fronted.
I'm afraid that by that time I will move them directly to F22 :)

 * Right now ansible on lockbox01 is using euca2ools to manage cloud
   instances, perhaps we could/should just move to nova now? Or this
   could perhaps wait for us to move lockbox01 to rhel7. 

I learned (the hard way) that nova/cider/neutron etc. commands are deprecated. 
The new preferred way is command
openstack from  python-openstackclient. However Icehouse use 0.3 version and 
you should not think about using this
command unless you have 1.0 version available (Juno or Kilo, not sure).
It probably does not matter if you use ansible modules, but you may consider it 
if you are calling commands directly.
#justsaying

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: users belonging to tenant in FedoraCloud

[Miroslav Suchý]

On 03/25/2015 02:40 AM, Kevin Fenzi wrote:
  The login here doesn't actually work for me in the new cloud; is it
  expected to, or were new passwords allocated?
 These were new randomly generated passwords. I can send you that one,
 but... see below. 

This is kind of tricky.
I have to first set a password for new user. So I can then upload his SSH key.
Then Kevin or somebody else have to tell you that password so you can log in 
and change it.
In OpenStack Kilo will be Forgoten password feature, where you will be able 
to reset it yourself.
However that is far feature for us.


  I'd like to use instances in the cloud for status reporting of tasks
  from the current atomic01.qa machine, and move off the deprecated
  fed-cloud02.
 Thats great, but the new cloud is not yet done. It's not open for
 business. ;) We are going to reinstall it at least one more time
 before we put it in service. So, I can send you info, but you should
 realize any instances you make can and will be completely vaporized
 when we reinstall, and possibly could break when we try and fix
 something else.  
  
  I'm not sure if anyone else was using the old cockpit tenant anymore;
  I briefly repurposed it for rpm-ostree work.  So perhaps the right
  thing here is to request a new atomic tenant.
 Sure, we could rename... makes sense, but we can do that when we
 reinstall more likely. 

OK. I will rename it to atomic in playbook.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: users belonging to tenant in FedoraCloud

[Miroslav Suchý]

On 03/24/2015 11:29 PM, Colin Walters wrote:
   - { name: cockpit, email: 'walt...@redhat.com', tenant: scratch, 
 password: {{cockpit_password}} }

Colin,
to which FAS account this maps? I need to know which SSH key I should upload 
for this account. Or you can even provide
me different SSH key, which is not in FAS (but then please send it with GPG 
signed email).

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Filters in our ansible.git

[Miroslav Suchý]

I created ./filter_plugins/openstack.py in our ansible.git to easy writing 
host_vars in our new cloud.
So instead of ids you can write names of networks, images...

So far I tested it on separate machine and it works, when I have this directory 
in ./ and I run ansible playbook in that
directory. And according the ansible documentation the directory structure is 
correct and those filters should be loaded
automaticaly without need to load them directly.

However:
$ sudo  rbac-playbook groups/copr-backend.yml
EXECV: /usr/bin/sudo -i /bin/bash -i -c /usr/bin/python2 
/usr/bin/ansible-playbook
/srv/web/infra/ansible/playbooks/groups/copr-backend.yml

PLAY [check/create instance] **

TASK: [spin UP VM using nova_compute] *
fatal: [copr-be-dev.cloud.fedoraproject.org - 127.0.0.1] = template error 
while templating string: no filter named
'image_name_to_id'

FATAL: all hosts have already failed -- aborting


This playbook use new
  tasks/persistent_cloud_new.yml
so it may not work completely, but the goal is to get past this error. I have 
no clue how I can debug it on lockbox.
Can somebody from @sysadmin-main investigate it please?
-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: crowdsourcing an interview on git

[Miroslav Suchý]

On 03/31/2015 10:48 PM, Matthew Miller wrote:
 * What is your favorite pro tip for using git?

Sometimes git pull takes long time. Sometimes git start garbage collecting in 
situation, where I was under time pressure.
After this line in crontab I have no such problems any more:

40 3 * * * locate --regex /\\\.git\$ | while read a; do ( cd $a; git fetch 
--all -t; git gc --aggressive; ) done
/dev/null 2/dev/null

And this in .gitconfig:
  [alias]
  lol = log --graph --decorate --pretty=oneline --abbrev-commit
Is great to quickly get overview of history (credits goes to jesusr).

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: OpenStack Icehouse + Fedora

[Miroslav Suchý]

On 03/02/2015 10:06 AM, Kashyap Chamarthy wrote:
 https://blog-rcritten.rhcloud.com/?p=5 -- Configure Keystone to use
 SSL in OpenStack

This great reading. I switched keystone to SSL and it works.
I will try to switch rest of the services.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New OpenStack instance - status

[Miroslav Suchý]

On 03/02/2015 04:00 AM, Kevin Fenzi wrote:
 I guess it it only rebooted after
 packstack first runs it could work.

That is what I meant. Only needed once, but still nice to have it automated.

  * routing between compute nodes and controller using public IP does
  not work. Not fatal right now, but nice to have.
 Yeah, not sure about that... 
 
 Other things: 
 
 https for keystone endpoint would be nice.

Not sure if this possible. All those services listen on specified port. E.g:
  http://fed-cloud09.cloud.fedoraproject.org:5000/v2.0
And I doubt that it can understand plain http and https on the same port.
Hmm. I see in /etc/keystone/keystone.conf
[ssl]
enable=False

I will investigate what needs to be done to enable it.

 Need to make sure we can get our ansible to spin up and manage
 instances, etc. 
 
 Perhaps we could spin up a dev copr on it to test... and if all looks
 well do another reinstall/reconfigure cycle and start using it. ;) 

I will try to migrate copr-fe-dev.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Request to become apprentice

[Miroslav Suchý]

On 01/29/2015 10:50 PM, Mikolaj Izdebski wrote:
 What are next steps I need to follow to become apprentice?

As nirik stated, apprentice wiki page is good start.

I would point out
  https://infrastructure.fedoraproject.org/infra/docs/sshaccess.txt
as good starting point
Followed by:
  https://infrastructure.fedoraproject.org/infra/docs/infra-git-repo.txt
And then reading rest of:
  https://infrastructure.fedoraproject.org/infra/docs/
Once you want to do something real and join some sysadmin-* group, you should 
set up:
  https://infrastructure.fedoraproject.org/infra/docs/2-factor.txt

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Ansible question

[Miroslav Suchý]

On 01/29/2015 05:30 PM, Toshio Kuratomi wrote:
   no_log: True

That did the job. Thanks!

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: UserKnownHostsFile for copr-*-dev machines

[Miroslav Suchý]

On 04/02/2015 06:38 PM, Kevin Fenzi wrote:
 The new ansible 1.9 version has a known_hosts module. ;) 
 
 So, stick at the top of your playbook: 
 
 - name: clean out old known_hosts
   local_action: known_hosts path=/root/.ssh/known_hosts 
 name=copr-be-dev.cloud.fedoraproject.org state=absent

Awesome. I will try it.

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Fedora Cloud classroom

[Miroslav Suchý]

On 04/30/2015 01:22 AM, Kevin Fenzi wrote:
 How about Fedora Infrastructure Private Servers and we can just call
 it FIPS. ;) 

Or Fedora Private Cloud - FPC in short :)

/me hides too

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: New Fedora Cloud

[Miroslav Suchý]

On 04/30/2015 02:54 AM, Stephen John Smoogen wrote:
 So the playbook fails currently because the interfaces ifcfg-br-ex is setup 
 and restarted before the software for the
 type of bridge is installed. I am not sure if you want to fix that and have 
 me rebuild one more time? Or just go with
 the rebuild I just finished.

I just done:
check status of network (both NM and network were stop and disabled - really 
weird)
chkconfig network on
shutdown -r +30
service network start
   ... and no able to log/ping that machine any more
after 30 minutes (when the machine restarted) everything was normal and network 
operational.
So I just run those playbook again (both controller and compute nodes) and it 
works now.


I am really not sure how to automate and not sure if this is worth the work.
There are still things which have to be done manually anyway (e.g setting of 
quota on volumes), therefore I would
proceed and leave it aside.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

New Fedora Cloud

[Miroslav Suchý]

Long story short:
  I declare new Fedora Cloud as final.

There is still lot of work, but that will be always the case. Please use it 
(but hold on production things for few days
in case there will be some problem).

I plan to announce Fedora classroom date for those interrested in setup of that 
OpenStack.
-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Fedora Cloud classroom

[Miroslav Suchý]

The log of this classroom:
http://meetbot.fedoraproject.org/fedora-classroom/2015-05-11/fedora-classroom.2015-05-11-15.02.log.html

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Dist Git for Copr

[Miroslav Suchý]

Dne 6.5.2015 v 21:08 Kevin Fenzi napsal(a):
 How about a short term and a longer term plan?
 
 Short term: have copr download and store the src.rpm from build urls. 
 This would at least make things reproducable and at least someone could
 download the src.rpm and send a patch. Along with this a easy way to
 mail the owner of a copr would be good.

Nope, short term is that user select SRPM and do file upload from form -- or 
provide url.
And the result will be stored in dist-git.
So they do not need to host src.rpm on some public site.

 Longer term: use pagure. Have people setup their project there and work
 with pagure folks to integrate copr. We could have an easy 'build this
 in copr' type thing and on the copr side a 'visit this project on
 pagure'. 

I really dislike to store project in pagure. Unless it have git-annex or 
similar and we store there just spec and
SOURCEX as in dist-git.
We could not force users to move their github project to pagure.

 I think trying to do a standalone dist-get will just cause problems
 when/if we want to move to something like pagure and has a lot of
 issues without building up a bunch of infrastructure around it. 

Why it should cause trouble.
The goal is to have literally copy of fedora-dist. Just have it on different 
HW/Storage.
Then every change would be just carbon copy.


-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Upstream for dist-git [RFC]

[Miroslav Suchý]

Hi,
Adam Šamalík took dist-git files from fedora-infra ansible.git. He separated 
what belongs to dist-git itself and what is
Fedora specific and with cooperation of Dan Mach and Palo Babinčák he created 
upstream for dist-git:

  https://github.com/release-engineering/dist-git

This is first attempt and request for comments.

The changes from ansible.git version are described here:
  https://github.com/release-engineering/dist-git/blob/master/changes.txt
and he extracted some code to be configuration driven:
  
https://github.com/release-engineering/dist-git/blob/master/configs/dist-git/dist-git.conf

Feel free to experiment with this project and we are looking for your questions 
and comments.


I have one question thou:
There is no license information in files header but two files:
  scripts/httpd/upload.cgi - GPLv1
  scripts/dist-git/pkgdb_sync_git_branches.py - GPLv2+
Everything else is without license.
Can I assume that we can realease the code under GPLv2+?
The author of upload.cgi seems to be Kevin F. - Kevin, are you willing to 
change license your file to GPLv2+ so we have
uniform license across all files?

Future plans:
 1) Listen to your initial feedback and do alternations according to your 
feedback
 2) After license clarification, announce this project to Red Hat and CentOS 
rel-engs and ask them to merge their
changes of dist-git to this upstream.
 3) Get this package into Fedora distribution
 4) Change Fedora dist-git server to use this package.

 10) Enjoy the benefits of common upstream.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Fedora Cloud questions and proposal

[Miroslav Suchý]

On 04/13/2015 03:54 PM, Kevin Fenzi wrote:
 Yeah. The one place I thought might be nice was if we wanted to reboot
 a compute node to update it, but then I got to thinking, why shouldn't
 we also just reboot the instances too and update them as well? ;) 

I just tried - when I reboot Compute Nodes, all VM located there will switch to 
Shut down state.
And will not be power on automatically. Hard reset will power them on.
However I am afraid that cloud-perstisten.yml will not handle it. When the 
machine is not unreachable it will spin up
new one.

Cold migration works. I tested it yesterday on our instance and it is 
functional.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Fedora Cloud questions and proposal

[Miroslav Suchý]

On 04/10/2015 06:04 PM, Kevin Fenzi wrote:
 I think it might be a good idea to have some swift space setup, but I
 am not sure what use cases we fully have for it, so I would say it
 should be somewhat small. 100GB or something? 
 This would also be backed by the equalogics? Or would it be
 distributed on the nodes?

Swift have built-in split/replica mechanism. So I think that with this size, I 
can steal some space from vg_server of nodes.

 Are there specific cases where live migrations would help us out a lot?

I do not know. Cold migration last several minutes. I agree that we can afford 
it. We are not bank or stock operator
where every outage cost pile of money.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

UserKnownHostsFile for copr-*-dev machines

[Miroslav Suchý]

Valentin and me are now playing quite a lot with copr-*-dev as part of new 
OpenStack testing and I always have to ask
somebody to wipe the entry from known_hosts on lockbox otherwise rbac will 
refuse to connect.

Can I suggest to put into ssh_config on lockbox:

Host copr-be-dev.cloud.fedoraproject.org copr-fe-dev.cloud.fedoraproject.org
UserKnownHostsFile /dev/null

Can I get two +1? And can somebody put it there then?
-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Fed-clou02 migration

[Miroslav Suchý]

Hi,
as you know we have new Fedora Cloud instance.

And we still have the *old* Fedora Cloud instance. I hereby declare fed-cloud02 
a.k.a old Fedora Cloud as deprecated.
There is currently 67 machines in running state. And bunch of VM in shutdown 
state.

I would kindly ask all owners to:
  * not create new VM on fed-cloud02, but rather use fed-cloud09
  * migrate your machines from fed-cloud02 to fed-cloud09
  * terminate your machines on fed-cloud02, which you do not use (especially 
those under transient tenant).

There is no hurry, we are under no press. However I would like to set up some 
dead line. Let say during June and July.
During July I would like to gather list of remaining VMs and write personal 
email to its owners.
In August - if there would be no reaction - I would suggest to power off (not 
terminate!) those remaining VMs and keep
them for brief period.
Sometime during fall terminate all machines and wipe old Fedora Cloud instance.

Once again - this time-frame is just proposal as I would like to avoid having 
old Cloud instance running infinitely.
If you have reason to have running it there and not migrating it, please raise 
your voice and we can alter the schedule.
-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Welcome to the batcave

[Miroslav Suchý]

Dne 25.9.2015 v 18:47 Kevin Fenzi napsal(a):
> I hope everyone enjoys the nice RHEL7, faster, larger, better batcave. 

Hurray!
I especially appreciate the presence of Ansible 1.9, with newer modules (e.g 
dnf module).

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Miroslav Suchý]

Dne 2.12.2015 v 12:14 Pierre-Yves Chibon napsal(a):
> So what do you folks think?

Copr is already migrated to python3 (in upstream) and I'm getting ready to use 
python3 on production servers soon.
But Copr (frontend and backend) is using Fedora it is no big deal anyway.
Well we will migrate frontend to python3, backend have to be postponed since it 
is using ansible modules, which are
python2 only.
Anything new we recently wrote (e.g. keygen) is python3 only. And if possible 
run on RHEL7 (again case of keygen).

Conclusion for me - use common sense. Write code python2/3 compatible. Migrate 
if possible. If libraries are python2
only (openstack, ansible) notify they owners and wait until there is python3 
version or if those authors refuse, choose
other alternative.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: ansible 2.0 on batcave01

[Miroslav Suchý]

Dne 30.1.2016 v 18:40 Kevin Fenzi napsal(a):
> The only other thing related to ansible 2.0 I can think of is that copr
> may need to adjust to the new API if it's using that directly, but it
> can do that on it's own timeframe. 

Nope. We are calling ansible python methods directly just on copr-be. And that 
is different machine, different instance
of ansible.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Support levels and RFR adjustments

[Miroslav Suchý]

Dne 25.1.2016 v 18:06 Kevin Fenzi napsal(a):
> Sure, but I was hoping to phase out cloud.fp.o in favor of
> fedorainfracloud. 

Hmm. There is a *lot* of people who have .repo file pointing to 
copr-be.cloud.fedoraproject.org.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: SPF and email forwarding

[Miroslav Suchý]

Dne 25.1.2016 v 16:46 Kevin Fenzi napsal(a):
> I don't think there's anything we can do here.

We can.
We can enable SRS:
  http://www.openspf.org/SRS
For postfix there exist:
  https://github.com/roehling/postsrsd
Unfortunately not packaged for Fedora.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Support levels and RFR adjustments

[Miroslav Suchý]

Dne 22.1.2016 v 22:33 Kevin Fenzi napsal(a):
> fedoraproject.org - Anything with this domain is something that has
> passed though our RFR process and we support fully. This means we
> update status, we alert on them anytime they have issues, we work on
> them anytime they are down, etc. 
> 
> fedorainfracloud.org - This comes with a lesser level of support,
> simply because our cloud doesn't have any kind of HA setup, so
> it will be down when doing maint or when there's problems. Services in
> this domain may be down when there is scheduled cloud maint. We
> monitor, but don't page off hours, we may work on issues only during
> business hours, etc. Services here may not have passed through our RFR
> process (perhaps we should have a parallel cloud process) 

I personally do not like fedorainfracloud.org. It is fine for hostnames of 
machines in cloud. I would rather see some
subdomain under fedoraproject.org - e.g. .devel.fedoraproject.org or 
.playground.fedoraproject.org or something similar.

I personally dislike the change of copr.fedoraproject.org to 
copr.fedorainfracloud.org. We have been usin this name for
past two years and it is referenced a lot (35k by Google):
  
https://www.google.com/search?lr==cs=%22copr.fedoraproject.org%22=copr.fedoraproject.org%22
This is not pure technical decision but marketing decision as well.
A lot of people are treating it as integral part of Fedora. So instead of 
changing hostname I would rather start RFR
process.

I think that some icon at the top or link at the bottom of page, which will 
clearly state the level of support will do
the same from technical POV, but will be better solution from marketing POV.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

SPF and email forwarding

[Miroslav Suchý]

Today I sent email to packager-spons...@fedoraproject.org and several email 
returned back due SPF protection.
Can someone either implement this:
  http://www.openspf.org/Best_Practices/Forwarding
or turn those email aliases to mailing list?

Mirek


 Přeposlaná zpráva 
Předmět: Undelivered Mail Returned to Sender
Datum: Mon, 25 Jan 2016 10:18:13 + (UTC)
Od: Mail Delivery System 
Komu: msu...@redhat.com

This is the mail system at host bastion02.phx2.fedoraproject.org.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

   The mail system

 (expanded from ):
host mail.leemhuis.info[195.137.212.29] said: 550 5.7.1
: Recipient address rejected: Please see

http://www.openspf.org/Why?s=mfrom;id=msuchy%40redhat.com;ip=209.132.181.3;r=basicbox7.server-home.net
(in reply to RCPT TO command)

 (expanded from ): host
kawka.in.waw.pl[178.62.225.244] said: 550-[SPF] 209.132.181.3 is not
allowed to send mail from redhat.com.  Please 550 see

http://www.openspf.org/Why?scope=mfrom;identity=msu...@redhat.com;ip=209.132.181.3
(in reply to RCPT TO command)

 (expanded from
): host mx.unil.ch[130.223.27.62]
said: 550 209.132.181.3 is not allowed to send mail from redhat.com (SPF
failure) (in reply to RCPT TO command)

 (expanded from
): host
smtpz4.laposte.net[194.117.213.1] said: 550 5.5.0 SPF: 209.132.181.3 is not
allowed to send mail. LPN007_401 (in reply to MAIL FROM command)



Reporting-MTA: dns; bastion02.phx2.fedoraproject.org
X-Postfix-Queue-ID: 23600607EA45
X-Postfix-Sender: rfc822; msuchy@redhat.com
Arrival-Date: Mon, 25 Jan 2016 10:16:53 + (UTC)

Final-Recipient: rfc822; fedora@leemhuis.info
Original-Recipient: rfc822;packager-sponsors@fedoraproject.org
Action: failed
Status: 5.7.1
Remote-MTA: dns; mail.leemhuis.info
Diagnostic-Code: smtp; 550 5.7.1 : Recipient address
rejected: Please see
http://www.openspf.org/Why?s=mfrom;id=msuchy%40redhat.com;ip=209.132.181.3;r=basicbox7.server-home.net

Final-Recipient: rfc822; zbyszek@in.waw.pl
Original-Recipient: rfc822;packager-sponsors@fedoraproject.org
Action: failed
Status: 5.0.0
Remote-MTA: dns; kawka.in.waw.pl
Diagnostic-Code: smtp; 550-[SPF] 209.132.181.3 is not allowed to send mail from
redhat.com.  Please 550 see
http://www.openspf.org/Why?scope=mfrom;identity=msuchy@redhat.com;ip=209.132.181.3

Final-Recipient: rfc822; Christian.Iseli@unil.ch
Original-Recipient: rfc822;packager-sponsors@fedoraproject.org
Action: failed
Status: 5.0.0
Remote-MTA: dns; mx.unil.ch
Diagnostic-Code: smtp; 550 209.132.181.3 is not allowed to send mail from
redhat.com (SPF failure)

Final-Recipient: rfc822; nicolas.mailhot@laposte.net
Original-Recipient: rfc822;packager-sponsors@fedoraproject.org
Action: failed
Status: 5.5.0
Remote-MTA: dns; smtpz4.laposte.net
Diagnostic-Code: smtp; 550 5.5.0 SPF: 209.132.181.3 is not allowed to send
mail. LPN007_401

--- Begin Message ---
Hi,
I prepared short survey for Fedora Sponsors:
  http://goo.gl/forms/2sdgGH5qYG
I will appreciate if you can file in your answers. The survey is anonymous.

I am sending another survey to new people who were recently sponsored. I plan 
to publish the results of both surveys at
Flock this summer and maybe we discover something what can be improved.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

--- End Message ---
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: ansible 2.0 on batcave01

[Miroslav Suchý]

Dne 12.2.2016 v 22:29 Kevin Fenzi napsal(a):
> But it should migrate sometime... 1.9.x isn't going to be supported all
> that much longer, so it should move to the new 2.0 api as soon as it
> can. 

F23 will stay on 1.9.x, isn't it... oh, there is an update filed for F23.
I really hope it did not get into stable.

> Would you like me to file a bug somewhere to track that, or it's
> already on the roadmap?

Not on the roadmap, but I'm putting it there. We will work on that next sprint 
(14 days from now).

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: COPR builds/services for other distros?

[Miroslav Suchý]

Dne 14.3.2016 v 23:58 Patrick Uiterwijk napsal(a):
> I would suggest you to look at our Request For Resource procedure.

OK. I created:
https://fedorahosted.org/fedora-infrastructure/ticket/5166

> - How well would at least the frontend and package download server work behind
>   a load balanced/HA setup (so that at least the directly user-facing 
> components
>   can be made highly available, the backend doesn't need to be HA)
> - What kind of support are we aiming for here? Fully supported core infra 
> app, or
>   still somewhat in the middle?

As we previously discussed, it will be mixed. Backend with builders can stay in 
Fedora Cloud and we will declare that
builders will have different SLA and it can take longer to fix it. Critical 
part is frontend and package repositories.
Dist-git and keygen can stay in cloud as well, as backend is the only one who 
use it.
So fully supported should be just frontend and yum/dnf repositories.

> - Are components still "randomly" crashing? I haven't had to restart services
>   a lot recently, so I think it's in a lot better state than previously?

:)

It never randomly crashed. We always take care to introduce very well hidden 
bugs and we are proud of it. And we always
rejected ideas for just easy-to-solve bugs.

*nod* recently the situation is quite stable. But Copr is still under heavy 
development so from time to time there is
always some bug. But the services itself are pretty stable now.

> - What amount of disk space are you currently using, and what are you 
> expecting
>   to be using over the short to medium term (next year or so, just 
> approximate is
>   fine)?

I put it in RFR ticket.

> - Who is/are going to be the main point of contact? If the previous answer was
>   "core infra app", who do we page at 3AM when everything falls over and we 
> don't
>   understand any of it? (not likely since I now know quite a bit about it, 
> but still
>   I'd like some names for fallback).

I put it in ticket.
Originally it was me and Valentin. Valentin left RH in December and does not 
have time for Fedora any more (I will
likely propose to remove him from sysadmin-cloud in few months). Clime is the 
replacement, and I'm slowly introducing
him into the setup and fedora infrastructure design and tools.
I have two other guys familar with the setup and code base. Adam has already 
root access to prod instance, Jakub not
yet, but he had access to dev instance for long time and in fact he is now 
mature enough that I would propose him to get
access to production instance as well.
So I have 4 contacts and at least one of them should be available at 3AM.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Best practice for 3rd party playbook

[Miroslav Suchý]

Hi,
I am in process in writing playbook for retrace.fedoraproject.org

ABRT team created:
https://github.com/abrt/ansible-role-retrace-server

Which we can use. I just wonder what is best practise for using such 3rd party 
roles?
Should I just copy it into our ansible.git? Or should I use ansible-galaxy 
command to sync it? Manually? Or when the
playbook is run?
-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Fwd: reuse space on retrace02 for retrace01

[Miroslav Suchý]

FYI


 Přeposlaná zpráva 
Předmět: reuse space on retrace02 for retrace01
Datum: Thu, 13 Apr 2017 16:07:37 +0200
Od: Miroslav Suchý <msu...@redhat.com>
Společnost: Red Hat Czech s.r.o.
Komu: abrt-devel-l...@redhat.com

Hi,
for Fedora we have two servers:
  retrace01.qa.fedoraproject.org
  retrace02.qa.fedoraproject.org

retrace.fedoraproject.org is alias for retrace01.
retrace02 is used for devel purposes. And according my quick search it is not 
used. Well the only use is for storing
daily DB backups. We have daily backups since 2015 and it takes 3.8 TB!
We have two other staging servers one for faf and other for retrace.

I plan to delete those daily backups. Which will give us 8 TB free space on 
retrace02.
On the retrace01 we are all the time on 99% of used storage so we need some 
space.
I propose to install glusterFS on retrace01,02 and use the concated space on 
retrace01.
This will gave us more space and we would be able to enable retrace for armhpf 
too.

Unless there is some objections I will work on that next week.
-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: retrace / faf issues

[Miroslav Suchý]

Dne 27.6.2017 v 16:17 Kevin Fenzi napsal(a):
>>> - retrace01.qa.fedoraproject.org is almost constantly alerting on swap
>>> being full. Not sure what to do about this, but perhaps we could add
>>> more swap or somehow limit it to use only memory for normal jobs?
>> Few months ago I set postgresql to use more agressive caching. So that is 
>> main culprint for consuming so much memory.
>> I can easily lower it by few percent. But... I see right now that there is 
>> 16GB swap and 8 GB is free. And total
>> available memory is 16 GB. Because 8GB free swap and 8GB are kernel 
>> buffers/cache. So when you see those errors and what
>> are the exact numbers in those alerts?
> retrace01.qa.fedoraproject.org
> 
> Looks like it alerted just a few min ago:
>   
> Swap
>   
> Notifications for this service have been disabled
>   CRITICAL06-27-2017 14:15:24 0d 0h 11m 8s3/3 SWAP 
> CRITICAL - 7%
> free (1011 MB out of 16383 MB)
>   
> Swap-Is-Low
>   
> Notifications for this service have been disabled
>   CRITICAL06-27-2017 14:15:03 0d 0h 11m 29s   4/4 SWAP 
> CRITICAL - 7%
> free (1002 MB out of 16383 MB)
> 
> 

OK. I lowered the DB cache settings. Please let me know if this happen again.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

SOP for retrace

[Miroslav Suchý]

Hi,
I created new SOP for retrace server (see attachemnt). I tried to git-push it, 
but got:

$ git push
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 2.02 KiB | 0 bytes/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: error: insufficient permission for adding an object to repository 
database ./objects
remote: fatal: failed to write object
error: remote unpack failed: unpack-objects abnormal exit
Killed by signal 1.
To ssh://lockbox01.phx2.fedoraproject.org/git/infra-docs
 ! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to 
'ssh://lockbox01.phx2.fedoraproject.org/git/infra-docs'

This worked in past for me. Did I lost some access? Can I get it back please or 
can someone apply the patch?
-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
>From d524e7cae19eec440a4ad14467ec31a219e11a85 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Miroslav=20Such=C3=BD?= 
Date: Wed, 17 May 2017 11:09:16 +0200
Subject: [PATCH] added retrace SOP

---
 docs/sysadmin-guide/sops/retrace.rst | 127 +++
 1 file changed, 127 insertions(+)
 create mode 100644 docs/sysadmin-guide/sops/retrace.rst

diff --git a/docs/sysadmin-guide/sops/retrace.rst b/docs/sysadmin-guide/sops/retrace.rst
new file mode 100644
index 000..bef3dc9
--- /dev/null
+++ b/docs/sysadmin-guide/sops/retrace.rst
@@ -0,0 +1,127 @@
+.. title: Retrace SOP
+.. slug: infra-retrace
+.. date: 2017-15-17
+.. taxonomy: Contributors/Infrastructure
+
+==
+retrace SOP
+==
+
+  Retrace server - provides complete tracebacks for unhandled crashes and
+  show aggregated information for developers
+
+Contact Information
+---
+
+Owner:
+  Fedora QA Devel, Fedora Infrastructure Team, ABRT team
+Contact:
+  #abrt, #fedora-admin, #fedora-noc
+Servers:
+  retrace*, faf*
+
+Purpose:
+  Provides complete tracebacks for unhandled crashes and
+  show aggregated information for developers.
+
+Description
+---
+
+The physical server runs two main servers: retrace-server and FAF.
+
+Retrace-server
+==
+
+The upstream for retrace server lives at:
+
+  https://github.com/abrt/retrace-server
+
+When user has ABRT client installed and some software crash with unhandled
+exception (traceback or core dump). User can send the request to retrace-server,
+which will install the same set of package plus debuginfo and return to user
+traceback where instead of plain pointers are names of functions. This is more
+useful for further debugging.
+
+Retrace-server can allow uploading of user coredump via WebUI, but this has been
+disabled in Fedora instance.
+
+FAF
+===
+
+When user decide that he want to report his crash then problem is sent to FAF.
+ABRT can be also configured to send microreports automatically without user
+intervention.
+FAF can agreggate the data and similar reports group into one entity (called
+Problems). FAF provides nice WebUI for developers who can see crashes in their
+packages. It lives at:
+
+  https://retrace.fedoraproject.org/faf/
+
+Playbook
+
+
+Playbook is splitted into several roles. There are two main roles
+
+ * abrt/faf
+ * abrt/retrace
+
+These are copy from upstream. You should never update it directly.
+The new version can be fetched from upstram using:
+
+ # cd ansible/abrt
+ # rm -rf faf retrace
+ # ansible-galaxy install -f -r requirements.yml --ignore-errors -p ./
+
+You should review the new differences and commit and push.
+
+Then there are some roles, which are local for our instance:
+
+ * abrt/faf-local - This is run *before* abrt/faf.
+ * abrt/retrace-local - This is run *after* abrt/retrace.
+ * abrt/retrace-local-pre - This is run *before* abrt/retrace.
+
+Services
+
+
+FAF and retrace-server are web applications. So you just need httpd running.
+
+Cron
+
+
+Both FAF and retrace-server has bunch of cron tasks. They are *not* under
+/etc/cron*, but are user crons. They are under users: faf and retrace.
+
+You can list those crons using:
+
+ * sudo -u faf crontab -l
+ * sudo -u retrace crontab -l
+
+All cronjobs should be Ansible managed. Just make sure if you delete some
+cron from Ansible that it does not remain on the server. (not always possible
+with state=absent)
+
+Directories
+---
+
+- /srv/ssd - fast disk, used for PostgreSQL storage
+- /srv - big fat disk, used for storing packages. Mainly:
+  - /srv/faf/lob
+  - /srv/retrace
+- /srv/faf/db-backup/ - Daily backups of DB. No rotating yet. Needs to be
+manually deleted occasionally.
+- /srv/faf/lob/InvalidUReport/ - Invalid reports, can be pretty big.
+No automatic removal too. Need to be purged manually occasionally.
+
+Front-page
+--
+
+The main web page is handled by package abrt-server-info-page, which can be
+controlled using:
+ 

Re: SOP for retrace

[Miroslav Suchý]

Dne 17.5.2017 v 15:14 Jeremy Cline napsal(a):
> Hi,
> 
> We've moved our documentation to Pagure[0] and turned it into a sphinx
> project. The readme provides guidance on how to contribute, but if you
> run into any trouble just let me know!>
> [0] https://pagure.io/infra-docs/

OK I created PR. But I would propose to do in old infra-docs location:

git rm -r *
echo 

Re: How to get invite for joining Fi-apprentice group?

[Miroslav Suchý]

Dne 21.11.2017 v 06:43 Marut Pandya napsal(a):
> How do i join  fi-apprentice group?
> My login name is- pandyamarut 
> please send me invite. 

You should attend Fedora Infra meeting:
https://fedoraproject.org/wiki/Infrastructure/Meetings

introduce yourself, tell everyone what you would like to do.
And then someone (usually nirik) will add you to that group.

Mirek
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Copr storage

[Miroslav Suchý]

We are running out of disk space in Copr. It will probably run out during 
Christmas so I am going to act before the weekend.

Right now we have 4GB volume for production and 4GB for dev machine (just 
because we can, but we never used that). My
plan is to shrink dev volume to 150 GB. And allocate 8 GB for production.

I plan to do that tomorrow.

Mirek
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org