Re: FBR: Fix the metadata handling for Rawhide messages

2018-04-20 Thread Pierre-Yves Chibon 
On Thu, Apr 19, 2018 at 10:12:53PM +0530, Sayan Chowdhury wrote:
> From 1bf3db45c8154d0a6a0f29836ec14ce24fa404c1 Mon Sep 17 00:00:00 2001
> From: Sayan Chowdhury 
> Date: Thu, 19 Apr 2018 22:07:12 +0530
> Subject: [PATCH 2/2] fedimg: Add the patch for the PR#103, fix processing
>  Rawhide messages
> 
> Signed-off-by: Sayan Chowdhury 
> ---
>  files/hotfix/fedimg/consumers.py | 11 +--
>  1 file changed, 1 insertion(+), 10 deletions(-)
> 
> diff --git a/files/hotfix/fedimg/consumers.py 
> b/files/hotfix/fedimg/consumers.py
> index ce4d662..8195321 100644
> --- a/files/hotfix/fedimg/consumers.py
> +++ b/files/hotfix/fedimg/consumers.py
> @@ -85,7 +85,7 @@ class FedimgConsumer(fedmsg.consumers.FedmsgConsumer):
>  try:
>  compose_metadata =
> fedfind.release.get_release(cid=compose_id).metadata
>  except fedfind.exceptions.UnsupportedComposeError:
> -LOG.debug("%r is unsupported compose" % compose_id)
> +_log.debug("%r is unsupported compose" % compose_id)
>  return
> 
> 
> @@ -106,15 +106,6 @@ class FedimgConsumer(fedmsg.consumers.FedmsgConsumer):
>  compose_metadata, 'images', 'payload',
>  'images', 'AtomicHost', 'x86_64'))
> 
> -images_meta = get_value_from_dict(
> -compose_metadata,
> -'images',
> -'payload',
> -'images',
> -'CloudImages',
> -'x86_64'
> -)
> -
>  if images_meta is None:
>  _log.debug('No compatible image found to process')
>  return

+1 for me


Pierre


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: Fix the metadata handling for Rawhide messages

2018-04-20 Thread Sinny Kumari 
On Fri, Apr 20, 2018 at 2:21 AM, Dusty Mabe  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
>
>
> On 04/19/2018 12:42 PM, Sayan Chowdhury wrote:
> > Hi,
> >
> > Here is a hotfix for fedimg. Right now Rawhide messages are not
> > getting processed. This hotfix will fix the issue.
> >
> > PR for the same[1]
> >
> > [1] https://github.com/fedora-infra/fedimg/pull/103
> >
> > +1s
> >
>
> reviewed the PR - +1, though I would like to include
> https://github.com/fedora-infra/fedimg/pull/104 as well
>
>
As far as I see, changes from PR#104 is included as well in this hotfix.


> Dusty
> -BEGIN PGP SIGNATURE-
>
> iQIzBAEBCAAdFiEEPb6zG5c6sV89tRYPMwLb1zlS5nEFAlrZAV0ACgkQMwLb1zlS
> 5nEscg//QoBhV8d9Iaq1DtEx5FdCEdFXnmba73/zwVXLfNQTLizFWV/FZONabVzb
> R1uH+zVzuPZpBLr68HyoGQLgViC+EHEYyxkJJc4JABk/MeL2kJH8SoABwnKWmAIe
> a2u9JBU+uq2SJOYdaOwHjQFBm2J2OG44vIxosWcR2LO3jqE0pyXtfnWtP8evJZgg
> 3QOtREhdFolWzUdsiX6Y7MPoH2kg3TNwrtHAEnD/92OGZi6W1WXbXPxBWhupo1SG
> NBnPGQ0LhJC0ypPzzCo/oa5gd7bfaf9ntdm+cxiMAV9u6s1zYhGc/NzGU1fZsV38
> OIZFU3uZWu80wnK67QNI/JSXG3/1rzne9rSXZYfinT1Jin/1iVk+Bh+I+t4+pX7a
> h0tRB+qGMRh7hBkGqO9odKWZdjjmLMol0+dkN5W4pHM60kyoI4Gj4MhiLa3qTTOV
> UQ1vhkuW6+e6u33UORXxeO4IyAJwuSPgjcSnA6cDVBS/cgEyir5d+lS5j/g0KYp4
> eTwK9KPxdrZwPPwT5ILL+fDsalbNqYYAc3x9nba3hHzc+kOA4W3ubog8IYB8/eJP
> 8ZIuIuxMYzpzPfwoWHFBmvkjdXLSnJq7jkCQ9VhntCDZAJAe0Mrq46iMRr7DaDO2
> F8dnaREU/6R4PH9W/f9vqNtxixIB8M08v8ZrdBd4txW4jkvIOWs=
> =cZTD
> -END PGP SIGNATURE-
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-leave@lists.
> fedoraproject.org
>



-- 
http://sinny.io/
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: Fix the metadata handling for Rawhide messages

2018-04-19 Thread Stephen John Smoogen 
+1 for the "we need to make this work since it b0rked now"

On 19 April 2018 at 16:51, Dusty Mabe  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
>
>
> On 04/19/2018 12:42 PM, Sayan Chowdhury wrote:
>> Hi,
>>
>> Here is a hotfix for fedimg. Right now Rawhide messages are not
>> getting processed. This hotfix will fix the issue.
>>
>> PR for the same[1]
>>
>> [1] https://github.com/fedora-infra/fedimg/pull/103
>>
>> +1s
>>
>
> reviewed the PR - +1, though I would like to include
> https://github.com/fedora-infra/fedimg/pull/104 as well
>
> Dusty
> -BEGIN PGP SIGNATURE-
>
> iQIzBAEBCAAdFiEEPb6zG5c6sV89tRYPMwLb1zlS5nEFAlrZAV0ACgkQMwLb1zlS
> 5nEscg//QoBhV8d9Iaq1DtEx5FdCEdFXnmba73/zwVXLfNQTLizFWV/FZONabVzb
> R1uH+zVzuPZpBLr68HyoGQLgViC+EHEYyxkJJc4JABk/MeL2kJH8SoABwnKWmAIe
> a2u9JBU+uq2SJOYdaOwHjQFBm2J2OG44vIxosWcR2LO3jqE0pyXtfnWtP8evJZgg
> 3QOtREhdFolWzUdsiX6Y7MPoH2kg3TNwrtHAEnD/92OGZi6W1WXbXPxBWhupo1SG
> NBnPGQ0LhJC0ypPzzCo/oa5gd7bfaf9ntdm+cxiMAV9u6s1zYhGc/NzGU1fZsV38
> OIZFU3uZWu80wnK67QNI/JSXG3/1rzne9rSXZYfinT1Jin/1iVk+Bh+I+t4+pX7a
> h0tRB+qGMRh7hBkGqO9odKWZdjjmLMol0+dkN5W4pHM60kyoI4Gj4MhiLa3qTTOV
> UQ1vhkuW6+e6u33UORXxeO4IyAJwuSPgjcSnA6cDVBS/cgEyir5d+lS5j/g0KYp4
> eTwK9KPxdrZwPPwT5ILL+fDsalbNqYYAc3x9nba3hHzc+kOA4W3ubog8IYB8/eJP
> 8ZIuIuxMYzpzPfwoWHFBmvkjdXLSnJq7jkCQ9VhntCDZAJAe0Mrq46iMRr7DaDO2
> F8dnaREU/6R4PH9W/f9vqNtxixIB8M08v8ZrdBd4txW4jkvIOWs=
> =cZTD
> -END PGP SIGNATURE-
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org



-- 
Stephen J Smoogen.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: Fix the metadata handling for Rawhide messages

2018-04-19 Thread Dusty Mabe 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 04/19/2018 12:42 PM, Sayan Chowdhury wrote:
> Hi,
> 
> Here is a hotfix for fedimg. Right now Rawhide messages are not
> getting processed. This hotfix will fix the issue.
> 
> PR for the same[1]
> 
> [1] https://github.com/fedora-infra/fedimg/pull/103
> 
> +1s
> 

reviewed the PR - +1, though I would like to include 
https://github.com/fedora-infra/fedimg/pull/104 as well

Dusty 
-BEGIN PGP SIGNATURE-

iQIzBAEBCAAdFiEEPb6zG5c6sV89tRYPMwLb1zlS5nEFAlrZAV0ACgkQMwLb1zlS
5nEscg//QoBhV8d9Iaq1DtEx5FdCEdFXnmba73/zwVXLfNQTLizFWV/FZONabVzb
R1uH+zVzuPZpBLr68HyoGQLgViC+EHEYyxkJJc4JABk/MeL2kJH8SoABwnKWmAIe
a2u9JBU+uq2SJOYdaOwHjQFBm2J2OG44vIxosWcR2LO3jqE0pyXtfnWtP8evJZgg
3QOtREhdFolWzUdsiX6Y7MPoH2kg3TNwrtHAEnD/92OGZi6W1WXbXPxBWhupo1SG
NBnPGQ0LhJC0ypPzzCo/oa5gd7bfaf9ntdm+cxiMAV9u6s1zYhGc/NzGU1fZsV38
OIZFU3uZWu80wnK67QNI/JSXG3/1rzne9rSXZYfinT1Jin/1iVk+Bh+I+t4+pX7a
h0tRB+qGMRh7hBkGqO9odKWZdjjmLMol0+dkN5W4pHM60kyoI4Gj4MhiLa3qTTOV
UQ1vhkuW6+e6u33UORXxeO4IyAJwuSPgjcSnA6cDVBS/cgEyir5d+lS5j/g0KYp4
eTwK9KPxdrZwPPwT5ILL+fDsalbNqYYAc3x9nba3hHzc+kOA4W3ubog8IYB8/eJP
8ZIuIuxMYzpzPfwoWHFBmvkjdXLSnJq7jkCQ9VhntCDZAJAe0Mrq46iMRr7DaDO2
F8dnaREU/6R4PH9W/f9vqNtxixIB8M08v8ZrdBd4txW4jkvIOWs=
=cZTD
-END PGP SIGNATURE-
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

FBR: Fix the metadata handling for Rawhide messages

2018-04-19 Thread Sayan Chowdhury 
Hi,

Here is a hotfix for fedimg. Right now Rawhide messages are not
getting processed. This hotfix will fix the issue.

PR for the same[1]

[1] https://github.com/fedora-infra/fedimg/pull/103

+1s

From b52a1dd1eda25bfbade026a1ebde93839fa3379b Mon Sep 17 00:00:00 2001
From: Sayan Chowdhury 
Date: Thu, 19 Apr 2018 22:05:27 +0530
Subject: [PATCH 1/2] fedimg: Add the intial files for PR#103

Signed-off-by: Sayan Chowdhury 
---
 files/hotfix/fedimg/consumers.py | 160 +++
 roles/fedimg/tasks/main.yml  |   8 ++
 2 files changed, 168 insertions(+)
 create mode 100644 files/hotfix/fedimg/consumers.py

diff --git a/files/hotfix/fedimg/consumers.py b/files/hotfix/fedimg/consumers.py
new file mode 100644
index 000..ce4d662
--- /dev/null
+++ b/files/hotfix/fedimg/consumers.py
@@ -0,0 +1,160 @@
+# -*- coding: utf-8 -*-
+# This file is part of fedimg.
+# Copyright (C) 2014-2017 Red Hat, Inc.
+#
+# fedimg is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# fedimg is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public
+# License along with fedimg; if not, see http://www.gnu.org/licenses,
+# or write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+#
+# Authors:  David Gay 
+#   Sayan Chowdhury 
+"""
+This is the `fedmsg consumer`_ that subscribes to the topic emitted after the
+completion of the nightly and production compose. The consumer on receving the
+message uploads the image using the API of the cloud providers.
+"""
+
+import logging
+import multiprocessing.pool
+
+import fedmsg.consumers
+import fedmsg.encoding
+import fedfind.release
+import fedfind.exceptions
+
+import fedimg.uploader
+
+from fedimg.config import PROCESS_COUNT, STATUS_FILTER
+from fedimg.utils import get_rawxz_urls, get_value_from_dict
+
+_log = logging.getLogger(__name__)
+
+
+class FedimgConsumer(fedmsg.consumers.FedmsgConsumer):
+"""
+A `fedmsg consumer`_ that listens to the pungi compose topics and kicks
+of the process to upload the images to various cloud providers.
+
+Attributes:
+topic (str): The topics this consumer is subscribed to. Set to
+``org.fedoraproject.prod.pungi.compose.status.change``.
+config_key (str): The key to set to ``True`` in the fedmsg config to
+enable this consumer. The key is ``fedimgconsumer.prod.enabled``.
+"""
+topic = ['org.fedoraproject.prod.pungi.compose.status.change']
+config_key = "fedimgconsumer.prod.enabled"
+
+def __init__(self, *args, **kwargs):
+_log.info("FedimgConsumer initializing")
+super(FedimgConsumer, self).__init__(*args, **kwargs)
+
+# Threadpool for upload jobs
+_log.info("Creating thread pool of %s process", PROCESS_COUNT)
+self.upload_pool = multiprocessing.pool.ThreadPool(
+processes=PROCESS_COUNT
+)
+_log.info("FedimgConsumer initialized")
+
+def consume(self, msg):
+"""
+This is called when we receive a message matching our topics.
+
+Args:
+msg (dict): The raw message from fedmsg.
+"""
+_log.info('Received %r %r', msg['topic'], msg['body']['msg_id'])
+
+msg_info = msg['body']['msg']
+if msg_info['status'] not in STATUS_FILTER:
+_log.debug('%s is not valid status' % msg_info['status'])
+return
+
+location = msg_info['location']
+compose_id = msg_info['compose_id']
+try:
+compose_metadata =
fedfind.release.get_release(cid=compose_id).metadata
+except fedfind.exceptions.UnsupportedComposeError:
+LOG.debug("%r is unsupported compose" % compose_id)
+return
+
+
+# Till F27, both cloud-base and atomic images were available
+# under variant CloudImages. With F28 and onward releases,
+# cloud-base image compose moved to cloud variant and atomic images
+# moved under atomic variant.
+prev_rel = ['26', '27']
+if msg_info['release_version'] in prev_rel:
+images_meta = get_value_from_dict(
+compose_metadata, 'images', 'payload', 'images', 'CloudImages',
+'x86_64')
+else:
+images_meta = get_value_from_dict(
+compose_metadata, 'images', 'payload', 'images',
+'Cloud', 'x86_64')
+