Copr post-mortem

2017-01-02 Thread Miroslav Suchý 
Hi,
today we experienced Copr failure and outage.

First we got reports that builds are failing due failure to sign resulting 
packages.
We had similar problem before Christmas, but investigation of logs shown that 
this was different issue.

It took us a moment to find that we run out of inodes on storage where GPG keys 
are stored.
The storage is only 1GB big, and default size of inodes was just 65k. Although 
we use only 40% of storage, we used all
inodes.

So we moved the data, formatted the partition with high -N option and moved 
data back.

Right now the Copr and package signing is operational again.

Thank to Clime, who helped me to restore the service.
-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org

Re: dist-git is failing to import

2017-01-02 Thread Martin Juhl 
Hi

Happy new year..

Still stuck here...

Any new ideas???

Regards

Martin

- Original meddelelse -
Fra: "mj" 
Til: "Michal Novotny" 
Sendt: mandag, 5. december 2016 19:14:25
Emne: Re: dist-git is failing to import

Have traced it a little further... 

in /usr/lib/python2.7/site-packages/pyrpkg/__init__.py: 

line 2386 (or close by): file_hash = self.lookasidecache.hash_file(f) 

Still no real error though... 

- Original meddelelse - 
Fra: "Michal Novotny"  
Til: "mj"  
Sendt: mandag, 5. december 2016 17:36:31 
Emne: Re: dist-git is failing to import 

On Mon, Dec 5, 2016 at 5:30 PM, Martin Juhl < m...@casalogic.dk > wrote: 


I'm not really any good with writing Python.. I can usually read it... 

Something like this (just gives the same error message)??? 

try: 
commands.upload(upload_files, replace=True) 
except Exception: 
log.exception("Failed") 





try: 
commands.upload(upload_files, replace=True) 
except Exception as e: 
log.exception("traceback: {0}".format(traceback.format_exc)) 
log.exception(str(e)) 

 
/Martin 

- Original meddelelse - 
Fra: "Michal Novotny" < cl...@redhat.com > 
Til: "mj" < m...@casalogic.dk > 
Sendt: mandag, 5. december 2016 17:25:19 
Emne: Re: dist-git is failing to import 

On Mon, Dec 5, 2016 at 5:16 PM, Martin Juhl < m...@casalogic.dk > wrote: 


Hi 

[root@copr02 new]# rpm -aq |grep pyrpkg 
pyrpkg-1.46-5.el7.noarch - It's from EPEL... 

What's your version... 

Seems to be here it fails: 

log.info ("save the source files into lookaside cache") 
commands.upload(upload_files, replace=True) <-- FAILS 
log.debug("git push") 

So is probably the pyrpkg library... what version do you use?? 





The same version and release (packaged for fc23) is on production. 

Try to wrap that code into try-except and print traceback an exception message. 
That might show us something. 

BQ_BEGIN 
/Martin 



- Original meddelelse - 
Fra: "Michal Novotny" < cl...@redhat.com > 
Til: "copr-devel" < copr-devel@lists.fedorahosted.org > 
Sendt: mandag, 5. december 2016 17:05:12 
Emne: Re: dist-git is failing to import 

On Mon, Dec 5, 2016 at 4:46 PM, Martin Juhl < m...@casalogic.dk > wrote: 


When trying to import, dist-git is failing: 

[15:12:20][DEBUG][dist_git.dist_git_importer][dist_git_importer:392] 1. Try to 
get task data 
[15:12:20][INFO][dist_git.dist_git_importer][dist_git_importer:484] 2. Task: 
9-epel7, importing the package: 
http://copr02.casalogic.lan/tmp/tmpuxT8L3/copr-mocks-1.5-1.el7.src.rpm 
[15:12:20][DEBUG][dist_git.dist_git_importer][dist_git_importer:361] 
downloading package 
http://copr02.casalogic.lan/tmp/tmpuxT8L3/copr-mocks-1.5-1.el7.src.rpm 
[15:12:20][DEBUG][dist_git.dist_git_importer][dist_git_importer:428] Verifying 
packagage, getting name and version. 
[15:12:20][DEBUG][dist_git.dist_git_importer][dist_git_importer:457] make sure 
repos exist: mrmeee/Fisk/copr-mocks 
[15:12:20][DEBUG][dist_git.dist_git_importer][dist_git_importer:415] importing 
srpm into the dist-git 
[15:12:20][DEBUG][dist_git.srpm_import][srpm_import:57] repo_dir: 
/tmp/tmpFpUmZy/copr-mocks 
[15:12:20][DEBUG][dist_git.srpm_import][srpm_import:77] clone the pkg 
repository into tmp directory 
[15:12:20][DEBUG][pyrpkg][__init__:1279] Checking out a specific branch 
ssh://copr-dist-git@localhost/mrmeee/Fisk/copr-mocks 
[15:12:20][DEBUG][pyrpkg][__init__:952] Running git clone -b epel7 
ssh://copr-dist-git@localhost/mrmeee/Fisk/copr-mocks --origin origin and 
logging output 
[15:12:20][INFO][pyrpkg][__init__:973] Cloning into 'copr-mocks'... 

[15:12:20][DEBUG][dist_git.srpm_import][srpm_import:79] import the source rpm 
into git and save filenames of sources 
[15:12:20][DEBUG][pyrpkg][__init__:680] Creating repo object from 
/tmp/tmpFpUmZy/copr-mocks 
[15:12:21][DEBUG][pyrpkg][__init__:1105] Running: rpm -qp --nosignature --qf 
%{NAME} /tmp/tmphDba9m/package.src.rpm 
[15:12:21][DEBUG][pyrpkg][__init__:1120] Running: rpm -qpl 
/tmp/tmphDba9m/package.src.rpm 
[15:12:21][INFO][dist_git.srpm_import][srpm_import:86] save the source files 
into lookaside cache 
[15:12:21][ERROR][dist_git.dist_git_importer][dist_git_importer:500] Exception 
raised during srpm import: 
Traceback (most recent call last): 
File "/usr/share/copr/dist_git/dist_git_importer.py", line 493, in do_import 
task.git_hash = self.git_import_srpm(task, fetched_srpm_path) 
File "/usr/share/copr/dist_git/dist_git_importer.py", line 419, in 
git_import_srpm 
return do_git_srpm_import(self.opts, filepath, task, tmp) 
File "/usr/share/copr/dist_git/srpm_import.py", line 112, in do_git_srpm_import 
raise PackageImportException("Failed to import the source rpm: 
{}".format(src_filepath)) 
PackageImportException: Failed to import the source rpm: 
/tmp/tmphDba9m/package.src.rpm 
[15:12:21][DEBUG][dist_git.dist_git_importer][dist_git_importer:465] Sending 
back: 
{"task_id": "9-epel7", "error": "srpm_import_failed"}

Re: COPR chroots additions/deletions

2017-01-02 Thread Michal Novotny 
I am glad it finally works. Let us know in case of any further problems.

Thank you

On Sun, Jan 1, 2017 at 6:16 PM, Sérgio Basto  wrote:

> On Sex, 2016-12-09 at 16:27 +, Sérgio Basto wrote:
> > On Sex, 2016-12-09 at 11:41 +, Michal Novotný wrote:
> > >
> > > >
> > > >
> > > > On Sáb, 2016-12-03 at 22:41 +0100, Michal Novotny wrote:
> > > >
> > > > I built successfully 2 packages for 26 and on 3rd fails [1] with
> > > > same
> > > > error:
> > > > Failed to synchronize cache for repo
> > > > 'coprbecloudfedoraprojectorg_results_sergiomb_kde4for23_fedora26i
> > > > 38
> > > > 6_de
> > > > vel', disabling.
> > > >
> > > > analyzing i686 root.log (to avoid confusions) :
> > > > error: nothing provides libjasper.so.1 needed by kdelibs-
> > > > 6:4.14.24-
> > > > 1.fc26.i686 but I just build kdelibs-6:4.14.26-1.fc26 at first
> > > > build !
> > > >
> > > >
> > > > I didn't understand what you mean by backend_rawhide_to_release
> > > > and
> > > > why
> > > > it fail to synchronize , what I can try ?
> > > Please, find out which package provides libjasper.so.1 and rebuild
> > > it
> > > for f26 before building your "3rd" package.
> > I already did that, kdelibs-6:4.14.26-1.fc26 was rebuilt with new
> > libjasper
>
> Now it is working well [1]
> still have Failed to synchronize cache for repo
> 'coprbecloudfedoraprojectorg_results_sergiomb_kde4for23_fedora26x86_64_devel',
> disabling.
> but it use packages from
> coprbecloudfedoraprojectorg_results_sergiomb_kde4for23_fedora26x86_64
>
> Thank you.
>
> [1]
> https://copr-be.cloud.fedoraproject.org/results/
> sergiomb/kde4for23/fedora-26-x86_64/00494325-kde-baseapps/root.log.gz
>
> > >
> > > >
> > > >
> > > >
> > > > Thanks.
> > > >
> > > > [1]
> > > > https://copr.fedorainfracloud.org/coprs/sergiomb/kde4for23/build/
> > > > 48
> > > > 5509
> > > > /
> > > > https://copr.fedorainfracloud.org/coprs/sergiomb/kde4for23/build/
> > > > 48
> > > > 5516
> > > > /
> > > > https://copr.fedorainfracloud.org/coprs/sergiomb/kde4for23/build/
> > > > 48
> > > > 5591
> > > > /
> > > ___
> > > copr-devel mailing list -- copr-devel@lists.fedorahosted.org
> > > To unsubscribe send an email to copr-devel-leave@lists.fedorahosted
> > > .o
> > > rg
> --
> Sérgio M. B.
>
> ___
> copr-devel mailing list -- copr-devel@lists.fedorahosted.org
> To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
>
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org