fixing wrong ticket link: [1] https://ovirt-jira.atlassian.net/browse/OVIRT-619
On Tue, Jul 5, 2016 at 11:45 AM, Nadav Goldin <[email protected]> wrote: > Opened a ticket for tracking[1], still didn't get to do more tests. > Also, this never happens in VDSM, probably because ovirt-engine.git is > 244MB while vdsm.git is only 52MB. > > another failure: > http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-db-from-3.6_el6_created/44/ > > > > > > [1] https://ovirt-jira.atlassian.net/browse/OVIRT-403 > > On Thu, Jun 30, 2016 at 4:11 AM, Nadav Goldin <[email protected]> wrote: >> I was more or less able to reproduce the problem. I ran git clone on >> ovirt-engine.git from one of the VMs in the Jenkins_CentOS cluster for >> 200 times, with time out set to 90 seconds, and 15 seconds between >> each clone. It had 13/200 failures, which is exactly 6.5%. This >> explains why we don't see it often, it might be more severe as this >> testing was done during the night when Jenkins/Gerrit aren't busy. >> During that time there were few, but not 13, exceptions in gerrit's >> error_log: >> [2016-06-29 19:18:26,933] [NioProcessor-1] WARN >> com.google.gerrit.sshd.GerritServerSession : Exception caught >> org.apache.sshd.common.SshException: Received 97 on unknown channel 0 >> at >> org.apache.sshd.common.session.AbstractConnectionService.getChannel(AbstractConnectionService.java:301) >> ..... >> Since it doesn't have the client IP log, its hard to tell if it is >> correlated, even if it is, not all attempts reached to the exception >> log. So there is a problem, independent of Jenkins itself. Will need >> to dig deeper to find out what is causing it.. >> >> >> >> >> On Wed, Jun 29, 2016 at 8:22 PM, Nadav Goldin <[email protected]> wrote: >>> 1. Its the second time it happens this week[1] >>> 2. Around a month a go, I did a log analyse of how often this happens, >>> and it was more than 10 times a week. >>> 3. After Shlomi resolved few issues on Gerrit, it seem to have gone away. >>> My guess is that this is network related or overload on Gerrit - it >>> either fails when trying to connect to Gerrit, or while cloning(like >>> in [1]). I didn't find any consistency in the error, which makes it >>> hard to reproduce. The current re-trigger Anton did was on a BM metal >>> slave, so I doubt its related to overload on the Jenkins slave itself. >>> >>> [1] >>> http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fc23-x86_64/3281/console >>> >>> On Wed, Jun 29, 2016 at 7:47 PM, Eyal Edri <[email protected]> wrote: >>>> Shlomi arw you running anything on gerrit now? >>>> If you're copying the content please stop as it might affect gerrit >>>> performance. >>>> >>>> On Jun 29, 2016 7:37 PM, "Anton Marchukov" <[email protected]> wrote: >>>>> >>>>> Hello All. >>>>> >>>>> I tried to clone manually and this works: >>>>> >>>>> [amarchuk@ovirt-srv22 ~]$ git clone >>>>> git://gerrit.ovirt.org/ovirt-engine.git >>>>> Cloning into 'ovirt-engine'... >>>>> remote: Counting objects: 784726, done. >>>>> remote: Compressing objects: 100% (204209/204209), done. >>>>> remote: Total 784726 (delta 360293), reused 777805 (delta 358840) >>>>> Receiving objects: 100% (784726/784726), 136.26 MiB | 28.66 MiB/s, done. >>>>> Resolving deltas: 100% (360293/360293), done. >>>>> >>>>> >>>>> But failed in the job >>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fc23-x86_64/3379/console >>>>> >>>>> So it is either not 100% reproducible or some Jenkins issue. >>>>> >>>>> Anybody did anything on Jenkins recently that can be correlated with this? >>>>> >>>>> Also gerrit plugin started to loose events... >>>>> >>>>> Anton. >>>>> >>>>> On Wed, Jun 29, 2016 at 6:14 PM, Piotr Kliczewski >>>>> <[email protected]> wrote: >>>>>> >>>>>> Some time ago jenkins did not update the patches with the score. Now I >>>>>> see that builds are not triggered. One of the builds that I triggered >>>>>> manually [1] failed with: >>>>>> >>>>>> 16:04:47 ERROR: Timeout after 10 minutes >>>>>> 16:04:47 ERROR: Error cloning remote repo 'origin' >>>>>> 16:04:47 hudson.plugins.git.GitException: Command "git fetch --tags >>>>>> --progress git://gerrit.ovirt.org/ovirt-engine.git >>>>>> +refs/heads/*:refs/remotes/origin/*" returned status code 143: >>>>>> 16:04:47 stdout: >>>>>> 16:04:47 stderr: >>>>>> 16:04:47 at >>>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1640) >>>>>> >>>>>> Thanks, >>>>>> Piotr >>>>>> >>>>>> [1] >>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-el7-x86_64/3381/console >>>>>> _______________________________________________ >>>>>> Infra mailing list >>>>>> [email protected] >>>>>> http://lists.ovirt.org/mailman/listinfo/infra >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Anton Marchukov >>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>> >>>>> >>>>> _______________________________________________ >>>>> Infra mailing list >>>>> [email protected] >>>>> http://lists.ovirt.org/mailman/listinfo/infra >>>>> >>>> >>>> _______________________________________________ >>>> Infra mailing list >>>> [email protected] >>>> http://lists.ovirt.org/mailman/listinfo/infra >>>> _______________________________________________ Infra mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/infra
