Re: [openstack-dev] [all] Zuul v3 Rollout Update - devstack-gate issues edition
On 10/12/2017 04:28 PM, Ian Wienand wrote: - logs issues Should be behind us. The logs partition ran out of inodes, causing log upload failures. Pruning jobs should have rectified this. This time it's true :) But please think about this with your jobs, and don't upload hundreds of little files unnecessarily. - Ubuntu package issues You may notice a range of issues with Ubuntu packages. The root cause is that our mirror is behind due a broken reprepro. Thanks to the efforts of jeblair and pabelanger, the ubuntu mirror has been restored. There should be no more issues relating to out of date mirrors. - system-config breakage resolved - devstack-gate cache copying resolved -i __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Zuul v3 Rollout Update - devstack-gate issues edition
On 2017-10-12 11:04:45 -0400 (-0400), Doug Hellmann wrote: [...] > Yesterday jeblair asked about deleting some of the docs-old content, > which I took to be about space but if it's about inodes and the log > pruning isn't helping then we could do some partial cleanup if it would > help. > > What I'm hoping to save for a little longer is the content from > openstack-manuals up to mitaka (newer stuff is relatively easy to > rebuild, but some of that old stuff isn't so much). So, if we can > preserve the diablo, essex, folsom, grizzly, havana, icehouse, juno, > kilo, liberty, and mitaka directories under /afs/openstack.org/docs-old > then anything else can be removed. That was entirely unrelated to the inode count on the logs site, unfortunately--different filesystems and different servers entirely. Thanks for the detail though, that's still stuff we need to know for better pruning down the docs and mirroring volumes in AFS. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Zuul v3 Rollout Update - devstack-gate issues edition
Excerpts from Jeremy Stanley's message of 2017-10-12 14:18:38 +: > On 2017-10-12 16:28:33 +1100 (+1100), Ian Wienand wrote: > [...] > > - logs issues > > > > Should be behind us. The logs partition ran out of inodes, causing > > log upload failures. Pruning jobs should have rectified this. > [...] > > Just getting caught up now that I have caffeine, but this problem > seems to have returned. The pruning passes are still underway (and > likely take days to traverse the ~0.8 billion inodes on that 12TB > volume so that much is expected), but it's apparent we're saving log > files from jobs faster than we can find and delete old ones at this > point. I expect our only recourse for now is to further reduce log > retention to something like 2 weeks instead of 4 until we can get > this under control (if we were already on Zuul v3 we could fold in > draft documentation publishing and combine those volumes to buy us > some breathing room, but such is life). Yesterday jeblair asked about deleting some of the docs-old content, which I took to be about space but if it's about inodes and the log pruning isn't helping then we could do some partial cleanup if it would help. What I'm hoping to save for a little longer is the content from openstack-manuals up to mitaka (newer stuff is relatively easy to rebuild, but some of that old stuff isn't so much). So, if we can preserve the diablo, essex, folsom, grizzly, havana, icehouse, juno, kilo, liberty, and mitaka directories under /afs/openstack.org/docs-old then anything else can be removed. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Zuul v3 Rollout Update - devstack-gate issues edition
On 2017-10-12 16:28:33 +1100 (+1100), Ian Wienand wrote: [...] > - logs issues > > Should be behind us. The logs partition ran out of inodes, causing > log upload failures. Pruning jobs should have rectified this. [...] Just getting caught up now that I have caffeine, but this problem seems to have returned. The pruning passes are still underway (and likely take days to traverse the ~0.8 billion inodes on that 12TB volume so that much is expected), but it's apparent we're saving log files from jobs faster than we can find and delete old ones at this point. I expect our only recourse for now is to further reduce log retention to something like 2 weeks instead of 4 until we can get this under control (if we were already on Zuul v3 we could fold in draft documentation publishing and combine those volumes to buy us some breathing room, but such is life). -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Zuul v3 Rollout Update - devstack-gate issues edition
There are still significant issues - logs issues Should be behind us. The logs partition ran out of inodes, causing log upload failures. Pruning jobs should have rectified this. - Ubuntu package issues You may notice a range of issues with Ubuntu packages. The root cause is that our mirror is behind due a broken reprepro. Unfortunately, we build our daily images against an external upstream mirror, so they have been built using later packages than our un-updated region mirrors provide, leading apt to great confusion. Some debugging notes on reprepro at [1], but I have to conclude the .db files are corrupt and I have no idea how to recreate these other than to start again. I think the most expedient solution here will be to turn /ubuntu on mirrors into a caching reverse proxy for upstream. However; - system-config breakage The system-config gate is broken due to an old pip pin with [2]. However, despite this merging several hours ago, zuulv2 doesn't seem to want to reload to pick this up. I have a suspicion that because it was merged by zuulv3 maybe zuulv2 missed it? I'm not sure, and don't think even turning the jobs -nv will help. - devstack-gate cache copying This means the original devstack-gate cache issues [3] remain unmerged at this point. [1] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-10-12.log.html#t2017-10-12T04:04:16 [2] https://review.openstack.org/511360 [3] https://review.openstack.org/511260 -i __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] Zuul v3 Rollout Update - devstack-gate issues edition
But, Mousie, thou art no thy-lane, In proving foresight may be vain; The best-laid schemes o' mice an' men Gang aft agley, An' lea'e us nought but grief an' pain, For promis'd joy! - To a Mouse, Rabbie Burns, 1785 We have awoken this fine morning to find ourselves having two different devstack-gate issues[1][2] that are related neither to each other, nor to Zuul v3 itself (although one of them only surfaces in Zuul v3) Given the typically long iteration time on working through base devstack-gate issues, it seems rather imprudent to flip the v3 switch until they are sorted. Consider the rollout on hold until the devstack-gate issues are sorted. We'll follow up when we it's a go again. Thanks! Monty [1] Ownership of the in-image artifact cache. Has a patch working through the gate now: https://review.openstack.org/#/c/511260/ [2] Issue with our Ubuntu mirrors being out of sync causing package version conflicts between mainline and UCA mirrors. Root cause and solutions are being worked. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
