** No longer affects: ubuntu
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about this bug go to:
** Changed in: linaro-ci
Status: Fix Committed = Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage
With the pipelining disabled, using the main archive for packages seems to have
fixed the issue.
Have not seen packages installation failure because of the archive problems and
the ec2 slaves deploy are quite stable now.
Closing this bug now, reopen it if it re appears.
Thanks!!!
Deepti.
**
Today we're having 403's on ci.linaro.org again, see attachment for a
sample log.
** Attachment added: i-18e33761.log
https://bugs.launchpad.net/linaro-android-infrastructure/+bug/932088/+attachment/3177116/+files/i-18e33761.log
--
You received this bug notification because you are a
On Wed, Jun 6, 2012 at 12:17 PM, Paul Sokolovsky
932...@bugs.launchpad.netwrote:
Today we're having 403's on ci.linaro.org again, see attachment for a
sample log.
** Attachment added: i-18e33761.log
+ + echo Acquire::http::Pipeline-Depth 0;^M
tee /etc/apt/apt.conf.d/99no-pipelining^M
Acquire::http::Pipeline-Depth 0;^M
Well, this command exactly disables pipelining.
so, what are the exact changes on a-b* that we have that fixes this
problem ?
The best way probably will be to compare
so, what are the exact changes on a-b* that we have that fixes this
problem ?
And it fixed it for Ubuntu version we use on android-build (Natty).
They may be different/additional issues with OS versions we use on ci.*.
That's why I think that we should just switch to custom AMI as a
sustainable
** Changed in: linaro-ci
Status: New = Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about
As suggested in the comment #53 have disabled pipeline in apt, and
switched to s3 mirror and I see the following error :
W: Failed to fetch http://us-
east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/precise/main/source/Sources
403 Forbidden
W: Failed to fetch http://us-
This seems to be stably fixed for linaro-android-infrastructure, closing
in it (may still be pertinent for linaro-ci and ubuntu).
** Changed in: linaro-android-infrastructure
Status: In Progress = Fix Released
--
You received this bug notification because you are a member of Ubuntu
** Changed in: linaro-ci
Milestone: 2012.05 = 2012.06
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about
I've been having failures in cbuild starting a natty instance for the
last 24 hours. Switching to S3 with no pipelining works around the
problem.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
similar errors with me as well since couple of hours
W: Failed to fetch http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists
/precise-updates/universe/binary-amd64/Packages 404 Not Found
W: Failed to fetch http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists
Just for stats, Linaro's https://android-build.linaro.org/jenkins/
didn't have any issues lately (is on S3 mirror w/o pipelining).
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu
This exact problem is happening for me for about 8 hours now. Is anybody
else seeing this, too?
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ natty-updates/main [...]
403 Forbidden
I also tried a workaround proposed by Scott Moser from
This exact problem a few minutes ago and I've been suffering it during
the last hour
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To
Thanks Scott!
I've just re-enabled S3 mirror, it works ok now:
Get:23 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/
natty/main g++-4.5 amd64 4.5.2-8ubuntu4 [6,511 kB]
Get:24 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/
natty/main g++ amd64
As of now (well, for couple of hours at least) we're having following
errors. These are known to happen from time to time, and dissolve after
some time (we usually attribute it to to ongoing package repository
update), but this ones seems like taking longer then usual, so figure
I'd post about it
Switching back to standard EC2 mirror (from S3 one) fixed that for now.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage
On Tue, 22 May 2012, Paul Sokolovsky wrote:
As of now (well, for couple of hours at least) we're having following
errors. These are known to happen from time to time, and dissolve after
some time (we usually attribute it to to ongoing package repository
update), but this ones seems like
** Changed in: linaro-ci
Assignee: (unassigned) = Deepti B. Kalakeri (deeptik)
** Changed in: linaro-ci
Milestone: None = 2012.05
** Changed in: linaro-ci
Importance: Undecided = High
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed
we hit again 404 Not Found error on ci.linaro.org since yesterday. As
mentioned in previous comments, us-east-1.ec2. is duplicated. afaics,
we don't use cloud-init, only sed call manipulates sources.list.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is
** Changed in: linaro-android-infrastructure
Milestone: 2012.04 = 2012.05
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage
Log (i-5adcf83d_slave.log) for the slave using the ami-b5ea34dc instance
which failed with the 404 error attached.
** Attachment added: Log_of_slave_404_error
https://bugs.launchpad.net/linaro-android-infrastructure/+bug/932088/+attachment/3103295/+files/i-5adcf83d_slave.log
--
You
Deepti, Ben pointed out, the log you posted has bad apt sources lines:
http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
not
http://us-east-1.ec2.us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
--
You received this bug notification because you are a member of Ubuntu
Bugs, which
Deepti...you are using the S3 bucket of:
us-east-1.ec2.us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
Which should be:
us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
Note the extra us-east-1.ec2. in the URL's from the error logs. If you
remove that, it should work for you.
--
You
On Wed, Apr 25, 2012 at 8:15 PM, Ben Howard
932...@bugs.launchpad.netwrote:
Deepti...you are using the S3 bucket of:
us-east-1.ec2.us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
Which should be:
us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
Note the extra us-east-1.ec2. in the URL's
On Wed, 25 Apr 2012, Deepti B. Kalakeri wrote:
On Wed, Apr 25, 2012 at 8:15 PM, Ben Howard
932...@bugs.launchpad.netwrote:
Deepti...you are using the S3 bucket of:
us-east-1.ec2.us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
Which should be:
According to the comments in the bug above we have been able to resolve the 404
Not Found error and HashSum issue by using the following 2 things.
1) sudo sed -i.dist
's,archive.ubuntu.com,archive.ubuntu.com.s3.amazonaws.com,g'
/etc/apt/sources.list
2) echo Acquire::http::Pipeline-Depth 0; |
On Tue, 24 Apr 2012, Deepti B. Kalakeri wrote:
According to the comments in the bug above we have been able to resolve the
404 Not Found error and HashSum issue by using the following 2 things.
1) sudo sed -i.dist
's,archive.ubuntu.com,archive.ubuntu.com.s3.amazonaws.com,g'
On Tue, Apr 24, 2012 at 6:30 PM, Scott Moser smo...@ubuntu.com wrote:
On Tue, 24 Apr 2012, Deepti B. Kalakeri wrote:
According to the comments in the bug above we have been able to resolve
the 404 Not Found error and HashSum issue by using the following 2 things.
1) sudo sed -i.dist
On Tue, 24 Apr 2012, Deepti B. Kalakeri wrote:
Well I have logs for the slave that used ami instance ami-b5ea34dc and
failed to start with the 404 failure in the morning.
Now the same slaves using ami-b5ea34dc instance are able to start without
the problem.
I am not an EC2 expert so am not
We just had a case of 10+ broken builds slave being started up in short
time. Typical Jenkins instance init log:
https://ci.linaro.org/jenkins/computer/i-185d097f/log
As you can see, Acquire::http::Pipeline-Depth 0; is set, but still
there's hashsum mismatch error (but the fact that dozen of
Also, if you look that log carefully, you can see following weird thing:
we do two apt-get updates in row, with sleep 2 inbetween. First update
hits *only*
http://security.ubuntu.com
http://archive.ubuntu.com
doesn't hit in-EC2 mirror at all. Second updates starts to hit http
That's with ami-b5ea34dc just in case.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about this bug go to:
Paul, your link to https://ci.linaro.org/jenkins/computer/i-185d097f/log
does not work for me, could you attach full log ?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2
Paul,
I also just noticed, you're not using the S3 backed mirrors:
sudo sed -i.dist
's,archive.ubuntu.com,archive.ubuntu.com.s3.amazonaws.com,g'
/etc/apt/sources.list
I'm not saying there was not a problem, only that I thought (and hoped)
you were using the s3 mirrors. I've just tried
Yes, those slave log links expire with slaves, that's why I pasted it
in . https://bugs.launchpad.net/linaro-android-
infrastructure/+bug/932088/comments/27/+download should give the same
info as if it was attached.
And yes, this latest issue and log from ci.linaro.org, which is not yet
uses S3
But on ci.linaro.org (non-S3 mirror):
$ grep -l -a Hash Sum mismatch *|wc
40 401309
So, I'm going to leverage same config to ci.linaro.org and see how it
goes (it uses different set of AMIs, so results may not match android-
build).
--
You received this bug notification because
Update on android-build.linaro.org (uses S3):
$ grep -l -a -E '^Failed to fetch.+s3' * | tail -n3
2012-04-03-slave-i-69bb4e0e.log
2012-04-04-slave-i-0f32c068.log
2012-04-04-slave-i-a5a153c2.log
$ grep -l -a Hash Sum mismatch * | tail -n3
2012-04-04-slave-i-25629242.log
The 403s are in fact weird, but they are explainable. The reason for
them is that the 'Sources', and 'Packages' files do not exist in
uncompressed form on S3 *or* on the *.archive.ubuntu.com mirrors (see
http://us.archive.ubuntu.com/ubuntu/dists/natty-updates/universe/source/
)
The reason you
James, Thanks for the info, we're looking into it.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about this bug
James, do you have some logs? I am not able to replicate this.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications
I personally not sure what happens here, but:
Just opening http://us-
east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/natty-
updates/ in a browser leads to 403 XML. Ok, suppose it's limited to only
EC2. From out EC2 machine:
ubuntu@ip-10-243-34-224:~$ wget
On the other hand:
$ grep -l -a -E '^Failed to fetch.+s3' * | tail -n3
2012-04-03-slave-i-69bb4e0e.log
2012-04-04-slave-i-0f32c068.log
2012-04-04-slave-i-a5a153c2.log
Last line matches the line in comment #15 which I made shortly before
turning off pipelining. Which means we didn't have (that
Ok, now I eyeballed few logs and tentative conclusion - thank you very
much, turning off pipelining seem to have solved our issues! Not even
those weird 403's don't pop up after 2012-04-04. If you're interested
what it was here's example:
2012-04-04-slave-i-a5a153c2.log:W: Failed to fetch
Our slaves only seem to be failing when accessing http://us-
east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/natty-
updates/ now. ubuntu/dists/natty is fine. Is there a permission problem
with that directory?
--
You received this bug notification because you are a member of Ubuntu
Sorry for delays with responses. The story on our side was that we
implemented auto-retries for failed apt-get invocations in our scripts
with exponential back-off on 2012-03-06, and that fixed it for us, at
least I never saw slaves failed to launch. Soon after that we were
swamped into other
$ grep -l -a -E '^Failed to fetch.+s3' *
...
2012-03-06-slave-i-63954a07.log
2012-03-06-slave-i-7d12c119.log
2012-03-06-slave-i-e1c51a85.log
2012-03-07-slave-i-31c66355.log
2012-03-07-slave-i-fbd9029f.log
2012-03-08-slave-i-5961c23d.log
2012-03-08-slave-i-5f40ee3b.log
echo Acquire::http::Pipeline-Depth 0; | sudo tee /etc/apt/apt.conf.d
/99no-pipelining - was added to our EC2 instance init scripts, I'll be
watching how it goes.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Previous note is about applying to android-build.linaro.org. Now also
applied to ci.linaro.org, where have slave apt-get issues, albeit errors
we get there are different %).
** Changed in: linaro-android-infrastructure
Status: Fix Released = In Progress
** Also affects: linaro-ci
On Wed, 4 Apr 2012, Paul Sokolovsky wrote:
$ grep -l -a -E '^Failed to fetch.+s3' *
...
2012-03-06-slave-i-63954a07.log
2012-04-02-slave-i-7fff1c18.log
snip
2012-04-04-slave-i-0f32c068.log
2012-04-04-slave-i-a5a153c2.log
So yes, we're still having it almost everyday, it's just
On Wed, 4 Apr 2012, Paul Sokolovsky wrote:
Previous note is about applying to android-build.linaro.org. Now also
applied to ci.linaro.org, where have slave apt-get issues, albeit errors
we get there are different %).
Thank you.
--
You received this bug notification because you are a member
I found another bug in APT that appears to explain the hashsum/size
mismatches. See Bug 948461
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent
** Tags added: mirrors
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about this bug go to:
Hi Paul,
Some more debugging info is in bug 948461, but it looks like, for the moment
if you set apt to use 'Acquire::http::Pipeline-Depth=0', then many of your
errors may go away.
Could you please try in your builds to do (prior to even apt-get update):
$ echo Acquire::http::Pipeline-Depth
We worked that around by adding retries for apt-get operations with
exponential backoff and eventual abort after 6 retries (64s wait last
time). Of 51 instances which were launched since that fix was deployed,
18 had retries kick in. And 10 still failed in the end (i.e. after 6
retries). However,
This latest round of issues is tracked in our bug lp:941784, which may
have more info.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To
Here's good AWS forum thread from 2008:
https://forums.aws.amazon.com/thread.jspa?threadID=21514 . According to
it, 403 Forbidden in AWS-speak means there may be unusually long
propagation delay.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed
The 416 errors are caused by Bug 798023.
I did find a minor bug in the flipping code that activates the S3 meta-
data. It should shake out by later tonight. Can you let me know what the
results of the next nights tests are?
--
You received this bug notification because you are a member of
We started to see following issues with beta S3 mirror:
Get:40 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/
natty/main zip amd64 3.0-3build1 [265 kB]
Failed to fetch
Have been working ok for android-build.linaro.org , and at the end of
our monthly cycle, we'd like to close this in linaro-android-
infrastructure project. We still hope this we be open in Ubuntu for
reference and to allow to communicate any issues/observations. Thanks.
** Changed in:
Ok this was deployed for https://android-build.linaro.org/ . Watching
builds, it all goes well.
We'll share issues/comments (if any) of how it goes for Ubuntu EC2 team
info here.
** Changed in: linaro-android-infrastructure
Milestone: None = 2012.02
** Changed in:
Deployed, I mean, source.list editing per
http://groups.google.com/group/ec2ubuntu/browse_thread/thread/507f58dd51b6b631
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2
** Branch linked: lp:~pfalcon/linaro-android-build-tools/ec2-s3-mirror
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage
** Changed in: linaro-android-infrastructure
Importance: Undecided = High
** Changed in: linaro-android-infrastructure
Status: New = Triaged
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Thanks for quick reply, Scott. I'll RFC switching to new S3 sources, and
unless our service stakeholder raise serious concerns with that, going
to try them soon.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
** Project changed: ubuntu-on-ec2 = ubuntu
** Tags added: ec2-images
** Changed in: ubuntu
Importance: Undecided = High
** Changed in: ubuntu
Status: New = Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Hi,
We're hoping to [very soon] retire the old EC2 mirror infrastructure and
replace it with mirrors backed by S3 [1].
When they go live, the rollover will be transparent to the user, but right
now you can selectively choose the S3 mirrors, which should provide better
uptime and speed.
69 matches
Mail list logo
