Re: [yocto] Storing Sstate in S3 success stories?
This doesn't seem to be an issue. I have multiple files with plus signs in their names that made it back down to my local cache without requiring a rebuild (including the whole Linux kernel) On Tue, Feb 26, 2019 at 11:35 AM Brian Walsh wrote: > On Mon, Feb 25, 2019 at 8:46 PM Timothy Froehlich > wrote: > > > > I've been spending a bit too long this past week trying to build up a > reproducable build infrastructure in AWS and I've got very little > experience with cloud infrastucture and I'm wondering if I'm going in the > wrong direction. I'm attempting to host my sstate_cache as a mirror in a > private S3 bucket, and I believe I have everything configured properly, > including exposing the bucket to http requests, since I can wget files that > I've previously synced up to the bucket. However if I add in the > SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM) > and barely seems to get anything. The EC2 instance is in the same region as > the S3 bucket, roles have been configured properly to allow access, etc. > > > > I'm not looking for help debugging this, I just want to know whether I'm > right that hosting my sstate in an S3 bucket should work. I've only been > able to find one mention of it being done with no reproduction hints. > > > > A lot of the files end up with plus signs in the name. This causes > problems with retrieving files through http access with S3. S3 > translates all plus signs to spaces, even those in the file path. So > if my-file_v1.0+g1241876 actually exists as named in S3 an http > request for that file will trigger the server to look for > "my-file_v1.0 g1241876" > > I ran into this problem trying to host an opkg repository in S3 for > upgrading. > > It may mostly work for you but there will be many files that it will > never be able to find in your S3 hosted sstate. > > Maybe this has been fixed by AWS. I noticed the problem a year or two ago. > > > https://stackoverflow.com/questions/36734171/how-to-decide-if-the-filename-has-a-plus-sign-in-it#36758133 > https://news.ycombinator.com/item?id=15398804 > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Storing Sstate in S3 success stories?
Hi Timothy, The S3 protocol is HTTP(S) based, the overhead per object is quite significant. This is not much of a problem for large files but the sstate_cache contains mostly lots of really small files. I think in this case you’re better of storing the cache on a secondary EBS volume that you can attach as a regular block device to the EC2 instance. You can swtich on deletion protection to make it survive EC2 termination. Since EBS volumes are quite a bit more expensive than S3 buckets you could make snapshots to transfer the state between build runs in stead then you can destroy the EBS volume when nothing is running. Alls the documentation about EBS is here https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html Cheers, Erik On 26 Feb 2019, 02:45 +0100, Timothy Froehlich , wrote: I've been spending a bit too long this past week trying to build up a reproducable build infrastructure in AWS and I've got very little experience with cloud infrastucture and I'm wondering if I'm going in the wrong direction. I'm attempting to host my sstate_cache as a mirror in a private S3 bucket, and I believe I have everything configured properly, including exposing the bucket to http requests, since I can wget files that I've previously synced up to the bucket. However if I add in the SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM) and barely seems to get anything. The EC2 instance is in the same region as the S3 bucket, roles have been configured properly to allow access, etc. I'm not looking for help debugging this, I just want to know whether I'm right that hosting my sstate in an S3 bucket should work. I've only been able to find one mention of it being done with no reproduction hints. -- ___ yocto mailing list yocto@yoctoproject.org https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.yoctoproject.org%2Flistinfo%2Fyoctodata=02%7C01%7C%7Cdc66c2ab9da44272220b08d69b8c2262%7C84df9e7fe9f640afb435%7C1%7C0%7C636867423506178146sdata=TjwcnaFaZQpk9iuxL16%2FosjuqEH2S7aXB16JjBIDGko%3Dreserved=0 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Storing Sstate in S3 success stories?
On Mon, Feb 25, 2019 at 8:46 PM Timothy Froehlich wrote: > > I've been spending a bit too long this past week trying to build up a > reproducable build infrastructure in AWS and I've got very little experience > with cloud infrastucture and I'm wondering if I'm going in the wrong > direction. I'm attempting to host my sstate_cache as a mirror in a private S3 > bucket, and I believe I have everything configured properly, including > exposing the bucket to http requests, since I can wget files that I've > previously synced up to the bucket. However if I add in the SSTATE_MIRRORS to > my build, bitbake slows to a crawl (it's a powerful VM) and barely seems to > get anything. The EC2 instance is in the same region as the S3 bucket, roles > have been configured properly to allow access, etc. > > I'm not looking for help debugging this, I just want to know whether I'm > right that hosting my sstate in an S3 bucket should work. I've only been able > to find one mention of it being done with no reproduction hints. > A lot of the files end up with plus signs in the name. This causes problems with retrieving files through http access with S3. S3 translates all plus signs to spaces, even those in the file path. So if my-file_v1.0+g1241876 actually exists as named in S3 an http request for that file will trigger the server to look for "my-file_v1.0 g1241876" I ran into this problem trying to host an opkg repository in S3 for upgrading. It may mostly work for you but there will be many files that it will never be able to find in your S3 hosted sstate. Maybe this has been fixed by AWS. I noticed the problem a year or two ago. https://stackoverflow.com/questions/36734171/how-to-decide-if-the-filename-has-a-plus-sign-in-it#36758133 https://news.ycombinator.com/item?id=15398804 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Storing Sstate in S3 success stories?
Well, based on the responses above I did some more research and it didn't seem like the file sizes should be causing problems on the scale that I was seeing so I investigated further. I realized that despite my build/sstate directory getting slowly larger, it wasn't actually getting the files and was leaving empty files behind. I tried just changing the https in my SSTATE_MIRRORs line just http and it worked perfectly, pulling down a half gig of sstate before I could tell it was even working. So there was likely some other misconfiguration in our AWS account that caused the https to fail (despite being able to wget individual files using https). Thanks for the responses! On Tue, Feb 26, 2019 at 12:19 AM Erik Hoogeveen wrote: > Hi Timothy, > > The S3 protocol is HTTP(S) based, the overhead per object is quite > significant. This is not much of a problem for large files but the > sstate_cache contains mostly lots of really small files. I think in this > case you’re better of storing the cache on a secondary EBS volume that you > can attach as a regular block device to the EC2 instance. You can swtich on > deletion protection to make it survive EC2 termination. > > Since EBS volumes are quite a bit more expensive than S3 buckets you > could make snapshots to transfer the state between build runs in stead then > you can destroy the EBS volume when nothing is running. > > Alls the documentation about EBS is here > https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html > > Cheers, > Erik > On 26 Feb 2019, 02:45 +0100, Timothy Froehlich , > wrote: > > I've been spending a bit too long this past week trying to build up a > reproducable build infrastructure in AWS and I've got very little > experience with cloud infrastucture and I'm wondering if I'm going in the > wrong direction. I'm attempting to host my sstate_cache as a mirror in a > private S3 bucket, and I believe I have everything configured properly, > including exposing the bucket to http requests, since I can wget files that > I've previously synced up to the bucket. However if I add in the > SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM) > and barely seems to get anything. The EC2 instance is in the same region as > the S3 bucket, roles have been configured properly to allow access, etc. > > I'm not looking for help debugging this, I just want to know whether I'm > right that hosting my sstate in an S3 bucket should work. I've only been > able to find one mention of it being done with no reproduction hints. > > -- > ___ > yocto mailing list > yocto@yoctoproject.org > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.yoctoproject.org%2Flistinfo%2Fyoctodata=02%7C01%7C%7Cdc66c2ab9da44272220b08d69b8c2262%7C84df9e7fe9f640afb435%7C1%7C0%7C636867423506178146sdata=TjwcnaFaZQpk9iuxL16%2FosjuqEH2S7aXB16JjBIDGko%3Dreserved=0 > > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Storing Sstate in S3 success stories?
Have you done any wireshark analysis on the traffic? My guess is that the round trip with network latency is bumping your build time by a factor of at least 100x. The state-cache is hammered on continuously, so have probably introduced a significant bottleneck. ..Ch:W.. On Mon, Feb 25, 2019 at 17:45 Timothy Froehlich wrote: > I've been spending a bit too long this past week trying to build up a > reproducable build infrastructure in AWS and I've got very little > experience with cloud infrastucture and I'm wondering if I'm going in the > wrong direction. I'm attempting to host my sstate_cache as a mirror in a > private S3 bucket, and I believe I have everything configured properly, > including exposing the bucket to http requests, since I can wget files that > I've previously synced up to the bucket. However if I add in the > SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM) > and barely seems to get anything. The EC2 instance is in the same region as > the S3 bucket, roles have been configured properly to allow access, etc. > > I'm not looking for help debugging this, I just want to know whether I'm > right that hosting my sstate in an S3 bucket should work. I've only been > able to find one mention of it being done with no reproduction hints. > > -- > ___ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto > -- “I would challenge anyone here to think of a question upon which we once had a scientific answer, however inadequate, but for which now the best answer is a religious one." -Sam Harris -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] Storing Sstate in S3 success stories?
I've been spending a bit too long this past week trying to build up a reproducable build infrastructure in AWS and I've got very little experience with cloud infrastucture and I'm wondering if I'm going in the wrong direction. I'm attempting to host my sstate_cache as a mirror in a private S3 bucket, and I believe I have everything configured properly, including exposing the bucket to http requests, since I can wget files that I've previously synced up to the bucket. However if I add in the SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM) and barely seems to get anything. The EC2 instance is in the same region as the S3 bucket, roles have been configured properly to allow access, etc. I'm not looking for help debugging this, I just want to know whether I'm right that hosting my sstate in an S3 bucket should work. I've only been able to find one mention of it being done with no reproduction hints. -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
