Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
On 01/06/2015 08:20 PM, Sam P. wrote: - Binaries built for different architectures (i686, x86_64, arm...) Do you have an example for this? I tried physics, but it does not have a native part. I will try and use git-annex or a similar solution, as discussed on irc. IIRC, the Physics bundle contains multiple binaries for box2D, one per architecture. So actually we've been doing multi-arch bundles, a bit like OSX, rather than one bundle per architecture. I'm not sure this scales well, but if we only support 3 archs that's not a concern. However, building multiarch bundles from source can get tricky because they need multiple cross-toolchains and all the target libraries. I bet the Physics' XO bundle was built manually, i.e. by taking pre-built binaries of the box2D from different places and dropping them into the right folder before invoking setup.py. This is far from being ideal, but the complexity of build systems tends to explode when you want to cover all the weird corner cases. - Bundles manually pinned by deployments. This is crucial for deployments that do their own QA and can't tolerate breakage during the school year. The ASLO updater never supported this very well, and thus many large deployments kept using the microformat updater along with a wiki page or a static html page hosted on their infrastructure. IMHO, deployment can easily deploy their own new ASLO copy and manually edit or rollback or not update the data files. It is relatively easy to deploy your own new ASLO frontend and updater dataset (part of frontend). I should probably reach out to developments doing this. Ask Walter, he knows them all. Sorry for dropping so many new requirements on you, but... Replacing a production system is a lot harder than designing something anew :-) That's fine :) Thanks, Sam -- _ // Bernie Innocenti \X/ http://codewiz.org ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
On Tue, Jan 06, 2015 at 05:39:06PM -0500, Bernie Innocenti wrote: +sugar-devel On 01/06/2015 04:52 AM, Sam P. wrote: Hi All, I was wondering about the possibility of putting the new ASLO bundles on the download.sl.o mirror. I was going to put them into a separate folder (named activities2), so there may be duplication from the old ASLO bundles, resulting in more space usage. This will hopefully be short term :) Go ahead, all our mirrors should be fine carrying a few hundred MBs of data. However, I don't believe you for a moment when you say it's going to be short term ;-) Activity bundles? One could also run hardlink(1) on the two trees, which would eliminate the duplication. -- James Cameron http://quozl.linux.org.au/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
On Wed, Jan 07, 2015 at 12:24:47AM +, Sam P. wrote: On Wed Jan 07 2015 at 8:54:19 am James Cameron qu...@laptop.org wrote: On Tue, Jan 06, 2015 at 05:39:06PM -0500, Bernie Innocenti wrote: +sugar-devel On 01/06/2015 04:52 AM, Sam P. wrote: Hi All, I was wondering about the possibility of putting the new ASLO bundles on the download.sl.o mirror. I was going to put them into a separate folder (named activities2), so there may be duplication from the old ASLO bundles, resulting in more space usage. This will hopefully be short term :) Go ahead, all our mirrors should be fine carrying a few hundred MBs of data. However, I don't believe you for a moment when you say it's going to be short term ;-) Activity bundles? One could also run hardlink(1) on the two trees, which would eliminate the duplication. I use a different file system structure to the old ASLO. It is very different, because the old ASLO uses it's own IDs whereas I use bundle IDs. Is it so different that the bundle .xo file content is different? Why? That would mean an .xo downloaded from old ASLO would be different to .xo downloaded for new ASLO, right? -- James Cameron http://quozl.linux.org.au/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
On Wed Jan 07 2015 at 8:54:19 am James Cameron qu...@laptop.org wrote: On Tue, Jan 06, 2015 at 05:39:06PM -0500, Bernie Innocenti wrote: +sugar-devel On 01/06/2015 04:52 AM, Sam P. wrote: Hi All, I was wondering about the possibility of putting the new ASLO bundles on the download.sl.o mirror. I was going to put them into a separate folder (named activities2), so there may be duplication from the old ASLO bundles, resulting in more space usage. This will hopefully be short term :) Go ahead, all our mirrors should be fine carrying a few hundred MBs of data. However, I don't believe you for a moment when you say it's going to be short term ;-) Activity bundles? One could also run hardlink(1) on the two trees, which would eliminate the duplication. I use a different file system structure to the old ASLO. It is very different, because the old ASLO uses it's own IDs whereas I use bundle IDs. ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
On Wed Jan 07 2015 at 8:39:12 am Bernie Innocenti ber...@codewiz.org wrote: +sugar-devel On 01/06/2015 04:52 AM, Sam P. wrote: Hi All, I was wondering about the possibility of putting the new ASLO bundles on the download.sl.o mirror. I was going to put them into a separate folder (named activities2), so there may be duplication from the old ASLO bundles, resulting in more space usage. This will hopefully be short term :) Go ahead, all our mirrors should be fine carrying a few hundred MBs of data. However, I don't believe you for a moment when you say it's going to be short term ;-) By the way, is the old Sugar updater compatible with the new aslo design? If it's not, then we'll have to keep around the old ASLO for years to support the user base. Your spot on there. The New ASLO has a new updater system. I was also wondering about how to move the files from the (new... not yet running) Bot Master on Freedom to the mirror, which is on Sunjammer. I was just planning on using SSHFS, but if you people know anything better I would like to use that. If propagation latency isn't an issue, you could export the files with rsync from sunjammer from a cronjob every 10 minutes. You could either use a public rsyncd, or ssh with a role account not the same account who owns the files). Use a forced command in authorized_keys for extra security. Slow propagation will cause some issues. So just SSHFS it? -- _ // Bernie Innocenti \X/ http://codewiz.org ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
On Wed Jan 07 2015 at 10:44:40 am James Cameron qu...@laptop.org wrote: On Wed, Jan 07, 2015 at 12:24:47AM +, Sam P. wrote: On Wed Jan 07 2015 at 8:54:19 am James Cameron qu...@laptop.org wrote: On Tue, Jan 06, 2015 at 05:39:06PM -0500, Bernie Innocenti wrote: +sugar-devel On 01/06/2015 04:52 AM, Sam P. wrote: Hi All, I was wondering about the possibility of putting the new ASLO bundles on the download.sl.o mirror. I was going to put them into a separate folder (named activities2), so there may be duplication from the old ASLO bundles, resulting in more space usage. This will hopefully be short term :) Go ahead, all our mirrors should be fine carrying a few hundred MBs of data. However, I don't believe you for a moment when you say it's going to be short term ;-) Activity bundles? One could also run hardlink(1) on the two trees, which would eliminate the duplication. I use a different file system structure to the old ASLO. It is very different, because the old ASLO uses it's own IDs whereas I use bundle IDs. Is it so different that the bundle .xo file content is different? Why? That would mean an .xo downloaded from old ASLO would be different to .xo downloaded for new ASLO, right? Well, the bundles should have the same contents as their old ASLO counterparts. Assuming the activity author builds the bundle from git using only setup.py, they will be the same. I could compute hashes of all the old ASLO bundles and link them, but I think we miss lots of potential to use the buildbots. For example, I will start inserting the repository field in all activities that do not have it yet, as well as a url pointing to the download page on the new ASLO. But that is only 1/2 the bundles. For every commit that the bots are notified about, the new ASLO builds a bundle as a development/latest copy. These are available in the new ASLO UI as devel version bundles. I plan to purge these on a cron job, so there is only 1 on the sugarlabs servers for each activity. But this does mean that we have lots of bundles that are not on the old ASLO. ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
On 01/06/2015 07:30 PM, Sam P. wrote: Slow propagation will cause some issues. So just SSHFS it? sshfs introduces too much latency and is unsuitable for serving. Instead, run an scp (or rsync over ssh) every time something changes, like for example as a post-upload hook. -- _ // Bernie Innocenti \X/ http://codewiz.org ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
+sugar-devel On 01/06/2015 04:52 AM, Sam P. wrote: Hi All, I was wondering about the possibility of putting the new ASLO bundles on the download.sl.o mirror. I was going to put them into a separate folder (named activities2), so there may be duplication from the old ASLO bundles, resulting in more space usage. This will hopefully be short term :) Go ahead, all our mirrors should be fine carrying a few hundred MBs of data. However, I don't believe you for a moment when you say it's going to be short term ;-) By the way, is the old Sugar updater compatible with the new aslo design? If it's not, then we'll have to keep around the old ASLO for years to support the user base. I was also wondering about how to move the files from the (new... not yet running) Bot Master on Freedom to the mirror, which is on Sunjammer. I was just planning on using SSHFS, but if you people know anything better I would like to use that. If propagation latency isn't an issue, you could export the files with rsync from sunjammer from a cronjob every 10 minutes. You could either use a public rsyncd, or ssh with a role account not the same account who owns the files). Use a forced command in authorized_keys for extra security. -- _ // Bernie Innocenti \X/ http://codewiz.org ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
On Wed, Jan 07, 2015 at 12:59:58AM +, Sam P. wrote: James wrote: Is it so different that the bundle .xo file content is different? Why? That would mean an .xo downloaded from old ASLO would be different to .xo downloaded for new ASLO, right? Well, the bundles should have the same contents as their old ASLO counterparts. Assuming the activity author builds the bundle from git using only setup.py, they will be the same. I could compute hashes of all the old ASLO bundles and link them, but I think we miss lots of potential to use the buildbots. For example, I will start inserting the repository field in all activities that do not have it yet, as well as a url pointing to the download page on the new ASLO. But that is only 1/2 the bundles. For every commit that the bots are notified about, the new ASLO builds a bundle as a development/latest copy. These are available in the new ASLO UI as devel version bundles. I plan to purge these on a cron job, so there is only 1 on the sugarlabs servers for each activity. But this does mean that we have lots of bundles that are not on the old ASLO. You're losing me here, you seem to be answering a different question. Return to being concrete. Browse-157 is available on the old ASLO using this link: http://download.sugarlabs.org/activities/4024/browse-157.xo This is backed by a file in the filesystem on sunjammer, which happens to have the name browse-157.xo in directory /srv/www-sugarlabs/activities/files/4024 and with an md5sum of b30e53a00416df65fee7ef1496762562 When Browse-157 is available on the new ASLO, how is the storage of the bundle different? And if it is different, why is it different? -- James Cameron http://quozl.linux.org.au/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
On 01/06/2015 07:59 PM, Sam P. wrote: But that is only 1/2 the bundles. For every commit that the bots are notified about, the new ASLO builds a bundle as a development/latest copy. These are available in the new ASLO UI as devel version bundles. I plan to purge these on a cron job, so there is only 1 on the sugarlabs servers for each activity. But this does mean that we have lots of bundles that are not on the old ASLO. We also need a way to keep around multiple bundle versions for the same activities. There are various reasons for this: - Bundles for older versions of Sugar (initially we may need to support only the latest Sugar which gets the new bundles, but eventually we'll end up with the same situation we have today in the old ASLO) - Binaries built for different architectures (i686, x86_64, arm...) - Bundles manually pinned by deployments. This is crucial for deployments that do their own QA and can't tolerate breakage during the school year. The ASLO updater never supported this very well, and thus many large deployments kept using the microformat updater along with a wiki page or a static html page hosted on their infrastructure. Sorry for dropping so many new requirements on you, but... Replacing a production system is a lot harder than designing something anew :-) -- _ // Bernie Innocenti \X/ http://codewiz.org ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [Systems] New ASLO Bundles on the Mirror
Hi Bernie, On Wed, 7 Jan 2015 12:12 pm Bernie Innocenti ber...@codewiz.org wrote: On 01/06/2015 07:59 PM, Sam P. wrote: But that is only 1/2 the bundles. For every commit that the bots are notified about, the new ASLO builds a bundle as a development/latest copy. These are available in the new ASLO UI as devel version bundles. I plan to purge these on a cron job, so there is only 1 on the sugarlabs servers for each activity. But this does mean that we have lots of bundles that are not on the old ASLO. We also need a way to keep around multiple bundle versions for the same activities. There are various reasons for this: - Bundles for older versions of Sugar (initially we may need to support only the latest Sugar which gets the new bundles, but eventually we'll end up with the same situation we have today in the old ASLO) I doing that already :) It works in updater, ui and back end. I revert to the latest version with support for that sugar version. Testing appreciated. - Binaries built for different architectures (i686, x86_64, arm...) Do you have an example for this? I tried physics, but it does not have a native part. I will try and use git-annex or a similar solution, as discussed on irc. - Bundles manually pinned by deployments. This is crucial for deployments that do their own QA and can't tolerate breakage during the school year. The ASLO updater never supported this very well, and thus many large deployments kept using the microformat updater along with a wiki page or a static html page hosted on their infrastructure. IMHO, deployment can easily deploy their own new ASLO copy and manually edit or rollback or not update the data files. It is relatively easy to deploy your own new ASLO frontend and updater dataset (part of frontend). I should probably reach out to developments doing this. Sorry for dropping so many new requirements on you, but... Replacing a production system is a lot harder than designing something anew :-) That's fine :) Thanks, Sam ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
