Re: [Sugar-devel] [SoaS] Recommendations for running Sugar on standard hardware (non-XOs)?
Uff, that is fairly dated! I would install latest Fedora with the Sugar Desktop Group as my first attempt. It won't be OLPC-pretty, it'll expect users to log in with username/password, but other than that it should work. hth, m On Tue, Apr 8, 2014 at 12:39 PM, Christoph Derndorfer christoph.derndor...@gmail.com wrote: Hi David, thanks for the link. Given that this page was last updated back in 2012 I'm wondering whether anyone knows if the instructions are still up-to-date and working? Cheers, Christoph On Tue, Apr 8, 2014 at 8:19 AM, David Beveridge bevh...@gmail.com wrote: I would try this method.., http://wiki.sugarlabs.org/go/Tutorials/Installation/Install_with_liveinst On Tue, Apr 8, 2014 at 4:36 AM, Christoph Derndorfer christoph.derndor...@gmail.com wrote: Hey everyone, a guy I know wants to set up a couple of older laptops with Sugar in a sort of learning lab in Germany and asked me what the best route for doing that was. Since I haven't dabbled in that area in quite a while I'm not sure what the best recommendations are these days: * using the SoaS version from late December? * installing Fedora 20 and running Sugar on top of that? * something entirely different? Any comments, suggestions, links, etc. would be much appreciated. Thanks, Christoph -- Christoph Derndorfer volunteer, OLPC (Austria) [www.olpc.at] editor, OLPC News [www.olpcnews.com] co-founder, TechnikBasteln® [www.technikbasteln.net] e-mail: christ...@derndorfer.eu ___ SoaS mailing list s...@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/soas ___ SoaS mailing list s...@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/soas -- Christoph Derndorfer volunteer, OLPC (Austria) [www.olpc.at] editor, OLPC News [www.olpcnews.com] co-founder, TechnikBasteln® [www.technikbasteln.net] e-mail: christ...@derndorfer.eu ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] shipped languages/locales
On Wed, Mar 26, 2014 at 7:59 AM, Gonzalo Odiard godi...@sugarlabs.orgwrote: I think was done to preserve space. Not sure if that is a concern now. Yep. It was definitely a concern for XO-1 and for XO-1.5 images for 2GB machines. m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Sugar Performance testing
On Mon, Mar 10, 2014 at 8:46 AM, Gonzalo Odiard godi...@sugarlabs.org wrote: Activity startup time is in the shell.log if sugar have debug enabled. +100 on Gonzalo's recommendations. Also a couple of notes, from memory, - Gobject Introspection turned out to be slow to instantiate (IIRC, dsd posted analysis this topic). - The network view leaks AP icons. You need a good number of APs visible for this to be noticeable, if ps_mem.py tells you that the memory usage is growing over time, check whether disabling wifi stops the memory leaks. This should be an easy fix -- I researched this couple years ago. - any tests run on SD cards depend on the SD card performance, not two are alike... hth! m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] One for Grandma and Grandpa
On Sat, Mar 1, 2014 at 10:33 AM, Christian Stroetmann stroetm...@ontolab.com wrote: Please, respect our trademarks and our copyrights. Hi Christian! Have you registered your trademarks anywhere? USPTO doesn't know about them, but you could have a registration elsewhere. And I googled your name/email address, and I find that you've been trolling kernel and FS developers about your copyrighted ideas in FS design. The thread at http://www.kernelhub.org/?p=2msg=281077 is a riot. You are an amazing troll. As Pavel said, fortunately, you can't copyright ideas. Chuck Norris managed to do it once, but you can't. Oh, and http://www.spinics.net/lists/reiserfs-devel/msg01543.html Your website is also a piece of art. This page seems to have all the trademarks you claim to own: http://www.ontonics.com/about/hinweis.htm -- it does not list any of the OLPC ripoff ones, but I see I'm/you're/we're loving it. I am sure you have secured a juicy license from McDonalds - http://www.youtube.com/watch?v=eBlD2N_AwgI This page shows your copyrighted ideas http://www.ontonics.com/innovation/pipeline.htm -- filesystems, suspend-to-NVRAM concepts. And you harass kernel developers who are actually doing stuff -- http://marc.info/?a=12160719884r=1w=2 How about Thanks, but NO, thanks ? m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] One for Grandma and Grandpa
On Wed, Mar 5, 2014 at 10:07 AM, Martin Langhoff martin.langh...@gmail.com wrote: Your website is also a piece of art. Actually, I feel bad. Seems like Apple stole his copyrighted ideas, right down to the ID, http://www.ontolinux.com/community/hardware.htm Christian, you should get in touch with Steve Jobs biographers and claim the credit you deserve on the creation of so many Apple devices, as well as the upcoming Retina Projector. cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] One for Grandma and Grandpa
On Wed, Mar 5, 2014 at 12:11 PM, Christian Stroetmann stroetm...@ontolab.com wrote: The referenced summary on marc.info can not be understood without prior knowledge and reading the whole threads. Oh, I think anyone reading your trolling emails should get some background on you based on your prior posts ( http://marc.info/?a=12160719884r=1w=2 ) , and your amazing website. Readers can form their own opinion. In this respect, it must even be clear for you that I do not say 'Thanks' for your informations. It is a pleasure to exchange emails with you. I'm loving it. Keep telling us all about how you have trademarks on the stuff that we have been working on for years. And the name of your lawyer again. Did kernel devs really steal your copyrighted ideas from OntoFS? What mean people they are! And Apple, robbers! they took your copyrighted ideas too! Can I buy a Retina Projector from you, or one of your distributors? I would rather spend my money on the original inventor! But be careful. Any idea you post on this mailing list, if it's good someone is going to perhaps implement it, and you'll have to argue with them. I'm sure your lawyer is very busy with all that chasing Apple, and you'll have to add chasing Sugar folks to his workload. Not nice. On the other hand, if you stop posting here, you'll be safe. [ everyone else: I'm sorry, this is too much fun, I stop here ] m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] One for Grandma and Grandpa
On Sat, Mar 1, 2014 at 11:11 AM, Christian Stroetmann stroetm...@ontolab.com wrote: Yes indeed we have created a new concept that is basically the transformation of the Sugar learning environment developed solely for very If all you have is a concept, and you are trying to register trademarks based on the work of Sugar and OLPC communities, that's not going to get you any friends in these lists. m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Fwd: [b2g] Introducing B2G to x86
On Fri, Feb 28, 2014 at 12:00 PM, Christian Stroetmann stroetm...@ontolab.com wrote: In this respect I wrote before, that I do not think that Lego is in control over its registered trademark 'LEGO' anymore, This has nothing to do with Sugar, nor OLPC. Perhaps a mailing list about trademarks is more appropriate (hint: you might find you are wrong). Many (most?) active members on this list have contributed a huge amount of their time over years to OLPC and Sugar, with a mix of volunteer and paid-for work. And essentially from the heart. If a person of subgroup is taking trademarks over something this group has worked on with passion is seen as a form of profiteering on the back of our devoted work. Unless you are going to hand over the trademarks to the SugarLabs non-profit, it just does not make you many friends around here. m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Feature proposal: ability to start an activity from inside another
On Tue, Jan 21, 2014 at 3:48 PM, Sam Parkinson sam.parkins...@gmail.com wrote: - Android has a really simple system where you can say go to www.abc.blogs.org in any app that can open it. That opens it in a web browser or in the blogs.org app if installed. This works because the blogs.org app says it can handle these types of links in its manifest. Yep, we discussed Android's 'intents' machinery, which seems the most elegant API design for this, back with Manuq, Gonzalo and dsd in Parana a while back. Manuq -- is building a basic intents API too large a task? Or perhaps changing existing activities is the big elephant? cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Mails caught as spam
Same here -- sugar-devel, IAEP and server-devel. Taught my gmail some manners. thanks for the heads up, m On Thu, Dec 19, 2013 at 8:43 AM, Peter Robinson pbrobin...@gmail.com wrote: On Thu, Dec 19, 2013 at 12:11 PM, Gonzalo Odiard gonz...@laptop.org wrote: Today I have found almost 10 mails from sugar-devel caught as spam by my gmail account. Half of them from Emil Dudev, and a few from a announce to a triage meeting. In all the cases, the mails have links. Anybody knows why these mails are marked in this way? If we have a clear reason, we can make suggestions to avoid this problem. I discovered a number of sugar mails in my gmail spam recently although I couldn't see any pattern in my collection. Peter ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Private vs Public conversations.
On Mon, Oct 28, 2013 at 11:01 PM, David Farning dfarn...@activitycentral.com wrote: I would like to thank everyone who has provided valuable feedback by participating on this thread. Ahem. You are casting fugly accusations, you can't stand back and thank everyone for their valuable feedback. 2. Martin's point about the right hand not always being aware of what the left hand is doing. This unfortunately seems to happen too frequently. Doing is the wrong word, AFAICS. Did anyone working for OLPC _who was a maintainer to SL's trees_ ever rejected or actively ignored patches like what you say? The list of candidates is very short, and they are all highly ethical professionals. m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Private vs Public conversations.
On Wed, Oct 23, 2013 at 12:26 PM, Walter Bender walter.ben...@gmail.com wrote: On Wed, Oct 23, 2013 at 12:04 PM, David Farning dfarn...@activitycentral.com wrote: I just wanted to bump this line of questions as, it is the critical I don't speak on behalf of the Association, but I think your positions are overstated. As far as I know, the Association is still pursing sales of XO laptops and is still supporting XO laptops in the field. Granted the pace of development is slowed and there is -- to my knowledge -- no team in place to develop an follow up to the XO 4.0. I don't have a clue as to what you mean by a technical philanthropy but it remains a non-profit associated dedicated to enhancing learning opportunities through one-to-one computing. The fact that the Association has private-sector partners is nothing new. It has had such partners since its founding in 2006. +1 on Walter's words, David's position is overstated. OLPC has shrunk its Sugar investment, that is true. But on the other points, nothing has changed significantly, OLPC has always had to find sources of funding. Given financial constraints, these are reasonable shifts. That's more like it ;-) there are ways to establish publicly disclosed and mutually beneficial relationships. In the meantime we are happy to provide deployments support while seeding and supporting projects we feel are beneficial to deployments such as School Server Community Edition and Sugar on Ubuntu. Seeding and supporting projects is how it's done. cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Private vs Public conversations.
On Mon, Oct 28, 2013 at 8:14 PM, David Farning dfarn...@activitycentral.com wrote: In a private conversation with an Association employee they told me that they conciser Activity Central a competitor because Activity Central increased deployments expectations. Their strategy with regard to Activity Central was to _not_ accept patches upstream with the goal of causing Activity Central and Dextrose to collapse under its their weight. As it was private conversation I am not sure how widely spread the opinion was held. I object very strongly to those statements; I hope it was not under my watch and I goes very much against the grain of everyone involved with Sugar and OLPC in all the time I was there. While I didn't always agree or like AC's work or strategies, I have been, on and off the record, always in favor of having a strong ecosystem. AC being the main player in that space, this translated in a strong advocacy for AC. As a professional in the foss world, this is not something I would accept in my team, and I don't think anyone in the team had the kind of personality to play such games. There were times where it was easy for OLPC to integrate patches, There were times when it was hard. I tried to signal that in advance because I have been on both sides of the integration game (and I continue to be -- now with Moodle) and I profoundly despise games such as the one being suggested. with a bad taste in my mouth, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Activity Central's Sugar related priorities.
On Mon, Oct 7, 2013 at 12:41 PM, David Farning dfarn...@activitycentral.com wrote: As a more incremental approach, Activity Central will continue our deployment-centric work by porting Dextrose to Ubuntu. From a deploy to XOs PoV that sounds like a ton of work. You'll grind against a lot of little problems. Fedora is no longer behind nor problematic. That was very much true in earlier times. Some innovative things in Fedora (ie: systemd) have been very well integrated with the Sugar stack. And some changes in the Ubuntu pipeline are likely to cause some havoc. From a work for AC customers already using Ubuntu, it probably makes more sense. Still, the odd directions Ubuntu seems to be going are a bit of a wildcard. I honestly hope that they settle a bit and make life for their downstreams a bit easier. cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Fwd: [XSCE] Re: Client side Moodle transparent auth broken in 13.2.0 stable
On Sun, Aug 11, 2013 at 1:15 PM, Jerry Vonau je...@laptop.org.au wrote: Good to hear from you Martin. Just to finish this thread off, I was not able to reproduce this behavior with the XO-1s that I have. This appears to affect Anna's machines only. Thanks for the hints to what might be the root cause. Thanks for the greeting! I had a season of detox after some severe burnout. I'm spending this weekend at Fedora Flock for personal enjoyment, and it's brought me back to the OLPC topic. About Anna's machines -- I suspect either an old OFW or bad/broken/misconfigured manufacturing data. cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Features: Background image in Home View
On Mon, Mar 25, 2013 at 11:48 AM, Manuel Quiñones ma...@laptop.org wrote: My first impression is that this feature can potentially hurt the clean design of Sugar at some points: - the icons color semantic - high contrast, accesibility Agreed. IMHO it can be improved by applying a washout -- mix the image with a white image at 50% alpha. I would like to know more about why is this a long requested feature, Identity is a strong human urge. Kids put stickers on them, adults buy cars of different colors :-) - Adding it only in the home view kills the ilussion of having three zoom views. The background should be displayed in all three, I think. +1 - Why it needs a restart? +1 m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/24/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Window handling for non-Python activities
On Mon, Feb 25, 2013 at 12:22 PM, Daniel Drake d...@laptop.org wrote: This didn't happen on older versions of Sugar (haven't checked why, maybe the failed to start screen didn't exist before?) This patch seems related: http://git.sugarlabs.org/sugar/mainline/commit/dc8f6ed7852f919fe7123d458706fb82430257e9 It was written to address a similar issue with a Flash Sugar app that would swap windows around. It is (or should be) in current Sugar -- was applied a while ago. The apps you mention perhaps use libsugarize? Perhaps something broke it? cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Query regarding xorg-x11-drv-dove package, on XO-4
Thanks for the report! Jon Nettlelton is working on that particular driver, and there was a big overhaul that landed in OS29. Could you create a ticket on dev.laptop.org, against 13.1.0? This has nothing to do with Sugar itself... thanks! m On Thu, Feb 21, 2013 at 12:07 PM, Ajay Garg a...@activitycentral.com wrote: Hi all. This is in relation to ticket http://bugs.sugarlabs.org/ticket/4438 Is there a way to know what are the changes between version-8 (on XO-4 build 30) and version-7 (on XO-4 build 21) for xorg-x11-drv-dove-0.3.5? Regards, Ajay Garg Dextrose Developer Activity Central: http://activitycentral.com ___ Devel mailing list de...@lists.laptop.org http://lists.laptop.org/listinfo/devel -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Hacking onto the appearing and hiding of OSK
On Thu, Jan 24, 2013 at 10:13 AM, Paul Fox p...@laptop.org wrote: i believe sugar already has code to detect the two modes, since that's how it knows whether to present the OSK or not. Yep. Ajay, I think Write shows you the way :-) m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Hacking onto the appearing and hiding of OSK
On Thu, Jan 24, 2013 at 12:02 PM, Gonzalo Odiard gonz...@laptop.org wrote: Write does not know what is the ebook switch state, that logic is in the osk. And that's correct. ebook mode is one reason to show the OSK. There are other reasons -- for example, - accesibility - typing in a different language from what your physical keyboard has That's why I suggest to Ajay to do as Write does :-) m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Issue on the XO when trying to register to a XS
On Mon, Dec 31, 2012 at 6:20 AM, lio...@olpc-france.org wrote: Do you already know this issue? What could be the next step to analyze the issue? What happens if you retry registration from the XO? OS versions on XO, XS? If the XO OS is recent, nothing comes to mind, except a transient network issue. Older XO OSs had some issues around registration, but none that I can recall with this behaviour. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [TRANSIENT] Peer XOs NOT shown in Neighborhood view when Power Management is enabled
On Wed, Dec 19, 2012 at 5:14 AM, Jerry Vonau jvo...@shaw.ca wrote: Think I found the problem, in powerd we're setting WOL based on this string: if grep -qi : :14B2 /proc/net/tcp but that string is not present in /proc/net/tcp so WOL is not set according to ethtool, but that string can be found in /proc/net/tcp6 avahi is bound to tcp6 when viewed with 'netstat -nat' This is reproducible in 12.1.0 and 13.1.0 Arghhh. Ouch. Does it behave better with: if grep -qi : :14B2 /proc/net/tcp* ? m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Porting Sugar on Android (Ice cream sandwich)
Hi Rajiv, your plan seems to have good goals, but is missing some understanding of what you can and cannot do. You cannot run Sugar (a Python-based window manager, based on traditional Linux sw stack) on the Android stack. Way too different. To reach your goals, however, you could try something like... - identify what good Android Apps exist that match Sugar Activities, those you don' t have to port, just replace :-) (ie: don't port Record!) - those Sugar Activities that don't have an Android counterpart, reimplement them as Android apps - Implement a shell -- replacing the standard Android shell -- that has the main features of Sugar shell - Implement some of the key services that make Sugar special: ad-hoc auto-networking, collaboration stack, the Journal. The most important, IMO, is the Journal, as Android's handling of my files is extremely poor. All these services will need integration into the shell and apps. Alternatively, you can wait for Android/Linux stack convergence projects (see Jolla / Sailfish) to mature. Maybe in a year or two there are ways to run Sugar inside Android or Android Apps on a Linux stack. hth, m On Tue, Dec 11, 2012 at 10:24 AM, RJV jv.ravichand...@gmail.com wrote: Hi, We are planning to port Sugar on Android and are faced with these options: 1. Sugar as an application on Android. 2. As a platform on top of the Linux platform. Can the Sugar build be used to create an app bundle to deploy to the app store? Are there any licensing issues? On 2., can someone share their experience, if any, please? -- Regards, Ravichandran J.V. http://ravichandranjv.blogspot.com XO 1.75 - 12.1, 0.96, Build 21, Q4D17 ___ Devel mailing list de...@lists.laptop.org http://lists.laptop.org/listinfo/devel -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Porting Sugar on Android (Ice cream sandwich)
I'm sorry! Looks like I misread your name. A faux pass typical of borderline autistic people -- get all the technical details clearly, mess up the other person's name. Hope the technical part was useful. No offense intended. cheers, martin On Tue, Dec 11, 2012 at 3:12 PM, RJV jv.ravichand...@gmail.com wrote: Thanks, Martin. Btw, where did you deduce the name Rajiv from? :) RJv On Wed, Dec 12, 2012 at 1:13 AM, Martin Langhoff martin.langh...@gmail.com wrote: Implement a shell -- replacing the standard Android shell -- that has the main features of Sugar shell -- Regards, Ravichandran J.V. http://ravichandranjv.blogspot.com XO 1.75 - 12.1, 0.96, Build 21, Q4D17 -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Major bugs for Software-Update
On Tue, Nov 27, 2012 at 11:42 AM, Ajay Garg a...@activitycentral.com wrote: http://bugs.sugarlabs.org/ticket/4274 http://bugs.sugarlabs.org/ticket/4275 Looks like I can play this game too :-/ # 4281 - Activity updater - crashes updating activity m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [TRANSIENT ISSUE] 3G-Modem not being recognised
On Wed, Nov 21, 2012 at 3:07 AM, Alan Jhonn Aguiar Schwyn alan...@hotmail.com wrote: is there a way to block suspend when a class of USB device is plugged? This will be very important.. Disable the automatic power-save (AKA power off usb..) It is already done in many cases. if one device is using the usb comunication. This can be made? What happens if you want to copy a large file to a pendrive? The system power-off the usb ever? In that case, we already do the right thing. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [TRANSIENT ISSUE] 3G-Modem not being recognised
Ajay, folks, please indicate OS version, XO model, steps to repro (even if intermittent), and collect kernel logs so we can see WTH is going on. Otherwise we can only say maybe and speculate -- good stuff for idle converstation at a bar, but not productive if you want to see the problem diagnosed and solved... cheers, martin On Tue, Nov 20, 2012 at 3:33 AM, Ajay Garg a...@activitycentral.com wrote: Hi all. I have been facing transient issues in getting the 3G-Modem recognised (via the appearence of the corresponding icon in the bottom-tray). There really is no fixed pattern; sometimes the icon appears when the modem is inserted; sometimes not. Same is the case when the XO is rebooted, while the modem is inserted. Upon booting, sometimes the icon appears; sometimes not. I don't remember such issues in 12.1.0; there, the icon appeared whenever it ought to. Is someone else too facing such transient issues; or am I the only one hit? Regards, Ajay Garg Dextrose Developer Activity Central: http://activitycentral.com ___ Devel mailing list de...@lists.laptop.org http://lists.laptop.org/listinfo/devel -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [TRANSIENT ISSUE] 3G-Modem not being recognised
On Wed, Nov 21, 2012 at 2:40 AM, James Cameron qu...@laptop.org wrote: I don't think having an end-user with no experience with USB IDs add an entry to the usb-inhibits file, or having to remember to turn off a major feature is the correct long term solution IMHO. You guys are driving OOB, so you short-term you can tweak usb-inhibits with the 3G modems you have. And you've hacked powerd and other bits of infra in the past, if there's a better way (ie: is there a way to block suspend when a class of USB device is plugged? or perhaps an NM hook could set the inhibit-suspend flag?) you are uniquely well positioned to know, try likely approaches, etc... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Renaming the Telescope activity to Scope
On Mon, Nov 19, 2012 at 11:58 PM, Chris Leonard cjlhomeaddr...@gmail.com wrote: As I recall, this activity was already renamed once from xoscope after it became clear it was colliding in name space with an oscilloscpe activity. Renames are a pain in infrastructure, and in upgrade handling for users. I would say prefer to retain the current name until there's an overwhelming case for change. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [FEATURE-BROKEN] Send to Friend feature not working in os11
How about filing a bug? :-) m On Fri, Nov 16, 2012 at 12:26 PM, Ajay Garg a...@activitycentral.com wrote: Hi all. On XO-1.75s, os11 image, the Send to Friend feature isn't working. Is it a known issue? Doing a simple textual send to search on bugs.sugarlabs.org did not yield anything useful. Regards, Ajay Garg Dextrose Developer Activity Central: http://activitycentral.com ___ Devel mailing list de...@lists.laptop.org http://lists.laptop.org/listinfo/devel -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Datastore is broken
Thanks for the heads up! Fix attached, m On Thu, Nov 8, 2012 at 1:11 PM, Daniel Narvaez dwnarv...@gmail.com wrote: Not sure if it breaks anything but it doesn't sound good, with the latest git: Traceback (most recent call last): File /home/buildbot/slave/fedora-17-32bit/build/install/lib/python2.7/site-packages/carquinyol/datastore.py, line 180, in _rebuild_index if os.path.exists(index_path): NameError: global name 'index_path' is not defined ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff 0001-Bring-back-index_path-variable.patch Description: Binary data ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Wed, Nov 7, 2012 at 2:31 AM, Simon Schampijer si...@schampijer.de wrote: Thanks Martin, Sam and Manuel for your work on this. The datastore runs here fine. I pushed the changes. Will be available in 0.97.2. Great! I was going to write a longer thank-you for the review, but I ran out of space. :-) m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Datastore is broken
On Wed, Nov 7, 2012 at 11:51 AM, Simon Schampijer si...@schampijer.de wrote: On 11/07/2012 02:45 PM, Daniel Narvaez wrote: Buildbot tests are failing, apparently because of a datastore issue I owe you two a beer. A mighty big beer. Thanks for spotting this thinko before it landed in an OS build. We fail because we try to access the index at the final path, but at this point we created it at the temp path only. The following patch does work for me. Complete agreement and signoff on your patch. It is a thinko on my part to be checking the size of the on-disk index, (which after a reboot in an ENOSPC condition would easily be 0, or some other corrupt value). The right thing to check is the index on the tmpfs, as Simon's patch does. thanks for the quick patch! m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Datastore is broken
Hmmm, mkdir??? I'd suggest touch /home/user/.sugar/default/datastore/index m On Wed, Nov 7, 2012 at 2:35 PM, Ignacio Rodríguez nachoe...@gmail.com wrote: ¡For repair!: mkdir /home/user/.sugar/default/datastore/index :) 2012/11/7 Martin Langhoff martin.langh...@gmail.com On Wed, Nov 7, 2012 at 11:51 AM, Simon Schampijer si...@schampijer.de wrote: On 11/07/2012 02:45 PM, Daniel Narvaez wrote: Buildbot tests are failing, apparently because of a datastore issue I owe you two a beer. A mighty big beer. Thanks for spotting this thinko before it landed in an OS build. We fail because we try to access the index at the final path, but at this point we created it at the temp path only. The following patch does work for me. Complete agreement and signoff on your patch. It is a thinko on my part to be checking the size of the on-disk index, (which after a reboot in an ENOSPC condition would easily be 0, or some other corrupt value). The right thing to check is the index on the tmpfs, as Simon's patch does. thanks for the quick patch! m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff -- Juan Ignacio Rodríguez CeibalJAM! Activity Central -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Multi-touch test activity
On Sun, Nov 4, 2012 at 8:29 PM, Bert Freudenberg b...@freudenbergs.de wrote: Ah, thanks. I wasn't even going to file a bug report about the aliasing because that is a limitation inherent to the kind of sensor we have. There are of course limitations, but we are in the process of tuning and tightening things on the IR. All sensor types have some forms of aliasing. If you know how, you can confuse capacitive sensors too :-) m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH] Sort the activities in the home in alphabetic order
On Thu, Oct 4, 2012 at 8:06 AM, godi...@sugarlabs.org wrote: The change is done in the favorites view and in the list view What was the sorting before in the fav view? More importantly -- Is this sorting over the English names, or the localized names? I am not sure what we would desire, but I'll note that if it's localized names, then every language will have a diffent home view icon order. I tend to believe that a stable ordering is preferrable... but I understand other factors may apply... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH] Sort the activities in the home in alphabetic order
On Thu, Oct 4, 2012 at 10:33 AM, Gonzalo Odiard godi...@sugarlabs.org wrote: We are sorting by localized name. It's true than the order will be different if you use a different language, but we think is better for the users. Less consistency in icon location better? How? I agree alphabetical is better for list view. Fav view, otoh, _any_ order is ok as long as it is stable... no? m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH] Sort the activities in the home in alphabetic order
On Thu, Oct 4, 2012 at 11:25 AM, Gonzalo Odiard godi...@sugarlabs.org wrote: You have less consistency only if you change the language. I generally agree. My only worry is that documentation screenshots will be all over the place. Have sense use the same order in the listview and in the favorites view too. yeah... ok. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH] Sort the activities in the home in alphabetic order
On Thu, Oct 4, 2012 at 3:15 PM, Gary Martin garycmar...@googlemail.com wrote: Martin, could you clarify this example a little more, did you have some specific documentation task in mind? I want to clarify that I see this patch as a big improvement, even with localized sorting. +100. We often prepare some screenshots for documentation (or the Sugar documentation sprint does). Because Sugar is icon-oriented (and light on strings) in the UI those screenshots are used and reused in documentation most of the time. In my experience, most local documentation efforts are a very quick remix of existing documentation. First-time deployments don't even have XOs in advance for the documentation team to get screenshots. If they have XOs, they often lack the expertise (they are first-time deployments, after all!). Gonzalo has seen this first-hand. With the proposed patch + non-localized strings for fav view sorting the home view remains (mostly) stable across all languages. Yes, it changes if you add/remove/replace activities, but only slightly. The relative position of activity icons remains mostly stable. With the proposed patch sorting on localized strings for fav view, each lang randomizes the sorting. Not so good, in my book, but not the end of the world either. Couple more notes - I completely support localized names sorting for list view. - For fav view the best sorting would be a fixed, OS build-time configurable sorting to group/order activities. cheers, martin - who worries too much about a sorting scheme kids immediately disable using the wherever I want it to be view. -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH v2 Browse] Remove temporary downloaded (cancelled) files SL #3973
On Wed, Oct 3, 2012 at 6:52 AM, Manuel Kaufmann humi...@gmail.com wrote: On Tue, Oct 2, 2012 at 5:35 PM, Martin Langhoff martin.langh...@gmail.com wrote: Actually, the answer is to avoid datetime. It's fine for high level work; when you care about times defined as epoch values, you Yes, you are right. But datetime is much more elegant Not at all. Ignoring the computer is not elegance. In this case, datetime is horrible. If you are writing a calendar (say, google calendar) yes, datetime is your friend. Anyway, I think we must add this function inside a GObject.idle_add() Note that this function needs to run only _once_ and it is extremely cheap/fast. Large files do not take a long time to be removed. Try it. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] GStreamer 1.0.0 has been released
On Wed, Oct 3, 2012 at 11:54 AM, Peter Robinson pbrobin...@gmail.com wrote: You can still port to gstreamer 1.0 on 13.1.0 and just test with non ugly codecs (vorbis / webm etc). Yep. Work now with the ugly codecs. Once vmeta is ready we'll have to retest debug, as it seems to be more fragile. Extra work :-( m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH artwork] Style active state of activities in home view for tap and click feedback
On Wed, Oct 3, 2012 at 4:53 PM, Manuel Quiñones ma...@laptop.org wrote: Add a light grey background, which is the same size as the grey outline that appears when the icon is hovered. Yes! This stuff is needed. Touch feedback needs to be bright, clear, and clearly larger than your finger so that you see it. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH artwork] Style active state of activities in home view for tap and click feedback
On Wed, Oct 3, 2012 at 5:12 PM, Manuel Quiñones ma...@laptop.org wrote: And enlarging a bit: http://dev.laptop.org/~manuq/shell-port/icon-hover2.png http://dev.laptop.org/~manuq/shell-port/icon-tap2.png I am partial towards the enlarged ones :-) cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Remove temporary downloaded (cancelled) files SL #3973
On Mon, Oct 1, 2012 at 3:14 PM, Manuel Kaufmann humi...@gmail.com wrote: This patch is a workaround to that behaviour. Every time that Browse is started it looks for all the .goutputstream files in the instance directory and checks its mtime. If it greater than 1 day it removes the old temporary file. Good, but not enough. Compare mtime also to our boot time (ie: now - uptime). If the file is from before we booted, needs to be nuked. http://planzero.org/blog/2012/01/26/system_uptime_in_python,_a_better_way cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH v2 Browse] Remove temporary downloaded (cancelled) files SL #3973
On Tue, Oct 2, 2012 at 9:59 AM, Manuel Kaufmann humi...@gmail.com wrote:
+try:
+uptime_proc = open('/proc/uptime', 'r').read()
+uptime_seconds = float(uptime_proc.split()[0])
+uptime = datetime.timedelta(seconds=uptime_seconds)
+except:
+logging.warning('/proc/uptime could not be read')
+uptime = None
I like.
+
+temp_path = os.path.join(self.get_activity_root(), 'instance')
+now = datetime.datetime.now()
Tiny optimization and elegance improvement: just calculate your cutoff
here. The cutoff is the largest of
- yesterday (now - 1 day)
- boottime (now - uptime) -- if you managed to read uptime
+for f in os.listdir(temp_path):
+if f.startswith('.goutputstream-'):
+fpath = os.path.join(temp_path, f)
+mtime = os.path.getmtime(fpath)
+mdate = datetime.datetime.fromtimestamp(mtime)
+delta = now - mdate
+if delta.days 0 or \
+(uptime is not None and (now - uptime) mdate):
replace this confitional with
if mdate cutoff:
+logging.warning('Removing old temporary file: %s', fpath)
+try:
+os.remove(fpath)
+except OSError:
+logging.error('Temporary file could not be '
+ 'removed: %s', fpath)
Should catch both OSError _and_ IOError. Or perhaps EnvironmentError
which, strange as the name might sound, is the granddaddy of both.
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH v2 Browse] Remove temporary downloaded (cancelled) files SL #3973
On Tue, Oct 2, 2012 at 2:52 PM, Manuel Kaufmann humi...@gmail.com wrote:
On Tue, Oct 2, 2012 at 1:43 PM, Martin Langhoff
martin.langh...@gmail.com wrote:
+except:
+logging.warning('/proc/uptime could not be read')
+uptime = None
Should I use EnvironmentError[1] here too?
Yep. A bare 'except:' hides other things like syntax errors.
What do you think about this?
Works as pseudocode. But I took a quick look at the datetime docs and
it is... horrible. We want to be doing math on the epoch (seconds
since 1970).
They refuse to document it properly, but the answer to the riddle is
datetime.datetime.now().strftime('%s.%f')
So this is probably closer, using only ints:
boot_time = int(boot_time) # discard the float part
now = datetime.datetime.now().strftime('%s')
cutoff = now - 24 * 60 * 60
if uptime is not None:
boot_time = now - uptime
cutoff = max(cutoff, boot_time)
enjoy,
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH v2 Browse] Remove temporary downloaded (cancelled) files SL #3973
On Tue, Oct 2, 2012 at 3:51 PM, Martin Langhoff
martin.langh...@gmail.com wrote:
Works as pseudocode. But I took a quick look at the datetime docs and
it is... horrible. We want to be doing math on the epoch (seconds
since 1970).
They refuse to document it properly, but the answer to the riddle is
datetime.datetime.now().strftime('%s.%f')
Actually, the answer is to avoid datetime. It's fine for high level
work; when you care about times defined as epoch values, you
import time
now = time.time()
so the code snippet should be (untested):
boot_time = int(boot_time) # discard the float part
now = int(time.time())
cutoff = now - 24 * 60 * 60
if uptime is not None:
boot_time = now - uptime
cutoff = max(cutoff, boot_time)
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 4/7] metadatareader: ignore .hidden files
On Mon, Oct 1, 2012 at 7:06 AM, Manuel Kaufmann humi...@gmail.com wrote: Just a comment: this patch removes the possibility to have a property named ..property, right? is this OK? I hope it is. I don't know, I'm not an experienced Sugar hacker :-) What I can tell you is that I have never seen a property, in our test datastores, that had a name starting with '.'. cheers, m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 1/7] Add ds_clean flag to trigger index rebuilds #2095, #2317
On Sun, Sep 30, 2012 at 12:24 PM, Manuel Kaufmann humi...@gmail.com wrote: On Fri, Sep 21, 2012 at 12:55 AM, Martin Langhoff mar...@laptop.org wrote: This gives us more complete coverage of cases where ENOSPC or other errors are hit when creating/updating datastore entries. I reviewed this patch and I think it's OK. I didn't test it because I'm not sure how to simulate that scenario. Repro procedures are explained in my '0/5' cover letter. Use and abuse dd for your evil purposes. Just a comment: why you didn't add a callback function (_delete_completion_cb) to the delete method and make create / update / delete consistent between them? In that method we can add self._mark_clean My understanding is that delete does not get structured as a trigger/callback because delete is supposed to be quick. Thought: we are dealing near ENOSPC, right? So, maybe the clean_flag will not be created (this is already handled in the patch) and this patch will not take effect the next time the XO restarts and we will be in the same situation again. That is by design! We can only save our clean flags, both for ds and index, if we have disk space. Our state is dirty when they are missing, and they may very well be missing because we could not write them to disk! Think about it this way: when we start up, we default to rebuilding our index, assuming the ds and index may be corrupt. Only if we see the two clean flags we skip the rebuild (which leads to a faster startup). hth, m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 3/7] metadatastore: store/change files on disk defensively #2317
On Mon, Oct 1, 2012 at 12:59 PM, Martin Langhoff mar...@laptop.org wrote: From that section I understand that we are encoding the data into utf-8 if value is unicode, but if it is just a basestring, why we are doing str()? That's actually not changed code.. only changes indentation. I don't like it, I very seriously don't like it, but fixing the problem is outside the scope of this patchset. Why do we do str(), I don't know. I believe it's a noop in this case. Actually, I know why. AIUI, data types coming from dbus can be ints and floats. And that var is coming from dbus. So we cannot write directly to a file without a cast. See http://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html#data-types The problem is that we are dealing with some raw binary data (ie: thumbnails) as if it were a string. You can get away with that as long as your string ops are 8-bit clean. Once we move to utf-8, they won't be. That table also indicates, in the 'notes' column, that all string data must be valid utf8. I don't know how we are getting the thumbnails through, and it seems likely that at some point it'll break, perhaps due to changes in dbus. We must change the thumbnail passing to be in a file, not too far in the future. cheers, m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] last-minute feature proposal for 0.98
On Fri, Sep 28, 2012 at 12:26 PM, Manuel Quiñones ma...@laptop.org wrote: 2012/9/28 Walter Bender walter.ben...@gmail.com: I am hoping to get this into the OLPC 13.1 build. Any change of getting it into Sugar 0.98? Looks like a good addition for me and patch looks fine. Hmmm, lunch time! Ah, ooops! I would add the field to src/carquinyol/indexstore.py _PROPERTIES_NOT_TO_INDEX cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Display only the URL in the URL entry SL #3553
Is this really an improvement in behaviour?
- The tabs are often too small to show the title.
- The title is more important for the user than the URL. No?
cheers,
m
On Mon, Sep 24, 2012 at 5:09 PM, Manuel Kaufmann humi...@gmail.com wrote:
The Title of the current page is no longer shown in the URL
entry. Now, it's only shown in the tab and the current URL is visible
all the time in the URL entry.
Signed-off-by: Manuel Kaufmann humi...@gmail.com
---
browser.py| 13 +
webtoolbar.py | 53 +
2 files changed, 18 insertions(+), 48 deletions(-)
diff --git a/browser.py b/browser.py
index de546f2..1c67beb 100644
--- a/browser.py
+++ b/browser.py
@@ -453,6 +453,10 @@ class Browser(WebKit.WebView):
# Scale text and graphics:
self.set_full_content_zoom(True)
+# This property is used to set the title immediatly the user
+# presses Enter on the URL Entry
+self._loading_uri = None
+
# Reference to the global history and callbacks to handle it:
self._global_history = globalhistory.get_global_history()
self.connect('notify::load-status', self.__load_status_changed_cb)
@@ -542,6 +546,15 @@ class Browser(WebKit.WebView):
def open_new_tab(self, url):
self.emit('new-tab', url)
+def _set_loading_uri(self, uri):
+self._loading_uri = uri
+
+def _get_loading_uri(self):
+return self._loading_uri
+
+loading_uri = GObject.property(type=str, setter=_set_loading_uri,
+ getter=_get_loading_uri)
+
def __run_file_chooser(self, browser, request):
picker = FilePicker(self)
chosen = picker.run()
diff --git a/webtoolbar.py b/webtoolbar.py
index 28bc015..1d531bc 100644
--- a/webtoolbar.py
+++ b/webtoolbar.py
@@ -47,7 +47,6 @@ class WebEntry(iconentry.IconEntry):
GObject.GObject.__init__(self)
self._address = None
-self._title = None
self._search_view = self._search_create_view()
self._search_window = Gtk.Window(type=Gtk.WindowType.POPUP)
@@ -57,8 +56,6 @@ class WebEntry(iconentry.IconEntry):
self.connect('focus-in-event', self.__focus_in_event_cb)
self.connect('populate-popup', self.__populate_popup_cb)
self.connect('key-press-event', self.__key_press_event_cb)
-self.connect('enter-notify-event', self.__enter_notify_event_cb)
-self.connect('leave-notify-event', self.__leave_notify_event_cb)
self._focus_out_hid = self.connect(
'focus-out-event', self.__focus_out_event_cb)
self._change_hid = self.connect('changed', self.__changed_cb)
@@ -79,18 +76,11 @@ class WebEntry(iconentry.IconEntry):
def _set_address(self, address):
self._address = address
-if address is not None and self.props.has_focus:
+if address is not None:
self._set_text(address)
address = GObject.property(type=str, setter=_set_address)
-def _set_title(self, title):
-self._title = title
-if title is not None and not self.props.has_focus:
-self._set_text(title)
-
-title = GObject.property(type=str, setter=_set_title)
-
def _search_create_view(self):
view = Gtk.TreeView()
view.props.headers_visible = False
@@ -146,21 +136,11 @@ class WebEntry(iconentry.IconEntry):
self._search_window.hide()
def __focus_in_event_cb(self, entry, event):
-self._set_text(self._address)
self._search_popdown()
def __focus_out_event_cb(self, entry, event):
-self._set_text(self._title)
self._search_popdown()
-def __enter_notify_event_cb(self, entry, event):
-if not entry.props.has_focus:
-self._set_text(self._address)
-
-def __leave_notify_event_cb(self, entry, event):
-if not entry.props.has_focus:
-self._set_text(self._title)
-
def __view_button_press_event_cb(self, view, event):
model = view.get_model()
@@ -241,7 +221,6 @@ class PrimaryToolbar(ToolbarBase):
self._tabbed_view = tabbed_view
self._loading = False
-self._title = _('Untitled')
toolbar = self.toolbar
activity_button = ActivityToolbarButton(self._activity)
@@ -310,7 +289,6 @@ class PrimaryToolbar(ToolbarBase):
self._loading_changed_hid = None
self._progress_changed_hid = None
self._session_history_changed_hid = None
-self._title_changed_hid = None
self._uri_changed_hid = None
if tabbed_view.get_n_pages():
@@ -324,25 +302,19 @@ class PrimaryToolbar(ToolbarBase):
def _connect_to_browser(self, browser):
if self._browser is not None:
-self._browser.disconnect(self._title_changed_hid)
Re: [Sugar-devel] [PATCH Browse] Display only the URL in the URL entry SL #3553
If there is concensus that it's better, then it's ok. It was one of
those things where I liked Browse more than Chrome, but perhaps it's a
personal quirk.
m
On Thu, Sep 27, 2012 at 10:33 AM, Gonzalo Odiard gonz...@laptop.org wrote:
This change was discussed with Manuq and Gary.
I can't find the discussion now.
CC both
Gonzalo
On Thu, Sep 27, 2012 at 11:25 AM, Martin Langhoff
martin.langh...@gmail.com wrote:
Is this really an improvement in behaviour?
- The tabs are often too small to show the title.
- The title is more important for the user than the URL. No?
cheers,
m
On Mon, Sep 24, 2012 at 5:09 PM, Manuel Kaufmann humi...@gmail.com
wrote:
The Title of the current page is no longer shown in the URL
entry. Now, it's only shown in the tab and the current URL is visible
all the time in the URL entry.
Signed-off-by: Manuel Kaufmann humi...@gmail.com
---
browser.py| 13 +
webtoolbar.py | 53
+
2 files changed, 18 insertions(+), 48 deletions(-)
diff --git a/browser.py b/browser.py
index de546f2..1c67beb 100644
--- a/browser.py
+++ b/browser.py
@@ -453,6 +453,10 @@ class Browser(WebKit.WebView):
# Scale text and graphics:
self.set_full_content_zoom(True)
+# This property is used to set the title immediatly the user
+# presses Enter on the URL Entry
+self._loading_uri = None
+
# Reference to the global history and callbacks to handle it:
self._global_history = globalhistory.get_global_history()
self.connect('notify::load-status',
self.__load_status_changed_cb)
@@ -542,6 +546,15 @@ class Browser(WebKit.WebView):
def open_new_tab(self, url):
self.emit('new-tab', url)
+def _set_loading_uri(self, uri):
+self._loading_uri = uri
+
+def _get_loading_uri(self):
+return self._loading_uri
+
+loading_uri = GObject.property(type=str, setter=_set_loading_uri,
+ getter=_get_loading_uri)
+
def __run_file_chooser(self, browser, request):
picker = FilePicker(self)
chosen = picker.run()
diff --git a/webtoolbar.py b/webtoolbar.py
index 28bc015..1d531bc 100644
--- a/webtoolbar.py
+++ b/webtoolbar.py
@@ -47,7 +47,6 @@ class WebEntry(iconentry.IconEntry):
GObject.GObject.__init__(self)
self._address = None
-self._title = None
self._search_view = self._search_create_view()
self._search_window = Gtk.Window(type=Gtk.WindowType.POPUP)
@@ -57,8 +56,6 @@ class WebEntry(iconentry.IconEntry):
self.connect('focus-in-event', self.__focus_in_event_cb)
self.connect('populate-popup', self.__populate_popup_cb)
self.connect('key-press-event', self.__key_press_event_cb)
-self.connect('enter-notify-event',
self.__enter_notify_event_cb)
-self.connect('leave-notify-event',
self.__leave_notify_event_cb)
self._focus_out_hid = self.connect(
'focus-out-event', self.__focus_out_event_cb)
self._change_hid = self.connect('changed', self.__changed_cb)
@@ -79,18 +76,11 @@ class WebEntry(iconentry.IconEntry):
def _set_address(self, address):
self._address = address
-if address is not None and self.props.has_focus:
+if address is not None:
self._set_text(address)
address = GObject.property(type=str, setter=_set_address)
-def _set_title(self, title):
-self._title = title
-if title is not None and not self.props.has_focus:
-self._set_text(title)
-
-title = GObject.property(type=str, setter=_set_title)
-
def _search_create_view(self):
view = Gtk.TreeView()
view.props.headers_visible = False
@@ -146,21 +136,11 @@ class WebEntry(iconentry.IconEntry):
self._search_window.hide()
def __focus_in_event_cb(self, entry, event):
-self._set_text(self._address)
self._search_popdown()
def __focus_out_event_cb(self, entry, event):
-self._set_text(self._title)
self._search_popdown()
-def __enter_notify_event_cb(self, entry, event):
-if not entry.props.has_focus:
-self._set_text(self._address)
-
-def __leave_notify_event_cb(self, entry, event):
-if not entry.props.has_focus:
-self._set_text(self._title)
-
def __view_button_press_event_cb(self, view, event):
model = view.get_model()
@@ -241,7 +221,6 @@ class PrimaryToolbar(ToolbarBase):
self._tabbed_view = tabbed_view
self._loading = False
-self._title = _('Untitled')
toolbar = self.toolbar
activity_button = ActivityToolbarButton(self._activity)
@@ -310,7
Re: [Sugar-devel] Performace in os3
On Thu, Sep 27, 2012 at 10:40 AM, Manuel Kaufmann humi...@gmail.com wrote: I'm working on my 1.75 XO with os3 and I feel it really slow. There are some examples: At least part of this will be due to a much slower video driver we are using temporarily. But yeah, it hurts and we need to have that fixed. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Display only the URL in the URL entry SL #3553
On Thu, Sep 27, 2012 at 11:46 AM, Gary Martin garycmar...@googlemail.com wrote: + exposing URLs for touch users. The cursor hover is the only way to expose URLs when browsing (unless tap to editing them) Tap-to-switch-to-URL-and-edit, starting with full URL is selected is fine IMHO. You are missing a key -, which is: - with several tabs open, titles are not readable; as can be seen on any modern webbrowser I'm still a +1, just wish the change was more designer 'pretty' ;) If there's consensus, I'll get on with the show, and we move on. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Sat, Sep 22, 2012 at 10:09 PM, Martin Langhoff mar...@laptop.org wrote: On Fri, Sep 21, 2012 at 11:36 AM, Martin Langhoff martin.langh...@gmail.com wrote: Reviews are boring. Testing is more fun! rpms for ARM at Reviews are boring, but necessary. As these patches have seen some reviews, and the ml is a bit awkward for tracking those, I've pushed a 'pu' (proposed updates) branch at http://dev.laptop.org/git/users/martin/sugar-datastore/ I have posted all the patches to the mailing list, and will continue to do so. The pu branch will have the latest, freshest set of patches (so: it rewinds/rebases as needed). m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH] RFC: Optimizer: simplify, avoid hashing files, remove 'checksums' dir
This patch changes the strategy used by optimize.
Instead of maintaining a 'checksum' field for every
file, and maintaining metadata about whether they are
linked in the 'checksums' dir, it will only ever hash
files that match size _exactly_ with a new file.
A DS that does not see 2 files of identical size will
never hash any file.
- Allows the datastore to handle large files (downloads,
video captures) without paying a high 'hashing' price.
- Removes the 'checksums' dirtree; which was redundant:
we can evaluate whether two files are hardlinked comparing
their inode.
- There are no users of 'checksum' metadata field out of
DS, so we remove it completely.
- Removal of checksum metadata and the 'checksums' dirtree
means less risk of out-of-date metadata.
Caveats:
- Assumes no program is producing large files of identical size.
On the XO, this is true. Some file formats with no compression
could trigger unwelcome hashing (ie: an activity that saves TIFF
or BMP files).
- The optimizer should run in a separate process, to avoid making the DS
unresponsive. Was written to avoid races, but would need to flock()
the file between hashing it and hardlinking it.
I need to read up on forking a long-running process from a dbus service.
---
Background: reviewing DS code and datastuctures, the checksums dirtree
and strategy in optimize.py looked problematic to me. If I am saving a large
video, I don't want it hashed (expensive op) if I can (cheaply) check that
no other file is the same size.
I could retain the 'checksum' metadata field -- and add some assurances
that it does not get stale. It would help with large and identical-sized
files, such as screenshots, which may otherwise end up hashed more than
once, perhaps many times.
---
src/carquinyol/datastore.py |4 -
src/carquinyol/filestore.py | 12 ---
src/carquinyol/layoutmanager.py |9 --
src/carquinyol/metadatastore.py |4 +-
src/carquinyol/optimizer.py | 225 ++-
5 files changed, 106 insertions(+), 148 deletions(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index a859dfe..0ee70d2 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -362,9 +362,6 @@ class DataStore(dbus.service.Object):
self._metadata_store.store(uid, props)
self._index_store.store(uid, props)
-if os.path.exists(self._file_store.get_file_path(uid)) and \
-(not file_path or os.path.exists(file_path)):
-self._optimizer.remove(uid)
self._file_store.store(uid, file_path, transfer_ownership,
lambda * args: self._update_completion_cb(async_cb,
async_err_cb,
@@ -489,7 +486,6 @@ class DataStore(dbus.service.Object):
self._mark_dirty()
try:
entry_path = layoutmanager.get_instance().get_entry_path(uid)
-self._optimizer.remove(uid)
self._index_store.delete(uid)
self._file_store.delete(uid)
self._metadata_store.delete(uid)
diff --git a/src/carquinyol/filestore.py b/src/carquinyol/filestore.py
index 0b34b69..38a4018 100644
--- a/src/carquinyol/filestore.py
+++ b/src/carquinyol/filestore.py
@@ -146,18 +146,6 @@ class FileStore(object):
if os.path.exists(file_path):
os.remove(file_path)
-def hard_link_entry(self, new_uid, existing_uid):
-existing_file = layoutmanager.get_instance().get_data_path(
-existing_uid)
-new_file = layoutmanager.get_instance().get_data_path(new_uid)
-
-logging.debug('removing %r', new_file)
-os.remove(new_file)
-
-logging.debug('hard linking %r - %r', new_file, existing_file)
-os.link(existing_file, new_file)
-
-
class AsyncCopy(object):
Copy a file in chunks in the idle loop.
diff --git a/src/carquinyol/layoutmanager.py b/src/carquinyol/layoutmanager.py
index 3179a98..ee8270c 100644
--- a/src/carquinyol/layoutmanager.py
+++ b/src/carquinyol/layoutmanager.py
@@ -35,9 +35,6 @@ class LayoutManager(object):
if not os.path.exists(self._root_path):
os.makedirs(self._root_path)
-self._create_if_needed(self.get_checksums_dir())
-self._create_if_needed(self.get_queue_path())
-
def _create_if_needed(self, path):
if not os.path.exists(path):
os.makedirs(path)
@@ -74,12 +71,6 @@ class LayoutManager(object):
def get_index_path(self):
return os.path.join(self._root_path, 'index')
-def get_checksums_dir(self):
-return os.path.join(self._root_path, 'checksums')
-
-def get_queue_path(self):
-return os.path.join(self.get_checksums_dir(), 'queue')
-
def find_all(self):
uids = []
for f in os.listdir(self._root_path):
diff --git a/src/carquinyol/metadatastore.py b/src/carquinyol/metadatastore.py
index
Re: [Sugar-devel] [PATCH Browse] Error page SL #3500
On Tue, Sep 25, 2012 at 12:43 PM, Manuel Kaufmann humi...@gmail.com wrote: ** Message: console message: @0: Not allowed to load local resource: file:///home/humitos/src/browse/browse.png What do you think? Should I go for the file:// approach anyway? Hmmm, security, tiny performance tweak, what should I choose? ;-) Good catch. I didn' t know it had been locked down this way. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 5/7] indexstore: exit on _flush() errors, work on tmpdir
Preparations for better index management.
- open_index() now can place the index on
a temporary location (ie: a tmpfs when
low on diskspace or at ENOSPC)
- don't set index_updated flag when on
a temporary location
- exit the process when hitting an error
in _flush(). This is normally caused by
running into ENOSPC during a session, and
is unrecoverable. dbus will respawn the
process, the new spawn has a chance at
cleanup, moving things to tmpfs.
Signed-off-by: Martin Langhoff mar...@laptop.org
---
This is v2 of the patch, making sure that later sessions
rebuild the on-disk index.
src/carquinyol/indexstore.py | 57 +-
1 file changed, 45 insertions(+), 12 deletions(-)
diff --git a/src/carquinyol/indexstore.py b/src/carquinyol/indexstore.py
index dc721e7..0bbd93f 100644
--- a/src/carquinyol/indexstore.py
+++ b/src/carquinyol/indexstore.py
@@ -235,12 +235,32 @@ class IndexStore(object):
self._database = None
self._flush_timeout = None
self._pending_writes = 0
-self._index_updated_path = os.path.join(
-layoutmanager.get_instance().get_root_path(), 'index_updated')
-
-def open_index(self):
-index_path = layoutmanager.get_instance().get_index_path()
-self._database = WritableDatabase(index_path, xapian.DB_CREATE_OR_OPEN)
+root_path=layoutmanager.get_instance().get_root_path()
+self._index_updated_path = os.path.join(root_path,
+'index_updated')
+self._std_index_path = layoutmanager.get_instance().get_index_path()
+ self._index_path = self._std_index_path
+
+def open_index(self, temp_path=False):
+# callers to open_index must be able to
+# handle an exception -- usually caused by
+# IO errors such as ENOSPC and retry putting
+# the index on a temp_path
+if temp_path:
+try:
+# mark the on-disk index stale
+self._set_index_updated(False)
+except:
+pass
+self._index_path = temp_path
+else:
+ self._index_path = self._std_index_path
+try:
+ self._database = WritableDatabase(self._index_path,
+ xapian.DB_CREATE_OR_OPEN)
+except Exception as e:
+ logging.error('Exception opening database')
+ raise
def close_index(self):
Close index database if it is open.
@@ -248,14 +268,18 @@ class IndexStore(object):
return
self._flush(True)
-self._database = None
+try:
+# does Xapian write in its destructors?
+self._database = None
+except Exception as e:
+logging.error('Exception tearing down database')
+raise
def remove_index(self):
-index_path = layoutmanager.get_instance().get_index_path()
-if not os.path.exists(index_path):
+if not os.path.exists(self._index_path):
return
-for f in os.listdir(index_path):
-os.remove(os.path.join(index_path, f))
+for f in os.listdir(self._index_path):
+os.remove(os.path.join(self._index_path, f))
def contains(self, uid):
postings = self._database.postlist(_PREFIX_FULL_VALUE + \
@@ -347,6 +371,9 @@ class IndexStore(object):
index_updated = property(get_index_updated)
def _set_index_updated(self, index_updated):
+if self._std_index_path != self._index_path:
+ # operating from tmpfs
+ return True
if index_updated != self.index_updated:
if index_updated:
index_updated_file = open(self._index_updated_path, 'w')
@@ -374,7 +401,13 @@ class IndexStore(object):
self._pending_writes += 1
if force or self._pending_writes _FLUSH_THRESHOLD:
-self._database.flush()
+try:
+self._database.flush()
+except Exception, e:
+logging.exception(e)
+logging.error(Exception during database.flush())
+# bail out to trigger a reindex
+sys.exit(1)
self._pending_writes = 0
self._set_index_updated(True)
else:
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Thu, Sep 20, 2012 at 11:55 PM, Martin Langhoff mar...@laptop.org wrote: In conditions of very low disk space available, or ENOSPC, the datastore misbehaves grossly: Reviews are boring. Testing is more fun! rpms for ARM at http://dev.laptop.org/~martin/ds-enospc/ How to test: - Set Sugar to verbose debugging - Make sure sugar logs are not on /, add a file in /etc/statetab.d containing /home/olpc/.sugar/default/logs - use dd if=/dev/zero of=/bigfile bs=1M count=numberofmegs to fill up your disk I usually create one large file to put the system where it has little disk space remaining, then add/remove smaller files to tighten the free space further, or to push all the way to ENOSPC. This is faster than recreating a very large file everytime. The datastore.org will be telling you whether we are using tmpfs for the index, when a rebuild is triggered, ENOSPC is hit, etc. It is also interesting to monitor the presence of two clean markers in the datastore dir, ds_clean and index_updated. You can use inotifywatch -e create -e delete /home/olpc/.sugar/default/datastore cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 3/7] metadatastore: store/change files on disk defensively #2317
On Thu, Sep 20, 2012 at 11:55 PM, Martin Langhoff mar...@laptop.org wrote:
-f = open(os.path.join(metadata_path, key), 'w')
-try:
-if isinstance(value, unicode):
-value = value.encode('utf-8')
-elif not isinstance(value, basestring):
-value = str(value)
-f.write(value)
-finally:
-f.close()
BTW, I simplified the structure of
f = open(path, 'w')
try:
f.write(value)
finally:
f.close()
because in practice, f.write() writes into a buffer, so it never
fails. It is open() and close() that throw exceptions. In fact, at
ENOSPC, close() was failing.
And the use of finally is misguided. In my code, which wraps
open/write/close in a single try, if f.write() does throw an
exception, Python will close the FH when it goes out of scope.
Simpler, saner.
Python won't leak FHs on you. (IOWs, it takes some work to leak FHs on
any modern scripting language).
cheers,
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 7/7] datastore: handle low-disk and ENOSPC conditions gracefully
With this commit, the datastore comes up even on ENOSPC and very
tight disk conditions, and allows entry deletions even when at
ENOSPC.
- Be conservative
- ds or index flags are dirty - rebuild
- less than 5MB available - rebuild
- migrated or upgraded - rebuild
- Only skip an index rebuild if things look very clean
and good. Skipping the index rebuild is an optimization.
- If a straight index open fails, we attempt a rebuild.
- Updating a partial index is unreliable, always rebuild
- When rebuilding the index, the new index is placed on a tmpdir
(on Fedora and OLPC builds, this is a tmpfs). It is only moved
to disk if we are not in low-disk-space-available conditions.
Signed-off-by: Martin Langhoff mar...@laptop.org
---
This is v2 of the patch, making sure the move to internal disk
works in more cases, and that failure is recorded.
Thanks Sam for the report!
---
src/carquinyol/datastore.py | 89 +++
1 file changed, 73 insertions(+), 16 deletions(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 01d175e..a859dfe 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -23,6 +23,8 @@ import uuid
import time
import os
import shutil
+import subprocess
+import tempfile
import dbus
import dbus.service
@@ -44,6 +46,7 @@ DS_LOG_CHANNEL = 'org.laptop.sugar.DataStore'
DS_SERVICE = org.laptop.sugar.DataStore
DS_DBUS_INTERFACE = org.laptop.sugar.DataStore
DS_OBJECT_PATH = /org/laptop/sugar/DataStore
+MIN_INDEX_FREE_BYTES = 1024 * 1024 * 5
logger = logging.getLogger(DS_LOG_CHANNEL)
@@ -70,35 +73,53 @@ class DataStore(dbus.service.Object):
root_path = layoutmanager.get_instance().get_root_path()
self._cleanflag = os.path.join(root_path, 'ds_clean')
-if migrated:
+if initiated:
+logging.debug('Initiate datastore')
self._rebuild_index()
+self._index_store.flush()
+self._mark_clean()
return
-try:
-self._index_store.open_index()
-except Exception:
-logging.exception('Failed to open index, will rebuild')
+if migrated:
self._rebuild_index()
+self._mark_clean()
return
-if initiated:
-logging.debug('Initiate datastore')
-self._index_store.flush()
-elif not self._index_store.index_updated:
-logging.debug('Index is not up-to-date, will update')
-self._update_index()
+rebuild = False
+stat = os.statvfs(root_path)
+da = stat.f_bavail * stat.f_bsize
+
+if not self._index_store.index_updated:
+logging.warn('Index is not up-to-date')
+rebuild = True
elif not os.path.exists(self._cleanflag):
-logging.debug('DS state is not clean, will update')
-self._update_index()
-self._mark_clean()
+logging.warn('DS state is not clean')
+rebuild = True
+elif da MIN_INDEX_FREE_BYTES:
+logging.warn('Disk space tight for index')
+rebuild = True
+
+if rebuild:
+logging.warn('Trigger index rebuild')
+self._rebuild_index()
+ else:
+# fast path
+try:
+self._index_store.open_index()
+except:
+logging.exception('Failed to open index')
+# try...
+self._rebuild_index()
+self._mark_clean()
+return
def _mark_clean(self):
try:
f = open(self._cleanflag, 'w')
os.fsync(f.fileno())
f.close()
-except Exception:
+except:
logging.exception(Could not mark the datastore clean)
def _mark_dirty(self):
@@ -135,8 +156,44 @@ class DataStore(dbus.service.Object):
Remove and recreate index.
self._index_store.close_index()
self._index_store.remove_index()
-self._index_store.open_index()
+
+# rebuild the index in tmpfs to better handle ENOSPC
+temp_index_path = tempfile.mkdtemp(prefix='sugar-datastore-index-')
+logger.warn('Rebuilding index in %s' % temp_index_path)
+self._index_store.open_index(temp_path=temp_index_path)
self._update_index()
+self._index_store.close_index()
+
+on_disk=False
+
+# can we fit the index on disk? get disk usage in bytes...
+index_du = subprocess.check_output(['/usr/bin/du', '-bs',
+temp_index_path])
+index_du = int(index_du.split('\t')[0])
+# disk available, in bytes
+index_path = layoutmanager.get_instance().get_index_path()
+stat = os.statvfs(index_path)
+da = stat.f_bavail * stat.f_bsize
+if da (index_du * 1.2) and da MIN_INDEX_FREE_BYTES: # 20% room for
growth
Re: [Sugar-devel] [PATCH Browse] Error page SL #3500
On Fri, Sep 21, 2012 at 6:56 PM, Manuel Kaufmann humi...@gmail.com wrote: I'm not sure to understand what you are saying here. I'm not doing this each time the function is called. Here, I just showed what I did to get the base64 file encoded. I mean, I did this just once and I put the result inside the HTML file. Epiphany does the same: it has hardcoded the image inside the html. Is that a bad idea? Yeah, it's a wasteful approach. Probably doesn't matter much. You have the file on disk, correct? So you have the file path. You know that the webkit engine can read the file, too (it's on the same machine, in a readable directory, etc). So just put img src=file:///path/to/file . That saves a some of pointless CPU cycles, some RAM. This old man doesn't like inefficient code, even in small details. Mostly because that means that when you code some that is hot (ie: is exercised a lot, so is a potential bottleneck or memory hog if inefficient) you have the right instincts. I've spent many years fixing performance issues in large systems where people coded without much care to efficiency (or error handling). So there's a gnome in the back of my mind that kicks me if I write careless code... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 1/2] Add ds_clean flag to trigger index rebuilds #2095, #2317
This gives us more complete coverage of cases where ENOSPC or
other errors are hit when creating/updating datastore entries.
Without this patch, using the Journal on ENOSPC sometimes leads
to an empty Journal after restart. Datastore entries exist on
disk but are not in the index (Xapian DB).
With this patch, failure to complete create/update/delete forces
an index rebuild on restart.
---
src/carquinyol/datastore.py | 25 -
1 file changed, 24 insertions(+), 1 deletion(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index de79500..6ede341 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -77,12 +77,28 @@ class DataStore(dbus.service.Object):
self._rebuild_index()
return
+ds_root_path=layoutmanager.get_instance().get_root_path()
+self._cleanflag = os.path.join(ds_root_path, 'ds_clean')
+
if initiated:
logging.debug('Initiate datastore')
self._index_store.flush()
elif not self._index_store.index_updated:
logging.debug('Index is not up-to-date, will update')
self._update_index()
+elif not os.path.exists(self._cleanflag):
+logging.debug('DS state is not clean, will update')
+self._update_index()
+self._mark_clean()
+
+
+def _mark_clean(self):
+try:
+ f = open(self._cleanflag, 'w')
+ os.fsync(f.fileno())
+ f.close()
+except Exception:
+ logging.exception(Could not mark the datastore clean)
def _open_layout(self):
Open layout manager, check version of data store on disk and
@@ -178,6 +194,7 @@ class DataStore(dbus.service.Object):
self.Created(uid)
self._optimizer.optimize(uid)
logger.debug('created %s', uid)
+self._mark_clean()
async_cb(uid)
@dbus.service.method(DS_DBUS_INTERFACE,
@@ -190,6 +207,8 @@ class DataStore(dbus.service.Object):
uid = str(uuid.uuid4())
logging.debug('datastore.create %r', uid)
+os.remove(self._cleanflag)
+
if not props.get('timestamp', ''):
props['timestamp'] = int(time.time())
@@ -232,6 +251,7 @@ class DataStore(dbus.service.Object):
self.Updated(uid)
self._optimizer.optimize(uid)
logger.debug('updated %s', uid)
+self._mark_clean()
async_cb()
@dbus.service.method(DS_DBUS_INTERFACE,
@@ -243,6 +263,8 @@ class DataStore(dbus.service.Object):
async_cb, async_err_cb):
logging.debug('datastore.update %r', uid)
+os.remove(self._cleanflag)
+
if not props.get('timestamp', ''):
props['timestamp'] = int(time.time())
@@ -393,6 +415,7 @@ class DataStore(dbus.service.Object):
in_signature='s',
out_signature='')
def delete(self, uid):
+os.remove(self._cleanflag)
self._optimizer.remove(uid)
self._index_store.delete(uid)
@@ -404,7 +427,7 @@ class DataStore(dbus.service.Object):
self.Deleted(uid)
logger.debug('deleted %s', uid)
-
+self._mark_clean()
@dbus.service.signal(DS_DBUS_INTERFACE, signature=s)
def Deleted(self, uid):
pass
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 2/2] Remove invalid/corrupt on-disk entries #2317
When operating close to ENOSPC, we sometimes end up with
incomplete or invalid on-disk entries. So we prune these
during index rebuild.
---
src/carquinyol/datastore.py |7 +++
1 file changed, 7 insertions(+)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 6ede341..ee6bbb5 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -175,6 +175,13 @@ class DataStore(dbus.service.Object):
self._index_store.store(uid, props)
except Exception:
logging.exception('Error processing %r', uid)
+logging.warn('Will attempt to delete corrupt entry %r',
uid)
+try:
+# self.delete(uid) only works on well-formed entries
:-/
+entry_path =
layoutmanager.get_instance().get_entry_path(uid)
+os.removedirs(entry_path)
+except Exception:
+logging.exception('Error deleting corrupt entry %r',
uid)
if not uids:
self._index_store.flush()
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] Datastore vs ENOSPC patches, progress
Things I am working on - Add ds_clean flag to trigger index rebuilds - patch posted for review - Remove invalid corrupt disk entries - patch posted for review - When hitting ENOSPC, use a tmpfs for index database -- working on it... m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
In conditions of very low disk space available, or ENOSPC, the datastore misbehaves grossly: - sometimes corrupts the index, and won't rebuild it, hiding valid entries from the user - if you edit an entry metadata, it will corrupt the whole entry - attempts at deleting entries fail, so users cannot get out of ENOSPC This patchseries is an attempt at fixing these issues. The overall user experience is not perfect when we hit ENOSPC -- metadata edits are ignored, sometimes the datastore process goes away (so the Journal is not responsive until a new datastore process is spawned and ready). However, it behaves sanely: - testing this code I do not lose ds entries when editing them - I can remove entries correctly, even at ENOSPC, effectively freeing disk space - the index works even at ENOSPC, though it may take slower for the system to start in those cases where we do rebuild it - the index recovers gracefully on restart when there is free disk space Martin Langhoff (7): Add ds_clean flag to trigger index rebuilds #2095, #2317 Remove invalid/corrupt on-disk entries #2317 metadatastore: store/change files on disk defensively #2317 metadatareader: ignore .hidden files indexstore: exit on _flush() errors, work on tmpdir datastore: make delete() more reliable, log properly datastore: handle low-disk and ENOSPC conditions gracefully src/carquinyol/datastore.py | 142 +-- src/carquinyol/indexstore.py| 52 ++ src/carquinyol/metadatareader.c |7 +- src/carquinyol/metadatastore.py | 51 +- 4 files changed, 197 insertions(+), 55 deletions(-) -- 1.7.10.4 ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 1/7] Add ds_clean flag to trigger index rebuilds #2095, #2317
This gives us more complete coverage of cases where ENOSPC or
other errors are hit when creating/updating datastore entries.
Without this patch, using the Journal on ENOSPC sometimes leads
to an empty Journal after restart. Datastore entries exist on
disk but are not in the index (Xapian DB).
With this patch, failure to complete create/update/delete forces
an index rebuild on restart.
Signed-off-by: Martin Langhoff mar...@laptop.org
---
src/carquinyol/datastore.py | 31 ++-
1 file changed, 30 insertions(+), 1 deletion(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index de79500..1f098b4 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -66,6 +66,9 @@ class DataStore(dbus.service.Object):
self._index_store = IndexStore()
self._index_updating = False
+root_path = layoutmanager.get_instance().get_root_path()
+self._cleanflag = os.path.join(root_path, 'ds_clean')
+
if migrated:
self._rebuild_index()
return
@@ -83,6 +86,25 @@ class DataStore(dbus.service.Object):
elif not self._index_store.index_updated:
logging.debug('Index is not up-to-date, will update')
self._update_index()
+elif not os.path.exists(self._cleanflag):
+logging.debug('DS state is not clean, will update')
+self._update_index()
+self._mark_clean()
+
+
+def _mark_clean(self):
+try:
+ f = open(self._cleanflag, 'w')
+ os.fsync(f.fileno())
+ f.close()
+except Exception:
+ logging.exception(Could not mark the datastore clean)
+
+def _mark_dirty(self):
+try:
+os.remove(self._cleanflag)
+except:
+pass
def _open_layout(self):
Open layout manager, check version of data store on disk and
@@ -178,6 +200,7 @@ class DataStore(dbus.service.Object):
self.Created(uid)
self._optimizer.optimize(uid)
logger.debug('created %s', uid)
+self._mark_clean()
async_cb(uid)
@dbus.service.method(DS_DBUS_INTERFACE,
@@ -190,6 +213,8 @@ class DataStore(dbus.service.Object):
uid = str(uuid.uuid4())
logging.debug('datastore.create %r', uid)
+self._mark_dirty()
+
if not props.get('timestamp', ''):
props['timestamp'] = int(time.time())
@@ -232,6 +257,7 @@ class DataStore(dbus.service.Object):
self.Updated(uid)
self._optimizer.optimize(uid)
logger.debug('updated %s', uid)
+self._mark_clean()
async_cb()
@dbus.service.method(DS_DBUS_INTERFACE,
@@ -243,6 +269,8 @@ class DataStore(dbus.service.Object):
async_cb, async_err_cb):
logging.debug('datastore.update %r', uid)
+self._mark_dirty()
+
if not props.get('timestamp', ''):
props['timestamp'] = int(time.time())
@@ -393,6 +421,7 @@ class DataStore(dbus.service.Object):
in_signature='s',
out_signature='')
def delete(self, uid):
+self._mark_dirty()
self._optimizer.remove(uid)
self._index_store.delete(uid)
@@ -404,7 +433,7 @@ class DataStore(dbus.service.Object):
self.Deleted(uid)
logger.debug('deleted %s', uid)
-
+self._mark_clean()
@dbus.service.signal(DS_DBUS_INTERFACE, signature=s)
def Deleted(self, uid):
pass
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 2/7] Remove invalid/corrupt on-disk entries #2317
When operating close to ENOSPC, we sometimes end up with
incomplete or invalid on-disk entries. So we prune these
during index rebuild.
Signed-off-by: Martin Langhoff mar...@laptop.org
---
src/carquinyol/datastore.py |8
1 file changed, 8 insertions(+)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 1f098b4..33c658d 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -22,6 +22,7 @@ import logging
import uuid
import time
import os
+import shutil
import dbus
import dbus.service
@@ -181,6 +182,13 @@ class DataStore(dbus.service.Object):
self._index_store.store(uid, props)
except Exception:
logging.exception('Error processing %r', uid)
+logging.warn('Will attempt to delete corrupt entry %r',
uid)
+try:
+# self.delete(uid) only works on well-formed entries
:-/
+entry_path =
layoutmanager.get_instance().get_entry_path(uid)
+shutil.rmtree(entry_path)
+except Exception:
+logging.exception('Error deleting corrupt entry %r',
uid)
if not uids:
self._index_store.flush()
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 4/7] metadatareader: ignore .hidden files
Ignore any file with a filename starting with a '.' -- metadatastore
writes its tempfiles prefixed with a '.' .
Signed-off-by: Martin Langhoff mar...@laptop.org
---
Note: this one needs a bit of testing.
---
src/carquinyol/metadatareader.c |7 ++-
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/src/carquinyol/metadatareader.c b/src/carquinyol/metadatareader.c
index d053a22..ee546f3 100644
--- a/src/carquinyol/metadatareader.c
+++ b/src/carquinyol/metadatareader.c
@@ -161,11 +161,8 @@ static PyObject *read_all_properties (const char
*metadata_path) {
dir_entry = readdir (dir_stream);
while (dir_entry != NULL) {
-// Skip . and ..
-if (dir_entry-d_name[0] == '.'
-(strlen (dir_entry-d_name) == 1 ||
-(dir_entry-d_name[1] == '.'
-strlen (dir_entry-d_name) == 2)))
+// Skip '.', '..', and any .hidden file
+if (dir_entry-d_name[0] == '.')
goto next_property;
if (add_property (metadata_path, dir_entry-d_name, dict, 1) == 0)
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 3/7] metadatastore: store/change files on disk defensively #2317
- only delete metadata files for keys that are being removed
- only write files when the data changes
- write/replace metadata files atomically, to avoid corrupting
existing data in case of an error
With this patch, we no longer corrupt metadata when trying
to edit/update a ds entry with the system hitting ENOSPC.
Signed-off-by: Martin Langhoff mar...@laptop.org
---
src/carquinyol/metadatastore.py | 51 ++-
1 file changed, 34 insertions(+), 17 deletions(-)
diff --git a/src/carquinyol/metadatastore.py b/src/carquinyol/metadatastore.py
index 5967017..52cc10f 100644
--- a/src/carquinyol/metadatastore.py
+++ b/src/carquinyol/metadatastore.py
@@ -14,27 +14,46 @@ class MetadataStore(object):
if not os.path.exists(metadata_path):
os.makedirs(metadata_path)
else:
+received_keys = metadata.keys()
for key in os.listdir(metadata_path):
-if key not in _INTERNAL_KEYS:
+if key not in _INTERNAL_KEYS and key not in received_keys:
os.remove(os.path.join(metadata_path, key))
metadata['uid'] = uid
for key, value in metadata.items():
+self._set_property(uid, key, value, md_path=metadata_path)
-# Hack to support activities that still pass properties named as
+def _set_property(self, uid, key, value, md_path=False):
+if not md_path:
+md_path = layoutmanager.get_instance().get_metadata_path(uid)
+# Hack to support activities that still pass properties named as
# for example title:text.
-if ':' in key:
-key = key.split(':', 1)[0]
-
-f = open(os.path.join(metadata_path, key), 'w')
-try:
-if isinstance(value, unicode):
-value = value.encode('utf-8')
-elif not isinstance(value, basestring):
-value = str(value)
-f.write(value)
-finally:
-f.close()
+if ':' in key:
+key = key.split(':', 1)[0]
+
+changed = True
+fpath = os.path.join(md_path, key)
+tpath = os.path.join(md_path, '.' + key)
+# FIXME: this codepath handles raw image data
+# str() is 8-bit clean right now, but
+# this won't last. We will need more explicit
+# handling of strings, int/floats vs raw data
+if isinstance(value, unicode):
+value = value.encode('utf-8')
+elif not isinstance(value, basestring):
+value = str(value)
+
+# avoid pointless writes; replace atomically
+if os.path.exists(fpath):
+stored_val = open(fpath, 'r').read()
+
+if stored_val == value:
+changed = False
+if changed:
+f = open(tpath, 'w')
+f.write(value)
+f.close()
+os.rename(tpath, fpath)
def retrieve(self, uid, properties=None):
metadata_path = layoutmanager.get_instance().get_metadata_path(uid)
@@ -55,6 +74,4 @@ class MetadataStore(object):
return None
def set_property(self, uid, key, value):
-metadata_path = layoutmanager.get_instance().get_metadata_path(uid)
-property_path = os.path.join(metadata_path, key)
-open(property_path, 'w').write(value)
+self._set_property(uid, key, value)
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 5/7] indexstore: exit on _flush() errors, work on tmpdir
Preparations for better index management.
- open_index() now can place the index on
a temporary location (ie: a tmpfs when
low on diskspace or at ENOSPC)
- don't set index_updated flag when on
a temporary location
- exit the process when hitting an error
in _flush(). This is normally caused by
running into ENOSPC during a session, and
is unrecoverable. dbus will respawn the
process, the new spawn has a chance at
cleanup, moving things to tmpfs.
Signed-off-by: Martin Langhoff mar...@laptop.org
---
src/carquinyol/indexstore.py | 52 --
1 file changed, 40 insertions(+), 12 deletions(-)
diff --git a/src/carquinyol/indexstore.py b/src/carquinyol/indexstore.py
index dc721e7..a702014 100644
--- a/src/carquinyol/indexstore.py
+++ b/src/carquinyol/indexstore.py
@@ -235,12 +235,27 @@ class IndexStore(object):
self._database = None
self._flush_timeout = None
self._pending_writes = 0
-self._index_updated_path = os.path.join(
-layoutmanager.get_instance().get_root_path(), 'index_updated')
-
-def open_index(self):
-index_path = layoutmanager.get_instance().get_index_path()
-self._database = WritableDatabase(index_path, xapian.DB_CREATE_OR_OPEN)
+root_path=layoutmanager.get_instance().get_root_path()
+self._index_updated_path = os.path.join(root_path,
+'index_updated')
+self._std_index_path = layoutmanager.get_instance().get_index_path()
+ self._index_path = self._std_index_path
+
+def open_index(self, temp_path=False):
+# callers to open_index must be able to
+# handle an exception -- usually caused by
+# IO errors such as ENOSPC and retry putting
+# the index on a temp_path
+if temp_path:
+self._index_path = temp_path
+else:
+ self._index_path = self._std_index_path
+try:
+ self._database = WritableDatabase(self._index_path,
+ xapian.DB_CREATE_OR_OPEN)
+except Exception as e:
+ logging.error('Exception opening database')
+ raise
def close_index(self):
Close index database if it is open.
@@ -248,14 +263,18 @@ class IndexStore(object):
return
self._flush(True)
-self._database = None
+try:
+# does Xapian write in its destructors?
+self._database = None
+except Exception as e:
+logging.error('Exception tearing down database')
+raise
def remove_index(self):
-index_path = layoutmanager.get_instance().get_index_path()
-if not os.path.exists(index_path):
+if not os.path.exists(self._index_path):
return
-for f in os.listdir(index_path):
-os.remove(os.path.join(index_path, f))
+for f in os.listdir(self._index_path):
+os.remove(os.path.join(self._index_path, f))
def contains(self, uid):
postings = self._database.postlist(_PREFIX_FULL_VALUE + \
@@ -347,6 +366,9 @@ class IndexStore(object):
index_updated = property(get_index_updated)
def _set_index_updated(self, index_updated):
+if self._std_index_path != self._index_path:
+ # operating from tmpfs
+ return True
if index_updated != self.index_updated:
if index_updated:
index_updated_file = open(self._index_updated_path, 'w')
@@ -374,7 +396,13 @@ class IndexStore(object):
self._pending_writes += 1
if force or self._pending_writes _FLUSH_THRESHOLD:
-self._database.flush()
+try:
+self._database.flush()
+except Exception, e:
+logging.exception(e)
+logging.error(Exception during database.flush())
+# bail out to trigger a reindex
+sys.exit(1)
self._pending_writes = 0
self._set_index_updated(True)
else:
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 6/7] datastore: make delete() more reliable, log properly
- use try/except to catch and log errors in datastore.log
(instead of echoing them to callers over dbus)
- use rmtree() to more reliably remove the entry
- get_entry_path() before we delete all the related
metadata.
Signed-off-by: Martin Langhoff mar...@laptop.org
---
src/carquinyol/datastore.py | 24
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 33c658d..01d175e 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -430,14 +430,22 @@ class DataStore(dbus.service.Object):
out_signature='')
def delete(self, uid):
self._mark_dirty()
-self._optimizer.remove(uid)
-
-self._index_store.delete(uid)
-self._file_store.delete(uid)
-self._metadata_store.delete(uid)
-
-entry_path = layoutmanager.get_instance().get_entry_path(uid)
-os.removedirs(entry_path)
+try:
+entry_path = layoutmanager.get_instance().get_entry_path(uid)
+self._optimizer.remove(uid)
+self._index_store.delete(uid)
+self._file_store.delete(uid)
+self._metadata_store.delete(uid)
+# remove the dirtree
+shutil.rmtree(entry_path)
+try:
+# will remove the hashed dir if nothing else is there
+os.removedirs(os.path.dirname(entry_path))
+except:
+pass
+except:
+logger.exception('Exception deleting entry')
+raise
self.Deleted(uid)
logger.debug('deleted %s', uid)
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 7/7] datastore: handle low-disk and ENOSPC conditions gracefully
With this commit, the datastore comes up even on ENOSPC and very
tight disk conditions, and allows entry deletions even when at
ENOSPC.
- Be conservative
- ds or index flags are dirty - rebuild
- less than 5MB available - rebuild
- migrated or upgraded - rebuild
- Only skip an index rebuild if things look very clean
and good. Skipping the index rebuild is an optimization.
- If a straight index open fails, we attempt a rebuild.
- Updating a partial index is unreliable, always rebuild
- When rebuilding the index, the new index is placed on a tmpdir
(on Fedora and OLPC builds, this is a tmpfs). It is only moved
to disk if we are not in low-disk-space-available conditions.
Signed-off-by: Martin Langhoff mar...@laptop.org
---
src/carquinyol/datastore.py | 87 +++
1 file changed, 71 insertions(+), 16 deletions(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 01d175e..67bda06 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -23,6 +23,8 @@ import uuid
import time
import os
import shutil
+import subprocess
+import tempfile
import dbus
import dbus.service
@@ -44,6 +46,7 @@ DS_LOG_CHANNEL = 'org.laptop.sugar.DataStore'
DS_SERVICE = org.laptop.sugar.DataStore
DS_DBUS_INTERFACE = org.laptop.sugar.DataStore
DS_OBJECT_PATH = /org/laptop/sugar/DataStore
+MIN_INDEX_FREE_BYTES = 1024 * 1024 * 5
logger = logging.getLogger(DS_LOG_CHANNEL)
@@ -70,35 +73,53 @@ class DataStore(dbus.service.Object):
root_path = layoutmanager.get_instance().get_root_path()
self._cleanflag = os.path.join(root_path, 'ds_clean')
-if migrated:
+if initiated:
+logging.debug('Initiate datastore')
self._rebuild_index()
+self._index_store.flush()
+self._mark_clean()
return
-try:
-self._index_store.open_index()
-except Exception:
-logging.exception('Failed to open index, will rebuild')
+if migrated:
self._rebuild_index()
+self._mark_clean()
return
-if initiated:
-logging.debug('Initiate datastore')
-self._index_store.flush()
-elif not self._index_store.index_updated:
-logging.debug('Index is not up-to-date, will update')
-self._update_index()
+rebuild = False
+stat = os.statvfs(root_path)
+da = stat.f_bavail * stat.f_bsize
+
+if not self._index_store.index_updated:
+logging.warn('Index is not up-to-date')
+rebuild = True
elif not os.path.exists(self._cleanflag):
-logging.debug('DS state is not clean, will update')
-self._update_index()
-self._mark_clean()
+logging.warn('DS state is not clean')
+rebuild = True
+elif da MIN_INDEX_FREE_BYTES:
+logging.warn('Disk space tight for index')
+rebuild = True
+
+if rebuild:
+logging.warn('Trigger index rebuild')
+self._rebuild_index()
+ else:
+# fast path
+try:
+self._index_store.open_index()
+except:
+logging.exception('Failed to open index')
+# try...
+self._rebuild_index()
+self._mark_clean()
+return
def _mark_clean(self):
try:
f = open(self._cleanflag, 'w')
os.fsync(f.fileno())
f.close()
-except Exception:
+except:
logging.exception(Could not mark the datastore clean)
def _mark_dirty(self):
@@ -135,8 +156,42 @@ class DataStore(dbus.service.Object):
Remove and recreate index.
self._index_store.close_index()
self._index_store.remove_index()
-self._index_store.open_index()
+
+# rebuild the index in tmpfs to better handle ENOSPC
+temp_index_path = tempfile.mkdtemp(prefix='sugar-datastore-index-')
+logger.warn('Rebuilding index in %s' % temp_index_path)
+self._index_store.open_index(temp_path=temp_index_path)
self._update_index()
+self._index_store.close_index()
+
+on_disk=False
+
+# can we fit the index on disk? get disk usage in bytes...
+index_du = subprocess.check_output(['/usr/bin/du', '-bs',
+temp_index_path])
+index_du = int(index_du.split('\t')[0])
+# disk available, in bytes
+index_path = layoutmanager.get_instance().get_index_path()
+stat = os.statvfs(index_path)
+da = stat.f_bavail * stat.f_bsize
+if da (index_du * 1.2) and da MIN_INDEX_FREE_BYTES: # 20% room for
growth
+logger.warn('Attempting to move tempfs index to disk')
+# move to internal disk
+try
Re: [Sugar-devel] Adventures in the land of ENOSPC
On Sat, Sep 15, 2012 at 8:03 AM, Sascha Silbe si...@activitycentral.com wrote: Martin Langhoff martin.langh...@gmail.com writes: Manuel Kaufmann has been looking at SL#394, and looking at the bug report, it struck me that it was reported backwards. I would have written: I filled up my disk and it knocked the system out, Sugar would not start again, etc. Oh, btw, it was with Browse. I agree. Every single mishandling of ENOSPC is a bug in the affected piece of software and should be fixed. However, there's a lot of broken In our case, 99% of the time is the datastore. And when it borks, it messes up its indexes so it _has_ the files and the metadata, but will show you an empty listing for the Journal. Similarly, the data store should take care to leave some free The datastore needs to handle ENOSPC correctly. That is much much deeper than what we can do activity-side. Sounds like SL#2317 [1], as Gonzalo already pointed out. Indeed, and the problem is that index_updated remains on disk all the time. I am causing ENOSPC artificially, and creating or updating journal entries against ENOSPC and... index_updated is never removed. BTW, to test this I am putting sugar's own logs on tmpfs, which you can do with a file like this: # cat /etc/statetab.d/olpc.debug /home/olpc/.sugar/default/logs cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] Debugging sugar-datastore (and similar dbus-connected services)
I'm debugging odd situations w Sugar datastore, and I wonder whether there is any tricks to debugging python programs that are run under dbus. I can see the sugar-datastore PID growing steadily while I test, which means that it's dying and respawning plenty, likely from unhandled exceptions... thanks, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Debugging sugar-datastore (and similar dbus-connected services)
On Wed, Sep 19, 2012 at 4:15 PM, Martin Langhoff martin.langh...@gmail.com wrote: I'm debugging odd situations w Sugar datastore, and I wonder whether there is any tricks to debugging python programs that are run under dbus. I can see the sugar-datastore PID growing steadily while I test, which means that it's dying and respawning plenty, likely from unhandled exceptions... Adding to /usr/bin/datastore-service # debugging logerr=open(os.path.join(log_dir, 'datastore.err'), 'w') sys.stderr=logerr does not help in the least. The file is created, but nothing is ever written there. Any hints? m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Debugging sugar-datastore (and similar dbus-connected services)
On Wed, Sep 19, 2012 at 4:54 PM, Gonzalo Odiard gonz...@laptop.org wrote: You already have the line export SUGAR_LOGGER_LEVEL=debug uncommented in .sugar/debug, right? Yes. And with that, datastore.log is very chatty, which is good. But when the datastore code hits an unhandled exception, the exception does not get logged. DBus restarts the process transparently, but that isn't so useful :-/ In terms of progress, I have these tracks underway, - Add a ds_clean flag file, rm it on the start of every create/update/delete op, create it on completion. Complements index_updated, and between the two we get good coverage. Missing either one, we get an index rebuild. This is done, and working well. - During index rebuild, delete incomplete entries. Not working well yet. With the two above, we can operate with the system at ENOSPC, and essentially we don't lose existing user data. New files fail to save, and metadata changes fail to get stored (so your edit, ie: a rename, is reverted in the journal, visibly). But we don't lose the existing Journal entries -- so behaviour is _almost_ civilized now :-) Behind the scenes, however, the datastore is dying and respawning. Which complicates things because what we are missing is to be able to delete entries while we are at ENOSPC. A delete is pretty straightforward, because we are removing files, and that succeeds even on ENOSPC. Except that we need to be able to maintain the xapian indexes. My plan right now is to catch the Xapian-is-hitting-ENOSPC exception, and trigger a Xapian index re-creation, putting the Xapian files in /tmp (which is a tmpfs in our builds and in Fedora upstream too). We'll make sure that the index_updated file on disk is removed so after a reboot the index is recreated once more. With Xapian index on /tmp, deleting Journal entries should just work. All dbus sockets are on tmpfs. However, to do all of the above with some sense that it's working correctly... yeah, I need to see the exceptions I hit. Right now I am guessing what could be going wrong in this black box...? which isn't a productive mode of life. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Error page SL #3500
On Wed, Sep 19, 2012 at 6:54 AM, Manuel Kaufmann humi...@gmail.com wrote:
On Tue, Sep 18, 2012 at 10:58 PM, James Cameron qu...@laptop.org wrote:
Neat. What method did you use to convert the image?
Python... It is my best friend :)
import base64
base64.b64encode(open('browse-logo.png', 'r').read())
You have the file on-disk. It is inefficient to base64-encode it. Just
say img src=file://path-to-image. Keep in mind location changes
slighly between rpm-installed activities and those in ~/Activities .
cheers,
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Not enough space adventure
On Tue, Sep 18, 2012 at 8:42 AM, Manuel Kaufmann humi...@gmail.com wrote: We where discussing about this last week[1] and we found the root issue of this problem: Sugar is not handling ENOSPC error. This could cause some problems at boot time when the XO is restarted, but as we discussed[2], Linux has made some improvement on this side and it seems that it recovers without problem (we need more testing here, I think). Thanks for diving into this! So, there are different problems to manage here: 1. What are we going to do when ENOSPC is reached? I did some work lastweek, and plan to hack on it tomorrow. 2. How are we going to avoid ENOSCP? ... - remove the annoying check of free space I hope that's not completely removed (I see Gonzalo's email...). It is not very effective preventing you from using activities, but it does prompt users to do something (remove stuff?). BTW, 50MB may be too high, some units with 2GB storage, installing modern builds, end up with 150MB free total :-/ - create a signal inside sugar-toolkit-gtk3 datastore that is emitted when free space is behind sugar3.datastore.datastore.SPACE_THRESHOLD. This check is done every time a model is updated or created by the datastore. A model? Maybe my lingo is a bit stale... a datastore entry you mean? - this signal can be connected from every activity that wants to handle this situation. For example in Browse we are cancelling the download in progress when we get that signal So there is a bit of confusion here... - When we start the download, if we get Content-Length, we compare it with disk space available, or with disk space available minus datastore.SPACE_THRESHOLD? - Related: can we ask the datastore to tell us its SPACE_THRESHOLD, so we don't hardcode it? Ah, I see you have a helper function. Personally, I would prefer to preserve a relatively high threshold (lower than 50MB, but say, 10MB) for the _warnings_. But still allow a download to complete. I would only cancel a download in Browse if will leave us with 1MB disk space. To put it in other words, we are investing too much effort in... warnings. And we are mixing a high threshold that is good for a warning, with something that disrupts an operation (cancel download). You may be downloading an activity that allows you to delete stuff :-) Finally, we seem to be missing the tmpfile cleanup related to incomplete downloads. That can invisibly eat a ton of disk space in a way that the end user cannot cleanup... cheers, m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Wed, Sep 12, 2012 at 7:50 AM, Manuel Kaufmann humi...@gmail.com wrote: I'm attaching an example that downloads a file with WebKit. I tried it setting up a tmpfs with 1Mb as you suggested and aftert that, I ran this script. WebKit tells us about the insufficient space on the disk by raising the error signal. I catched it and I printed the arguments of the signal: That's very good handling on webkit's side. And does it remove the file? IOWs, when you get the signal, what do you see in the tmpfs? Is there a file filling it, or has the file been removed? cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Wed, Sep 12, 2012 at 8:20 AM, Manuel Kaufmann humi...@gmail.com wrote: OK. I was taking a look at the datastore source code and I understood that the file is copied, using the traditional way and when the copy finishes the source file is unlinked. Ouch! I haven't reviewed the code (have to run some personal errands today, can't dig into it as I'd like), but if you are correct, there are two worries - Downloading a 100MB file needs 200MB+ free. - All that passing around the file to update the percentage bar is causing pointless file copies -- if you are downloading a large file, it is pretty screwed. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Wed, Sep 12, 2012 at 8:24 AM, Manuel Kaufmann humi...@gmail.com wrote: On Wed, Sep 12, 2012 at 9:21 AM, Martin Langhoff martin.langh...@gmail.com wrote: That's very good handling on webkit's side. And does it remove the file? IOWs, when you get the signal, what do you see in the tmpfs? Is there a file filling it, or has the file been removed? Oh, sorry. I forgot to mention this. The file is still there with the size that webkit could get (in my case 904kb). Webkit didn't remove it. Hmmm. Ok, so it's leaving it up to the download manager code (our browse code) to handle the remains. AIUI, Webkit will be using the filename (or filename template) we passed it, so we know enough to find the file and remove it in that case. We should :-) cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Tue, Sep 11, 2012 at 9:01 AM, Manuel Kaufmann humi...@gmail.com wrote: If we are going to use this approach, I have to change the logic about where I should check this. +200. I hadn't realized this. The main two things you absolutely need are - check at the start of the download that it'll fit (with X room to spare) - make damn sure that we handle ENOSPC if the disk does fill up Checking in the progress bar... it's optional. Up to you if you want to check there _as well_. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Tue, Sep 11, 2012 at 4:32 PM, Manuel Kaufmann humi...@gmail.com wrote: I didn't find an exception raised by Sugar when this happens. The check is done[1] when a new Model is created[2] or updated[3] inside the Journal, and if there are less than 50Mb a ModelAlert is shown but no exception is raised. Maybe we can write the portion of code here to raise that exception before show the ModelAlert and if it's not managed we can call the ModelAlert. [1] sugar/src/jarabe/journal/journalactivity.py L336 [2] sugar/src/jarabe/journal/journalactivity.py L274 [3] sugar/src/jarabe/journal/journalactivity.py L265 Forget about check_available_space() and pretty messages to the user. The real problem is hitting out-of-disk space. Can you run a test that bypasses this code so it really gets to ENOSPC? How does ENOSPC appear in Python? Py documentation is pretty incomplete: http://docs.python.org/library/exceptions.html#exceptions.IOError This says IOError 28 http://stackoverflow.com/questions/6998083/python-causing-ioerror-errno-28-no-space-left-on-device-results-32766-h I guess you'll have to try yourself. Try mkdir /tmp/foo sudo mount -t tmpfs -o size=1M none /tmp/foo Now write there to a file in /tmp/foo to see what happens, exactly. The file is downloaded in a temporary directory and then copied to the Journal: temp_path = os.path.join(activity.get_activity_root(), 'instance') fd, self._dest_path = tempfile.mkstemp(dir=temp_path, suffix=download.get_suggested_filename(), prefix='tmp') I guess activity.get_activity_root() is in the main FS, not in a tmpfs. Luckily. That is something like (I guess): ~/.sugar/default/org.laptop.WebActivity/instance Ah, cool. So it's in the same FS. When the download finishes: (...) self.dl_jobject = datastore.create() (...) datastore.write(self.dl_jobject, transfer_ownership=True, reply_handler=self.__internal_save_cb, error_handler=self.__internal_error_cb, timeout=360) (...) Oh, that expects a callback. If there is an exception, maybe we don't get it on this side? :-( def __progress_change_cb(self, download, something): progress = self._download.get_progress() self.dl_jobject.metadata['progress'] = str(int(progress * 100)) datastore.write(self.dl_jobject) so, we are updating a model inside the datastore... If we had the ENOSPC exception we could manage it there. I insist this goes in a different ticket and a different patch. Being callback, its is probably going over dbus, so, ummm. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Tue, Sep 11, 2012 at 9:07 PM, Manuel Kaufmann humi...@gmail.com wrote:
This is what I got:
[olpc@xo-07-30-50 ~]$ mkdir /tmp/foo
[olpc@xo-07-30-50 ~]$ sudo mount -t tmpfs -o size=1M none /tmp/foo
[olpc@xo-07-30-50 ~]$ python
Python 2.7.3 (default, Apr 30 2012, 23:07:00)
[GCC 4.7.0 20120416 (Red Hat 4.7.0-2)] on linux2
Type help, copyright, credits or license for more information.
f = open('/tmp/foo/cl.txt', 'w')
f.write('w' * 200 ** 2)
(...)
f.write('w' * 2000)
Traceback (most recent call last):
File stdin, line 1, in module
IOError: [Errno 28] No space left on device
I guess that's what we'll want to grep for in the sugar datastore code ;-)
Hmmm, reading the code in download.py. AIUI Browse lets webkit run the
downloading (I cannot see any write() calls around). When we say
self._download.set_destination_uri('file://' + self._dest_path)
self._download.start()
we pass the ball to webkit. So in this case, we most of the writing is
being done in webkit. I do wonder what happens if webkit gets ENOSPC.
Does it manage to report it to Browse? Does it cleanup the partial
tmpfile? If not, should we file a bug upstream?
Also, IIRC, when you give a file to the datastore, the datastore does
not make a copy. It should make a hardlink to it. I hope the current
datastore is still sane in that regard. Still, if you are extremely
tight for space the datastore saves metadata, which could hit ENOSPC.
So I see 3 places where we could run out of space
- Empty tmpfile setup, extremely unlikely; you may want to implement
a try/except for consistency and elegance and good karma...
- During the webkit download -- most probable stage where you'll hit
it. How does Webkit behave? Does Browse need to do anything?
- During datastore.write() -- I truly believe that the
datastore.write() call doesn't literally make a copy of the file --
should just create/update the hardlink and associated metadata
(including progress bar). It is not very likely that we'll run out of
space during this call, but it should be handled, I believe.
Does that make sense? TBH, my datastore knowledge is extremely rusty.
It has been a long time...
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Mon, Sep 10, 2012 at 5:33 PM, Manuel Kaufmann humi...@gmail.com wrote: If there are less than 50Mb free on the Hard Disk the downloading process is canceled and this is informed to the user via an Alert. Perhaps it is a stupid question but... in many HTTP requests you get the size of the transfer in the HTTP headers, in the form of the Content-Length header. This is specially true with downloads of large files (less true for dynamic content like PHP-generated pages). +# http://stackoverflow.com/questions/787776/find-free-disk-space-in-python-on-os-x +def free_space(path): Do you really need that comment? Doing s.f_bavail * s.f_frsize is just common sense. +global k +s = os.statvfs(path) +return (s.f_bavail * s.f_frsize) / 1024 / 1024 Points for using f_bavail instead of bfree :-) Don't divide it yet. Return bytes -- only convert to MB if you are displaying to users. You are going to need bytes to compare with content-length which is in bytes too. progress = self._download.get_progress() -self.dl_jobject.metadata['progress'] = str(int(progress * 100)) Here, get the content-length... and compare with the free bytes (plus some padding, say, 15%). Maybe we still want to preserve a minimum disk space. For example: don't let a download put you below 50MB or 10MB free -- in that case do the multiplication in a constant definition. MIN_DISKFREE_AFTER_DOWNLOAD = 50 * 1024 * 1024 If the request does not have content-length, it is sane to assume it will be small (dynamically generated content) so that padding you preserved is very likely to fit it. And then, you need the most clear step: if the write fails, and the error / exception tells you it's ENOSPC, you delete the object. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Mon, Sep 10, 2012 at 5:49 PM, Martin Langhoff martin.langh...@gmail.com wrote: Perhaps it is a stupid question but... in many HTTP requests you get s/question/suggestion/ m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [DESIGN] set zoom to initial value in Browse
On Mon, Sep 10, 2012 at 10:07 AM, Gonzalo Odiard gonz...@laptop.org wrote: Actual size is used in Read too. http://imgs.xkcd.com/store/imgs/actual_size_200.png m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Mon, Sep 10, 2012 at 6:02 PM, Manuel Kaufmann humi...@gmail.com wrote: On Mon, Sep 10, 2012 at 9:49 PM, Martin Langhoff martin.langh...@gmail.com wrote: Perhaps it is a stupid question but... in many HTTP requests you get the size of the transfer in the HTTP headers, in the form of the Content-Length header. I didn't follow this approach because of this comment: http://bugs.sugarlabs.org/ticket/394#comment:5 We address _that_ possibiliy (concurrent processes using up that disk space) with the error handling I was proposing later in my comments. So, putting that aside for a moment why a fixed guess is better than the actual content-length? It does make sense to declare a minimum buffer, but you have good reliably info in content-length, use it! And then, you need the most clear step: if the write fails, and the error / exception tells you it's ENOSPC, you delete the object. Will Sugar recover immediately after getting ENOSPC and deleting the object? Dunno. I hope so. Force the bug, see what you get. If things are reasonably ok, you'll get an exception in the Sugar code, hopefully the file is removed automatically from the Journal. So if all is good, you are expected to handle the exception in Browse. In terms of the Journal/datastore, you'd get the same situation copying from a USB stick -- try a file that is too large to fit in the free space you have... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Mon, Sep 10, 2012 at 6:39 PM, James Cameron qu...@laptop.org wrote: Some users pay for data by volume. I don't think that cancelling a download is the right approach. It should be paused, giving the user a chance to clean up. Rather complex -- I haven't seen any webbrowser correctly implement that correctly. Also, I agree with Martin that additional use of Content-Length headers can be made, to warn the user against trying a download larger than available space. Thanks I disagree with Bernie in #394 comment 5 ... because the Content-Length of all concurrent downloads can be used in a calculation. We agree in disagreeing with Bernie, but no, trying to have a download controller pooling download information is another big ball of complexity I would not want to get into. Specially since it is ineffective -- _any other program in the system may be pushing you to ENOSPC_. The kernel acts as the final FS controller. What we _must_ have is solid recovery from ENOSPC. I think datastore has it (that's worth a test :-) ) and if datastore has it, then Browse can handle the exception gracefully with a small amount of coding effort... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problems starting the XO
On Tue, Sep 4, 2012 at 3:38 PM, Ajay Garg a...@activitycentral.com wrote: May be this could be improved sometime in the future :) Fix the packaging, and don't use --force. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problems starting the XO
On Sat, Sep 1, 2012 at 4:25 PM, Ajay Garg a...@activitycentral.com wrote: Thanks Chris and Mikus for the replies. Hi Ajay, please ansert cjb's question. It has a good chance of leading to your answer. What deps are you ignoring? Why --force? Also -- /var/log/olpc-dm* logs are likely to have important info. Please post them. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Unicode strings in translations
On Thu, Aug 16, 2012 at 9:29 AM, Benjamin Berg ben...@sugarlabs.org wrote: Disclaimer: I have not read the entire discussion. and you've missed a bit :-) So, in Sugar we are mixing python unicode strings, and utf-8 encoded No. We have been _temporarily_ mixing Python unicode UTF-8 with ASCII strings. We used to import gtk, which had the side effect of telling Python to assume any string constant in the code to be unicode UTF-8 instead of ASCII. As part of the gtk3 transition, we discovered importing gtk3 doesn't have that side effect. But we want Python to assume string constants to be UTF-8, so should tell it so explicitly instead of relying on side-effects :-) -- look at the emails from dnarvaez and manuq. Manuq - btw, I agree with upstream that this is not a GTK3/PyGI bug, it is an improvement. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Unicode strings in translations
On Wed, Aug 15, 2012 at 9:20 AM, Manuel Kaufmann humi...@gmail.com wrote: Oh, it's OK. I agree with the result. Now, let's check what Python say if I use my default encoding (UTF8) for this simple task: len(camión) 7 CAREFUL HERE. You don't understand what is happening -- it is not as simple as you think it is. When you say len(camión), you are writing that from a terminal (Gnome's Terminal, Sugar Terminal, xterm) that is set to use utf-8. However, Python expects the sequence between characters to be straight ASCII (with a codepage). So your terminal IS sending to Python what looks like 7 chars -- definitely 7 bytes. However, there is an ASCII representation of camión that has 6 bytes, using the Latin-1 codepage. In fact, install an old Linux system, open an xterm or a VT, retry your example and you'll probably see that camión has 6 bytes. I agree we should all use Unicode, specifically UTF-8, everywhere. We should also make an effort to understand the mechanics of what is actually happening behind the scenes. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Unicode strings in translations
On Wed, Aug 15, 2012 at 9:48 AM, Daniel Narvaez dwnarv...@gmail.com wrote: I think the reason this worked in gtk2 activities is that importing gtk had this side effect (it's in the pango module really) /* set the default python encoding to utf-8 */ PyUnicode_SetDefaultEncoding(utf-8); Oh, can you do that? Excellent. Perhaps it's a good idea to include that in early-execution Sugar shell code, and in Sugar toolkit. Now I don't really like changing default encoding as a side effect but the simpler solution is probably to just reinstate that, either in Perhaps inelegant if you were note expecting it. However, I think it is a sane statement: all Python code in Sugar and activities is in utf-8. Any strings are utf-8,even if we forget to add the u prefix... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Unicode strings in translations
On Wed, Aug 15, 2012 at 11:40 AM, S. Daniel Francis fran...@sugarlabs.org wrote: So, the Python strings can be encoded in a Unicode compatible charset like utf-8, the Python Unicode type is a way to encode a string if you don't like to add a header and the recommended way to work in the program internally, so you mustn't use it for output, you will have to encode the content of type PyUnicode in a PyString with the UTF-8 charset for the output and it'll not generate any conflict. In general, yes, the switch to assuming strings are in UTF-8 format will not cause any conflict. Specially since we used to have that before, as others pointed out. Outside of existing Sugar code, you have to be careful with libraries that deal with binary data. Before utf-8, handling binary data and handling strings was just about the same. For example, a sequence of bytes and a sequence of chars would both work transparently with len() and general array manipulations (ie: myvar[33]). So for example, a pure python zip compression/decompression implementation now needs to clearly define it is _not_ working on utf-8 streams. We aren't really making an ASCII to UTF-8 transition, we are restoring UTF-8-as-default. So this is not an issue. But anytime you make such a transition, you have to review retest any code working with raw binary data. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Unicode strings in translations
On Wed, Aug 15, 2012 at 7:12 PM, Manuel Kaufmann humi...@gmail.com wrote: Take a look at this. Following what I understood from your email, if I ... We are veering far far offtopic from the subject. But string encoding is an important topic, so I'll go offtopic. [humitos@michifus ~]$ cat test.py #!/usr/bin/python # -*- coding: utf-8 -*- What is the context? Where are you typing this script? An xterm? A VT? In OFW? Over a serial port? Over SSH? In all cases, keystrokes have to be interpreted before you get the ó, and the OS needs to decide what input it'll give to the editor, and the editor needs to decide whether it will apply any translation. s = 'camión' Think about this: this line in your script could be written in a number of ways! ISO 8859-1 (Latin codepage), UTF-16, UTF-8, UTF-32 just to list the ones _you_ see most often. But even in Unicode, there are _two_ ways to say ó -- you can say letter o with acute or acute, composable + letter o. Oops! Try to install Yudit, or use iconv to transcode your nice python script to a few other encodings -- then look at it with a hex viewer. The thing is, when the python interpreter starts up, and reads _the script_ it doesn't know what encoding it is in. UTF-8 looks essentially identical to ISO 8859-1 -- so it cannot decide beforehand. You say, I'm in a modern system, it'll be utf-8! But perhaps it's an old script. Or your text editor is old and pre-unicode. How will Python know? Same applies to data files. That CSV file you are opening, maybe comes from MS Excel on Windows 3.11, German edition. Oops! But we are more used to thinking about data files -- every thing you know about data files also applies to Python program files (they are string data!) and to user input in the UI (except that you can usually ask the UI toolkit or the env what encoding it's feeding you). hth, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Unicode strings in translations
On Mon, Aug 13, 2012 at 7:15 PM, S. Daniel Francis fran...@sugarlabs.org wrote: I look some people is needing to know more about Unicode: Hi danielf, one thing to keep in mind is that UTF-8 is one of the possible ways to represent Unicode data. We will, most of the time, use UTF-8 mixed up with straight old ASCII. Our python code files are one example. But that is actually a bit of a trick. A very practical, pragmatic trick -- for example: we mostly didn't need a major rewrite of our text editors, or script parsers. That's great. But at the same time, we pay for it with a degree of ambiguity (is it an ASCII encoded file, or is it a UTF-8 encoded file?). So what we are out to do here is to reduce the ambiguity, declaring in all the relevant spots that these are UTF-8. The BOM header didn't become popular (for good technical reasons). Is there a way to tell the Python interpreter that assume that our code (inc strings in our code) should be assumed to be unicode? If so, we can set that in sugar-session. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Unicode strings in translations
On Tue, Aug 14, 2012 at 5:18 PM, S. Daniel Francis fran...@sugarlabs.org wrote: strings as Unicode and encode them as utf-8? I think there are too few cases when an activity gets translated strings and needs to work directly with unicode. Probably because you do not speak chinese or any other asian language :) We can represent all unicode characters in utf-8 and that's one of the danielf -- I think you are tangled in your confused terminology. UTF-8 _is a form of Unicode_. So when you say I think there are too few cases when an activity gets translated strings and needs to work directly with unicode, it is... well, wrong for any software that will be used internationally. Perhaps you mean something else, like the Python type Unicode. But in general discussion, people do use Unicode to mean... Unicode ;-) cheers, -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
