Re: [Sugar-devel] [SoaS] Recommendations for running Sugar on standard hardware (non-XOs)?
Uff, that is fairly dated! I would install latest Fedora with the Sugar Desktop Group as my first attempt. It won't be "OLPC-pretty", it'll expect users to log in with username/password, but other than that it should work. hth, m On Tue, Apr 8, 2014 at 12:39 PM, Christoph Derndorfer wrote: > Hi David, > > thanks for the link. > > Given that this page was last updated back in 2012 I'm wondering whether > anyone knows if the instructions are still up-to-date and working? > > Cheers, > Christoph > > > On Tue, Apr 8, 2014 at 8:19 AM, David Beveridge wrote: >> >> I would try this method.., >> >> http://wiki.sugarlabs.org/go/Tutorials/Installation/Install_with_liveinst >> >> >> On Tue, Apr 8, 2014 at 4:36 AM, Christoph Derndorfer >> wrote: >> > Hey everyone, >> > >> > a guy I know wants to set up a couple of older laptops with Sugar in a >> > sort >> > of "learning lab" in Germany and asked me what the best route for doing >> > that >> > was. Since I haven't dabbled in that area in quite a while I'm not sure >> > what >> > the best recommendations are these days: >> > >> > * using the SoaS version from late December? >> > * installing Fedora 20 and running Sugar on top of that? >> > * something entirely different? >> > >> > Any comments, suggestions, links, etc. would be much appreciated. >> > >> > Thanks, >> > Christoph >> > >> > -- >> > Christoph Derndorfer >> > >> > volunteer, OLPC (Austria) [www.olpc.at] >> > editor, OLPC News [www.olpcnews.com] >> > co-founder, TechnikBasteln® [www.technikbasteln.net] >> > >> > e-mail: christ...@derndorfer.eu >> > >> > >> > ___ >> > SoaS mailing list >> > s...@lists.sugarlabs.org >> > http://lists.sugarlabs.org/listinfo/soas >> > >> ___ >> SoaS mailing list >> s...@lists.sugarlabs.org >> http://lists.sugarlabs.org/listinfo/soas > > > > > -- > Christoph Derndorfer > > volunteer, OLPC (Austria) [www.olpc.at] > editor, OLPC News [www.olpcnews.com] > co-founder, TechnikBasteln® [www.technikbasteln.net] > > e-mail: christ...@derndorfer.eu > > > ___ > Sugar-devel mailing list > Sugar-devel@lists.sugarlabs.org > http://lists.sugarlabs.org/listinfo/sugar-devel > -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] shipped languages/locales
On Wed, Mar 26, 2014 at 7:59 AM, Gonzalo Odiard wrote: > I think was done to preserve space. > Not sure if that is a concern now. > Yep. It was definitely a concern for XO-1 and for XO-1.5 images for 2GB machines. m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Sugar Performance testing
On Mon, Mar 10, 2014 at 8:46 AM, Gonzalo Odiard wrote: > Activity startup time is in the shell.log if sugar have debug enabled. +100 on Gonzalo's recommendations. Also a couple of notes, from memory, - Gobject Introspection turned out to be slow to instantiate (IIRC, dsd posted analysis this topic). - The network view leaks AP icons. You need a good number of APs "visible" for this to be noticeable, if ps_mem.py tells you that the memory usage is growing over time, check whether disabling wifi stops the memory leaks. This should be an easy fix -- I researched this couple years ago. - any tests run on SD cards depend on the SD card performance, not two are alike... hth! m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] One for Grandma and Grandpa
On Wed, Mar 5, 2014 at 12:11 PM, Christian Stroetmann wrote: > The referenced summary on marc.info can not be understood without prior > knowledge and reading the whole threads. Oh, I think anyone reading your trolling emails should get some background on you based on your prior posts ( http://marc.info/?a=12160719884&r=1&w=2 ) , and your amazing website. Readers can form their own opinion. > In this respect, it must even be clear for you that I do not say 'Thanks' > for your informations. It is a pleasure to exchange emails with you. "I'm loving it". Keep telling us all about how you have trademarks on the stuff that we have been working on for years. And the name of your lawyer again. Did kernel devs really steal your copyrighted ideas from OntoFS? What mean people they are! And Apple, robbers! they took your copyrighted ideas too! Can I buy a Retina Projector from you, or one of your distributors? I would rather spend my money on the original inventor! But be careful. Any idea you post on this mailing list, if it's good someone is going to perhaps implement it, and you'll have to argue with them. I'm sure your lawyer is very busy with all that chasing Apple, and you'll have to add chasing Sugar folks to his workload. Not nice. On the other hand, if you stop posting here, you'll be safe. [ everyone else: I'm sorry, this is too much fun, I stop here ] m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] One for Grandma and Grandpa
On Wed, Mar 5, 2014 at 10:07 AM, Martin Langhoff wrote: > Your website is also a piece of art. Actually, I feel bad. Seems like Apple stole his copyrighted ideas, right down to the ID, http://www.ontolinux.com/community/hardware.htm Christian, you should get in touch with Steve Jobs biographers and claim the credit you deserve on the creation of so many Apple devices, as well as the upcoming Retina Projector. cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] One for Grandma and Grandpa
On Sat, Mar 1, 2014 at 10:33 AM, Christian Stroetmann wrote: > Please, respect our trademarks and our copyrights. Hi Christian! Have you registered your trademarks anywhere? USPTO doesn't know about them, but you could have a registration elsewhere. And I googled your name/email address, and I find that you've been trolling kernel and FS developers about your "copyrighted ideas" in FS design. The thread at http://www.kernelhub.org/?p=2&msg=281077 is a riot. You are an amazing troll. As Pavel said, fortunately, you can't copyright ideas. Chuck Norris managed to do it once, but you can't. Oh, and http://www.spinics.net/lists/reiserfs-devel/msg01543.html Your website is also a piece of art. This page seems to have all the trademarks you claim to own: http://www.ontonics.com/about/hinweis.htm -- it does not list any of the OLPC ripoff ones, but I see "I'm/you're/we're loving it". I am sure you have secured a juicy license from McDonalds - http://www.youtube.com/watch?v=eBlD2N_AwgI This page shows your "copyrighted ideas" http://www.ontonics.com/innovation/pipeline.htm -- filesystems, suspend-to-NVRAM concepts. And you harass kernel developers who are actually doing stuff -- http://marc.info/?a=12160719884&r=1&w=2 How about "Thanks, but NO, thanks" ? m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] One for Grandma and Grandpa
On Sat, Mar 1, 2014 at 11:11 AM, Christian Stroetmann wrote: > Yes indeed we have created a new concept that is basically the > transformation of the Sugar learning environment developed solely for very If all you have is a concept, and you are trying to register trademarks based on the work of Sugar and OLPC communities, that's not going to get you any friends in these lists. m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Fwd: [b2g] Introducing B2G to x86
On Fri, Feb 28, 2014 at 12:00 PM, Christian Stroetmann wrote: > In this respect I wrote before, that I do not think that Lego is in control > over its registered trademark 'LEGO' anymore, This has nothing to do with Sugar, nor OLPC. Perhaps a mailing list about trademarks is more appropriate (hint: you might find you are wrong). Many (most?) active members on this list have contributed a huge amount of their time over years to OLPC and Sugar, with a mix of volunteer and paid-for work. And essentially from the heart. If a person of subgroup is taking trademarks over something this group has worked on with passion is seen as a form of profiteering on the back of our devoted work. Unless you are going to hand over the trademarks to the SugarLabs non-profit, it just does not make you many friends around here. m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Feature proposal: ability to start an activity from inside another
On Tue, Jan 21, 2014 at 3:48 PM, Sam Parkinson wrote: > - Android has a really simple system where you can say go to > www.abc.blogs.org in any app that can open it. That opens it in a web > browser or in the blogs.org app if installed. This works because the > blogs.org app says it can handle these types of links in its manifest. Yep, we discussed Android's 'intents' machinery, which seems the most elegant API design for this, back with Manuq, Gonzalo and dsd in Parana a while back. Manuq -- is building a basic intents API too large a task? Or perhaps changing existing activities is the big elephant? cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Mails caught as spam
Same here -- sugar-devel, IAEP and server-devel. Taught my gmail some manners. thanks for the heads up, m On Thu, Dec 19, 2013 at 8:43 AM, Peter Robinson wrote: > On Thu, Dec 19, 2013 at 12:11 PM, Gonzalo Odiard wrote: >> Today I have found almost 10 mails from sugar-devel >> caught as spam by my gmail account. >> >> Half of them from Emil Dudev, and a few from a announce to a triage meeting. >> In all the cases, the mails have links. >> >> Anybody knows why these mails are marked in this way? >> If we have a clear reason, we can make suggestions to avoid this problem. > > I discovered a number of sugar mails in my gmail spam recently > although I couldn't see any pattern in my collection. > > Peter > ___ > Sugar-devel mailing list > Sugar-devel@lists.sugarlabs.org > http://lists.sugarlabs.org/listinfo/sugar-devel -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Private vs Public conversations.
On Mon, Oct 28, 2013 at 11:01 PM, David Farning wrote: > I would like to thank everyone who has provided valuable feedback by > participating on this thread. Ahem. You are casting fugly accusations, you can't stand back and thank everyone for their valuable feedback. > 2. Martin's point about the right hand not always being aware of what > the left hand is doing. This unfortunately seems to happen too > frequently. "Doing" is the wrong word, AFAICS. Did anyone working for OLPC _who was a maintainer to SL's trees_ ever rejected or "actively ignored" patches like what you say? The list of candidates is very short, and they are all highly ethical professionals. m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Private vs Public conversations.
On Mon, Oct 28, 2013 at 8:14 PM, David Farning wrote: > In a private conversation with an Association employee they told me > that they conciser Activity Central a competitor because Activity > Central increased deployments expectations. Their strategy with regard > to Activity Central was to _not_ accept patches upstream with the goal > of causing Activity Central and Dextrose to collapse under its their > weight. As it was private conversation I am not sure how widely spread > the opinion was held. I object very strongly to those statements; I hope it was not under my watch and I goes very much against the grain of everyone involved with Sugar and OLPC in all the time I was there. While I didn't always agree or like AC's work or strategies, I have been, on and off the record, always in favor of having a strong ecosystem. AC being the main player in that space, this translated in a strong advocacy for AC. As a professional in the foss world, this is not something I would accept in my team, and I don't think anyone in the team had the kind of personality to play such games. There were times where it was easy for OLPC to integrate patches, There were times when it was hard. I tried to signal that in advance because I have been on both sides of the integration game (and I continue to be -- now with Moodle) and I profoundly despise games such as the one being suggested. with a bad taste in my mouth, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Private vs Public conversations.
On Wed, Oct 23, 2013 at 12:26 PM, Walter Bender wrote: > On Wed, Oct 23, 2013 at 12:04 PM, David Farning > wrote: >> I just wanted to bump this line of questions as, it is the critical > > I don't speak on behalf of the Association, but I think your positions > are overstated. As far as I know, the Association is still pursing > sales of XO laptops and is still supporting XO laptops in the field. > Granted the pace of development is slowed and there is -- to my > knowledge -- no team in place to develop an follow up to the XO 4.0. I > don't have a clue as to what you mean by a "technical philanthropy" > but it remains a non-profit associated dedicated to enhancing learning > opportunities through one-to-one computing. The fact that the > Association has private-sector partners is nothing new. It has had > such partners since its founding in 2006. +1 on Walter's words, David's position is overstated. OLPC has shrunk its Sugar investment, that is true. But on the other points, nothing has changed significantly, OLPC has always had to find sources of funding. >> Given financial constraints, these are reasonable shifts. That's more like it ;-) >> there are ways to establish publicly disclosed and mutually beneficial >> relationships. In the meantime we are happy to provide deployments >> support while seeding and supporting projects we feel are beneficial >> to deployments such as School Server Community Edition and Sugar on >> Ubuntu. "Seeding and supporting projects" is how it's done. cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Activity Central's Sugar related priorities.
On Mon, Oct 7, 2013 at 12:41 PM, David Farning wrote: > As a more incremental approach, Activity Central will continue our > deployment-centric work by porting Dextrose to Ubuntu. >From a "deploy to XOs PoV" that sounds like a ton of work. You'll grind against a lot of little problems. Fedora is no longer behind nor problematic. That was very much true in earlier times. Some innovative things in Fedora (ie: systemd) have been very well integrated with the Sugar stack. And some changes in the Ubuntu pipeline are likely to cause some havoc. >From a "work for AC customers already using Ubuntu", it probably makes more sense. Still, the odd directions Ubuntu seems to be going are a bit of a wildcard. I honestly hope that they settle a bit and make life for their downstreams a bit easier. cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Fwd: [XSCE] Re: Client side Moodle transparent auth broken in 13.2.0 stable
On Sun, Aug 11, 2013 at 1:15 PM, Jerry Vonau wrote: > Good to hear from you Martin. Just to finish this thread off, I was not able > to reproduce this behavior with the XO-1s that I have. This appears to > affect Anna's machines only. Thanks for the hints to what might be the root > cause. Thanks for the greeting! I had a season of detox after some severe burnout. I'm spending this weekend at Fedora Flock for personal enjoyment, and it's brought me back to the OLPC topic. About Anna's machines -- I suspect either an old OFW or bad/broken/misconfigured manufacturing data. cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Fwd: [XSCE] Re: Client side Moodle transparent auth broken in 13.2.0 stable
On Tue, Aug 6, 2013 at 3:32 PM, Jerry Vonau wrote: > This behavior was noted with XO-1s only, all others(1.5,1.75,4) appear work > fine in testing. That is _very_ weird. Does the affected unit have any problem reading its own serial number or uuid? Does the Sugar Control Panel show the serial number correctly? cheers, m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Features: Background image in Home View
On Mon, Mar 25, 2013 at 11:48 AM, Manuel Quiñones wrote: > My first impression is that this feature can potentially hurt the > clean design of Sugar at some points: > > - the icons color semantic > - high contrast, accesibility Agreed. IMHO it can be improved by applying a "washout" -- mix the image with a white image at 50% alpha. > I would like to know more about why is this a long requested feature, Identity is a strong human urge. Kids put stickers on them, adults buy cars of different colors :-) > - Adding it only in the home view kills the ilussion of having three > zoom views. The background should be displayed in all three, I think. +1 > - Why it needs a restart? +1 m -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/24/en/User:Martin_Langhoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Window handling for non-Python activities
On Mon, Feb 25, 2013 at 12:22 PM, Daniel Drake wrote: > This didn't happen on older versions of Sugar (haven't checked why, > maybe the "failed to start" screen didn't exist before?) This patch seems related: http://git.sugarlabs.org/sugar/mainline/commit/dc8f6ed7852f919fe7123d458706fb82430257e9 It was written to address a similar issue with a Flash Sugar app that would swap windows around. It is (or should be) in current Sugar -- was applied a while ago. The apps you mention perhaps use libsugarize? Perhaps something broke it? cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Query regarding "xorg-x11-drv-dove" package, on XO-4
Thanks for the report! Jon Nettlelton is working on that particular driver, and there was a big overhaul that landed in OS29. Could you create a ticket on dev.laptop.org, against 13.1.0? This has nothing to do with Sugar itself... thanks! m On Thu, Feb 21, 2013 at 12:07 PM, Ajay Garg wrote: > Hi all. > > This is in relation to ticket http://bugs.sugarlabs.org/ticket/4438 > Is there a way to know what are the changes between version-8 (on XO-4 > build 30) and version-7 (on XO-4 build 21) for xorg-x11-drv-dove-0.3.5? > > > > Regards, > > Ajay Garg > Dextrose Developer > Activity Central: http://activitycentral.com > ___ > Devel mailing list > de...@lists.laptop.org > http://lists.laptop.org/listinfo/devel > -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Hacking onto the "appearing" and "hiding" of OSK
On Thu, Jan 24, 2013 at 12:02 PM, Gonzalo Odiard wrote: > Write does not know what is the ebook switch state, that logic is in the > osk. And that's correct. ebook mode is one reason to show the OSK. There are other reasons -- for example, - accesibility - typing in a different language from what your physical keyboard has That's why I suggest to Ajay to do as Write does :-) m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Hacking onto the "appearing" and "hiding" of OSK
On Thu, Jan 24, 2013 at 10:13 AM, Paul Fox wrote: > i believe sugar already has code to detect the two modes, since > that's how it knows whether to present the OSK or not. Yep. Ajay, I think Write shows you the way :-) m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Issue on the XO when trying to register to a XS
On Mon, Dec 31, 2012 at 6:20 AM, wrote: > Do you already know this issue? > What could be the next step to analyze the issue? What happens if you retry registration from the XO? OS versions on XO, XS? If the XO OS is recent, nothing comes to mind, except a transient network issue. Older XO OSs had some issues around registration, but none that I can recall with this behaviour. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [TRANSIENT] Peer XOs NOT shown in Neighborhood view when Power Management is enabled
On Wed, Dec 19, 2012 at 5:14 AM, Jerry Vonau wrote: > Think I found the problem, in powerd we're setting WOL based on this > string: > > if grep -qi ": :14B2" /proc/net/tcp > > but that string is not present in /proc/net/tcp so WOL is not set > according to ethtool, but that string can be found in /proc/net/tcp6 > > avahi is bound to tcp6 when viewed with 'netstat -nat' > > This is reproducible in 12.1.0 and 13.1.0 Arghhh. Ouch. Does it behave better with: if grep -qi ": :14B2" /proc/net/tcp* ? m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Porting Sugar on Android (Ice cream sandwich)
I'm sorry! Looks like I misread your name. A faux pass typical of borderline autistic people -- get all the technical details clearly, mess up the other person's name. Hope the technical part was useful. No offense intended. cheers, martin On Tue, Dec 11, 2012 at 3:12 PM, RJV wrote: > Thanks, Martin. Btw, where did you deduce the name "Rajiv" from? :) > > RJv > > On Wed, Dec 12, 2012 at 1:13 AM, Martin Langhoff > wrote: >> >> Implement a shell -- replacing the standard Android shell -- that >> has the main features of Sugar shell > > > > > -- > Regards, > > Ravichandran J.V. > http://ravichandranjv.blogspot.com > XO 1.75 - 12.1, 0.96, Build 21, Q4D17 > -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Porting Sugar on Android (Ice cream sandwich)
Hi Rajiv, your plan seems to have good goals, but is missing some understanding of what you can and cannot do. You cannot run Sugar (a Python-based window manager, based on traditional Linux sw stack) on the Android stack. Way too different. To reach your goals, however, you could try something like... - identify what good Android Apps exist that match Sugar Activities, those you don' t have to port, just replace :-) (ie: don't port Record!) - those Sugar Activities that don't have an Android counterpart, reimplement them as Android apps - Implement a shell -- replacing the standard Android shell -- that has the main features of Sugar shell - Implement some of the key services that make Sugar special: ad-hoc auto-networking, collaboration stack, the Journal. The most important, IMO, is the Journal, as Android's handling of "my files" is extremely poor. All these services will need integration into the shell and apps. Alternatively, you can wait for Android/Linux stack convergence projects (see Jolla / Sailfish) to mature. Maybe in a year or two there are ways to run Sugar inside Android or Android Apps on a Linux stack. hth, m On Tue, Dec 11, 2012 at 10:24 AM, RJV wrote: > Hi, > > We are planning to port Sugar on Android and are faced with these options: > > 1. Sugar as an application on Android. > 2. As a platform on top of the Linux platform. > > Can the Sugar build be used to create an app bundle to deploy to the app > store? Are there any licensing issues? > > On 2., can someone share their experience, if any, please? > -- > Regards, > > Ravichandran J.V. > http://ravichandranjv.blogspot.com > XO 1.75 - 12.1, 0.96, Build 21, Q4D17 > > > ___ > Devel mailing list > de...@lists.laptop.org > http://lists.laptop.org/listinfo/devel > -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Major bugs for "Software-Update"
On Tue, Nov 27, 2012 at 11:42 AM, Ajay Garg wrote: > http://bugs.sugarlabs.org/ticket/4274 > http://bugs.sugarlabs.org/ticket/4275 Looks like I can play this game too :-/ # 4281 - Activity updater - crashes updating activity m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [TRANSIENT ISSUE] 3G-Modem not being recognised
On Wed, Nov 21, 2012 at 3:07 AM, Alan Jhonn Aguiar Schwyn wrote: >> is there a way to block suspend when a class of USB device is plugged? > > This will be very important.. Disable the "automatic power-save" (AKA power > off usb..) It is already done in many cases. > if one device is using the usb comunication. This can be made? > What happens if you want to copy a large file to a pendrive? The system > power-off > the usb ever? In that case, we already do the right thing. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [TRANSIENT ISSUE] 3G-Modem not being recognised
On Wed, Nov 21, 2012 at 2:40 AM, James Cameron wrote: >> I don't think having an end-user with no experience with USB IDs add >> an entry to the usb-inhibits file, or having to remember to turn off >> a major feature is the correct long term solution IMHO. You guys are driving OOB, so you short-term you can tweak usb-inhibits with the 3G modems you have. And you've hacked powerd and other bits of infra in the past, if there's a better way (ie: is there a way to block suspend when a class of USB device is plugged? or perhaps an NM hook could set the inhibit-suspend flag?) you are uniquely well positioned to know, try likely approaches, etc... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [TRANSIENT ISSUE] 3G-Modem not being recognised
Ajay, folks, please indicate OS version, XO model, steps to repro (even if intermittent), and collect kernel logs so we can see WTH is going on. Otherwise we can only say "maybe" and speculate -- good stuff for idle converstation at a bar, but not productive if you want to see the problem diagnosed and solved... cheers, martin On Tue, Nov 20, 2012 at 3:33 AM, Ajay Garg wrote: > Hi all. > > I have been facing transient issues in getting the 3G-Modem recognised (via > the appearence of the corresponding icon in the bottom-tray). > > There really is no fixed pattern; sometimes the icon appears when the modem > is inserted; sometimes not. > Same is the case when the XO is rebooted, while the modem is inserted. Upon > booting, sometimes the icon appears; sometimes not. > > I don't remember such issues in 12.1.0; there, the icon appeared whenever it > ought to. > > > Is someone else too facing such transient issues; or am I the only one hit? > > > > Regards, > > Ajay Garg > Dextrose Developer > Activity Central: http://activitycentral.com > > ___ > Devel mailing list > de...@lists.laptop.org > http://lists.laptop.org/listinfo/devel > -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Renaming the Telescope activity to Scope
On Mon, Nov 19, 2012 at 11:58 PM, Chris Leonard wrote: > As I recall, this activity was already renamed once from xoscope after > it became clear it was colliding in name space with an oscilloscpe > activity. Renames are a pain in infrastructure, and in upgrade handling for users. I would say prefer to retain the current name until there's an overwhelming case for change. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [sugar-build] Recent improvements
On Fri, Nov 16, 2012 at 12:10 PM, Daniel Narvaez wrote: > * Much cleaner source code. I got rid of jhbuild and replaced it with > a simple python script which covers our need (and it's much easier to > customize). Whooo! After _years_ of discussion about getting rid of jhbuild, it's done! I think we should quickly tag Sugar 1.0 ;-) m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [FEATURE-BROKEN] "Send to Friend" feature not working in os11
How about filing a bug? :-) m On Fri, Nov 16, 2012 at 12:26 PM, Ajay Garg wrote: > Hi all. > > On XO-1.75s, os11 image, the "Send to Friend" feature isn't working. > > Is it a known issue? > Doing a simple textual "send to" search on bugs.sugarlabs.org did not yield > anything useful. > > > Regards, > > Ajay Garg > Dextrose Developer > Activity Central: http://activitycentral.com > > ___ > Devel mailing list > de...@lists.laptop.org > http://lists.laptop.org/listinfo/devel > -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Datastore is broken
Thanks for the heads up! Fix attached, m On Thu, Nov 8, 2012 at 1:11 PM, Daniel Narvaez wrote: > Not sure if it breaks anything but it doesn't sound good, with the latest git: > > Traceback (most recent call last): > File > "/home/buildbot/slave/fedora-17-32bit/build/install/lib/python2.7/site-packages/carquinyol/datastore.py", > line 180, in _rebuild_index > if os.path.exists(index_path): > NameError: global name 'index_path' is not defined > ___ > Sugar-devel mailing list > Sugar-devel@lists.sugarlabs.org > http://lists.sugarlabs.org/listinfo/sugar-devel -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff 0001-Bring-back-index_path-variable.patch Description: Binary data ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Datastore is broken
Hmmm, mkdir??? I'd suggest touch /home/user/.sugar/default/datastore/index m On Wed, Nov 7, 2012 at 2:35 PM, Ignacio Rodríguez wrote: > ¡For repair!: > mkdir /home/user/.sugar/default/datastore/index > :) > > > 2012/11/7 Martin Langhoff >> >> On Wed, Nov 7, 2012 at 11:51 AM, Simon Schampijer >> wrote: >> > On 11/07/2012 02:45 PM, Daniel Narvaez wrote: >> >> Buildbot tests are failing, apparently because of a datastore issue >> >> I owe you two a beer. A mighty big beer. Thanks for spotting this >> thinko before it landed in an OS build. >> >> > We fail because we try to access the index at the final path, but at >> > this >> > point we created it at the temp path only. The following patch does work >> > for >> > me. >> >> Complete agreement and signoff on your patch. It is a thinko on my >> part to be checking the size of the on-disk index, (which after a >> reboot in an ENOSPC condition would easily be 0, or some other corrupt >> value). The right thing to check is the index on the tmpfs, as Simon's >> patch does. >> >> thanks for the quick patch! >> >> >> >> m >> -- >> martin.langh...@gmail.com >> mar...@laptop.org -- Software Architect - OLPC >> - ask interesting questions >> - don't get distracted with shiny stuff - working code first >> - http://wiki.laptop.org/go/User:Martinlanghoff > > > > > -- > Juan Ignacio Rodríguez > CeibalJAM! > Activity Central > -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Datastore is broken
On Wed, Nov 7, 2012 at 11:51 AM, Simon Schampijer wrote: > On 11/07/2012 02:45 PM, Daniel Narvaez wrote: >> Buildbot tests are failing, apparently because of a datastore issue I owe you two a beer. A mighty big beer. Thanks for spotting this thinko before it landed in an OS build. > We fail because we try to access the index at the final path, but at this > point we created it at the temp path only. The following patch does work for > me. Complete agreement and signoff on your patch. It is a thinko on my part to be checking the size of the on-disk index, (which after a reboot in an ENOSPC condition would easily be 0, or some other corrupt value). The right thing to check is the index on the tmpfs, as Simon's patch does. thanks for the quick patch! m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Wed, Nov 7, 2012 at 2:31 AM, Simon Schampijer wrote: > Thanks Martin, Sam and Manuel for your work on this. The datastore runs here > fine. I pushed the changes. Will be available in 0.97.2. Great! I was going to write a longer thank-you for the review, but I ran out of space. :-) m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Multi-touch test activity
On Sun, Nov 4, 2012 at 8:29 PM, Bert Freudenberg wrote: > Ah, thanks. I wasn't even going to file a bug report about the aliasing > because that is a limitation inherent to the kind of sensor we have. There are of course limitations, but we are in the process of tuning and tightening things on the IR. All sensor types have some forms of aliasing. If you know how, you can confuse capacitive sensors too :-) m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH] Sort the activities in the home in alphabetic order
On Thu, Oct 4, 2012 at 3:15 PM, Gary Martin wrote: > Martin, could you clarify this example a little more, > did you have some specific documentation task in mind? I want to clarify that I see this patch as a big improvement, even with localized sorting. +100. We often prepare some screenshots for documentation (or the Sugar documentation sprint does). Because Sugar is icon-oriented (and light on strings) in the UI those screenshots are used and reused in documentation most of the time. In my experience, most local documentation efforts are a very quick "remix" of existing documentation. First-time deployments don't even have XOs in advance for the documentation team to get screenshots. If they have XOs, they often lack the expertise (they are first-time deployments, after all!). Gonzalo has seen this first-hand. With the proposed patch + non-localized strings for fav view sorting the home view remains (mostly) stable across all languages. Yes, it changes if you add/remove/replace activities, but only slightly. The relative position of activity icons remains mostly stable. With the proposed patch sorting on localized strings for fav view, each lang "randomizes" the sorting. Not so good, in my book, but not the end of the world either. Couple more notes - I completely support localized names sorting for list view. - For fav view the "best" sorting would be a fixed, OS build-time configurable sorting to group/order activities. cheers, martin - who worries too much about a sorting scheme kids immediately disable using the "wherever I want it to be" view. -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH] Sort the activities in the home in alphabetic order
On Thu, Oct 4, 2012 at 11:25 AM, Gonzalo Odiard wrote: > You have less consistency only if you change the language. I generally agree. My only worry is that documentation screenshots will be all over the place. > Have sense use the same order in the listview and in the favorites view too. yeah... ok. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH] Sort the activities in the home in alphabetic order
On Thu, Oct 4, 2012 at 10:33 AM, Gonzalo Odiard wrote: > We are sorting by localized name. > It's true than the order will be different if you use a different language, > but we think is better for the users. Less consistency in icon location better? How? I agree alphabetical is better for list view. Fav view, otoh, _any_ order is ok as long as it is stable... no? m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH] Sort the activities in the home in alphabetic order
On Thu, Oct 4, 2012 at 8:06 AM, wrote: > The change is done in the favorites view and in the list view What was the sorting before in the fav view? More importantly -- Is this sorting over the English names, or the localized names? I am not sure what we would desire, but I'll note that if it's localized names, then every language will have a diffent home view icon order. I tend to believe that a stable ordering is preferrable... but I understand other factors may apply... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH artwork] Style active state of activities in home view for tap and click feedback
On Wed, Oct 3, 2012 at 5:12 PM, Manuel Quiñones wrote: > And enlarging a bit: > > http://dev.laptop.org/~manuq/shell-port/icon-hover2.png > http://dev.laptop.org/~manuq/shell-port/icon-tap2.png I am partial towards the enlarged ones :-) cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH artwork] Style active state of activities in home view for tap and click feedback
On Wed, Oct 3, 2012 at 4:53 PM, Manuel Quiñones wrote: > Add a light grey background, which is the same size as the grey > outline that appears when the icon is hovered. Yes! This stuff is needed. Touch feedback needs to be bright, clear, and clearly larger than your finger so that you see it. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] GStreamer 1.0.0 has been released
On Wed, Oct 3, 2012 at 11:54 AM, Peter Robinson wrote: > You can still port to gstreamer 1.0 on 13.1.0 and just test with non > ugly codecs (vorbis / webm etc). Yep. Work now with the ugly codecs. Once vmeta is ready we'll have to retest & debug, as it seems to be more fragile. Extra work :-( m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH v2 Browse] Remove temporary downloaded (cancelled) files SL #3973
On Wed, Oct 3, 2012 at 6:52 AM, Manuel Kaufmann wrote: > On Tue, Oct 2, 2012 at 5:35 PM, Martin Langhoff > wrote: >> Actually, the answer is to avoid datetime. It's fine for high level >> work; when you care about times defined as epoch values, you > > Yes, you are right. But datetime is much more elegant Not at all. Ignoring the computer is not elegance. In this case, datetime is horrible. If you are writing a calendar (say, google calendar) yes, datetime is your friend. > Anyway, I think we must add this function inside a GObject.idle_add() Note that this function needs to run only _once_ and it is extremely cheap/fast. Large files do not take a long time to be removed. Try it. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH v2 Browse] Remove temporary downloaded (cancelled) files SL #3973
On Tue, Oct 2, 2012 at 3:51 PM, Martin Langhoff
wrote:
> Works as pseudocode. But I took a quick look at the datetime docs and
> it is... horrible. We want to be doing math on the epoch (seconds
> since 1970).
>
> They refuse to document it properly, but the answer to the riddle is
>
>datetime.datetime.now().strftime('%s.%f')
Actually, the answer is to avoid datetime. It's fine for high level
work; when you care about times defined as epoch values, you
import time
now = time.time()
so the code snippet should be (untested):
boot_time = int(boot_time) # discard the float part
now = int(time.time())
cutoff = now - 24 * 60 * 60
if uptime is not None:
boot_time = now - uptime
cutoff = max(cutoff, boot_time)
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH v2 Browse] Remove temporary downloaded (cancelled) files SL #3973
On Tue, Oct 2, 2012 at 2:52 PM, Manuel Kaufmann wrote:
> On Tue, Oct 2, 2012 at 1:43 PM, Martin Langhoff
> wrote:
>>> +except:
>>> +logging.warning('/proc/uptime could not be read')
>>> +uptime = None
>
> Should I use EnvironmentError[1] here too?
Yep. A bare 'except:' hides other things like syntax errors.
> What do you think about this?
Works as pseudocode. But I took a quick look at the datetime docs and
it is... horrible. We want to be doing math on the epoch (seconds
since 1970).
They refuse to document it properly, but the answer to the riddle is
datetime.datetime.now().strftime('%s.%f')
So this is probably closer, using only ints:
boot_time = int(boot_time) # discard the float part
now = datetime.datetime.now().strftime('%s')
cutoff = now - 24 * 60 * 60
if uptime is not None:
boot_time = now - uptime
cutoff = max(cutoff, boot_time)
enjoy,
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH v2 Browse] Remove temporary downloaded (cancelled) files SL #3973
On Tue, Oct 2, 2012 at 9:59 AM, Manuel Kaufmann wrote:
> +try:
> +uptime_proc = open('/proc/uptime', 'r').read()
> +uptime_seconds = float(uptime_proc.split()[0])
> +uptime = datetime.timedelta(seconds=uptime_seconds)
> +except:
> +logging.warning('/proc/uptime could not be read')
> +uptime = None
I like.
> +
> +temp_path = os.path.join(self.get_activity_root(), 'instance')
> +now = datetime.datetime.now()
Tiny optimization and elegance improvement: just calculate your cutoff
here. The cutoff is the largest of
- "yesterday" (now - 1 day)
- boottime (now - uptime) -- if you managed to read uptime
> +for f in os.listdir(temp_path):
> +if f.startswith('.goutputstream-'):
> +fpath = os.path.join(temp_path, f)
> +mtime = os.path.getmtime(fpath)
> +mdate = datetime.datetime.fromtimestamp(mtime)
> +delta = now - mdate
> +if delta.days > 0 or \
> +(uptime is not None and (now - uptime) > mdate):
replace this confitional with
if mdate < cutoff:
> +logging.warning('Removing old temporary file: %s', fpath)
> +try:
> +os.remove(fpath)
> +except OSError:
> +logging.error('Temporary file could not be '
> + 'removed: %s', fpath)
Should catch both OSError _and_ IOError. Or perhaps EnvironmentError
which, strange as the name might sound, is the granddaddy of both.
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Remove temporary downloaded (cancelled) files SL #3973
On Tue, Oct 2, 2012 at 8:24 AM, Manuel Kaufmann wrote: > On Tue, Oct 2, 2012 at 6:40 AM, Martin Langhoff > wrote: >> Good, but not enough. Compare mtime also to our boot time (ie: now - >> uptime). If the file is from before we booted, needs to be nuked. > > You say that I should add this test as well, right? I mean, compare if > the file is older than 1 day AND check the mtime with our boot time or > remove my test and use only this one? Both checks. As you say, a long uptime is a definite possibility. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Remove temporary downloaded (cancelled) files SL #3973
On Mon, Oct 1, 2012 at 3:14 PM, Manuel Kaufmann wrote: > This patch is a workaround to that behaviour. Every time that Browse > is started it looks for all the ".goutputstream" files in the > "instance" directory and checks its mtime. If it greater than 1 day it > removes the old temporary file. Good, but not enough. Compare mtime also to our boot time (ie: now - uptime). If the file is from before we booted, needs to be nuked. http://planzero.org/blog/2012/01/26/system_uptime_in_python,_a_better_way cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 3/7] metadatastore: store/change files on disk defensively #2317
On Mon, Oct 1, 2012 at 12:59 PM, Martin Langhoff wrote: >> From that section I understand that we are encoding the data into >> utf-8 if value is "unicode", but if it is just a "basestring", why we >> are doing "str()"? > > That's actually not changed code.. only changes indentation. I don't > like it, I very seriously don't like it, but fixing the problem is > outside the scope of this patchset. Why do we do str(), I don't know. > I believe it's a noop in this case. Actually, I know why. AIUI, data types coming from dbus can be ints and floats. And that var is coming from dbus. So we cannot write directly to a file without a cast. See http://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html#data-types > The problem is that we are dealing with some raw binary data (ie: > thumbnails) as if it were a string. You can get away with that as long > as your string ops are 8-bit clean. Once we move to utf-8, they won't > be. That table also indicates, in the 'notes' column, that all string data must be valid utf8. I don't know how we are getting the thumbnails through, and it seems likely that at some point it'll break, perhaps due to changes in dbus. We must change the thumbnail passing to be in a file, not too far in the future. cheers, m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 3/7] metadatastore: store/change files on disk defensively #2317
On Mon, Oct 1, 2012 at 12:46 PM, Manuel Kaufmann wrote: > Sorry for the Top-Posting, maybe you didn't see the section that I > included in my last email. Ah, yeah, didn't catch that. > From that section I understand that we are encoding the data into > utf-8 if value is "unicode", but if it is just a "basestring", why we > are doing "str()"? That's actually not changed code.. only changes indentation. I don't like it, I very seriously don't like it, but fixing the problem is outside the scope of this patchset. Why do we do str(), I don't know. I believe it's a noop in this case. The problem is that we are dealing with some raw binary data (ie: thumbnails) as if it were a string. You can get away with that as long as your string ops are 8-bit clean. Once we move to utf-8, they won't be. > The FIXME comment refers to a problem we will have with Python 3 in > the future, right? In the future in general. Yes, we'll hit it with Python 3, perhaps even sooner. m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 3/7] metadatastore: store/change files on disk defensively #2317
On Sun, Sep 30, 2012 at 2:10 PM, Manuel Kaufmann wrote: > The only thing that I do not understand well is this comment / > section. Can you explain to me this a bit more? Thanks What is unclear? The only thing I could add is a pre-amble: Previous code deleted all metadata entries for a given DS entry and then recreated them, on every call to update(). This caused immediate loss of metadata in the case of ENOSPC or any other I/O problem. With this patch: - only delete metadata files for keys that are being removed - only write files when the data changes - write/replace metadata files atomically, to avoid corrupting existing data in case of an error With this patch, we no longer corrupt metadata when trying to edit/update a ds entry with the system hitting ENOSPC. Signed-off-by: Martin Langhoff -- now, if you read carefully, it is the same thing I say in the last paragraph :-) m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 1/7] Add ds_clean flag to trigger index rebuilds #2095, #2317
On Sun, Sep 30, 2012 at 12:24 PM, Manuel Kaufmann wrote: > On Fri, Sep 21, 2012 at 12:55 AM, Martin Langhoff wrote: >> This gives us more complete coverage of cases where ENOSPC or >> other errors are hit when creating/updating datastore entries. > > I reviewed this patch and I think it's OK. I didn't test it because > I'm not sure how to simulate that scenario. Repro procedures are explained in my '0/5' cover letter. Use and abuse dd for your evil purposes. > Just a comment: why you didn't add a callback function > (_delete_completion_cb) to the "delete" method and make create / > update / delete consistent between them? In that method we can add > self._mark_clean My understanding is that delete does not get structured as a trigger/callback because delete is supposed to be quick. > Thought: we are dealing near ENOSPC, right? So, maybe the clean_flag > will not be created (this is already handled in the patch) and this > patch will not take effect the next time the XO restarts and we will > be in the same situation again. That is by design! We can only save our clean flags, both for ds and index, if we have disk space. Our state is "dirty" when they are missing, and they may very well be missing because we could not write them to disk! Think about it this way: when we start up, we default to rebuilding our index, assuming the ds and index may be corrupt. Only if we see the two clean flags we skip the rebuild (which leads to a faster startup). hth, m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 4/7] metadatareader: ignore ".hidden" files
On Mon, Oct 1, 2012 at 7:06 AM, Manuel Kaufmann wrote: > Just a comment: this patch removes the possibility to have a property > named "..property", right? is this OK? I hope it is. I don't know, I'm not an experienced Sugar hacker :-) What I can tell you is that I have never seen a property, in our test datastores, that had a name starting with '.'. cheers, m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] last-minute feature proposal for 0.98
On Fri, Sep 28, 2012 at 12:26 PM, Manuel Quiñones wrote: > 2012/9/28 Walter Bender : >> I am hoping to get this into the OLPC 13.1 build. Any change of >> getting it into Sugar 0.98? > > Looks like a good addition for me and patch looks fine. Hmmm, lunch time! Ah, ooops! I would add the field to src/carquinyol/indexstore.py _PROPERTIES_NOT_TO_INDEX cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Display only the URL in the URL entry SL #3553
On Thu, Sep 27, 2012 at 11:46 AM, Gary Martin wrote: > + exposing URLs for touch users. The cursor hover is the only way to expose > URLs when browsing (unless tap to editing them) Tap-to-switch-to-URL-and-edit, starting with "full URL is selected" is fine IMHO. You are missing a key "-", which is: - with several tabs open, titles are not readable; as can be seen on any modern webbrowser > I'm still a +1, just wish the change was more designer 'pretty' ;) If there's consensus, I'll get on with the show, and we move on. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Performace in os3
On Thu, Sep 27, 2012 at 10:40 AM, Manuel Kaufmann wrote: > I'm working on my 1.75 XO with os3 and I feel it really slow. There > are some examples: At least part of this will be due to a much slower video driver we are using temporarily. But yeah, it hurts and we need to have that fixed. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Display only the URL in the URL entry SL #3553
If there is concensus that it's better, then it's ok. It was one of
those things where I liked Browse more than Chrome, but perhaps it's a
personal quirk.
m
On Thu, Sep 27, 2012 at 10:33 AM, Gonzalo Odiard wrote:
> This change was discussed with Manuq and Gary.
> I can't find the discussion now.
> CC both
>
> Gonzalo
>
> On Thu, Sep 27, 2012 at 11:25 AM, Martin Langhoff
> wrote:
>>
>> Is this really an improvement in behaviour?
>>
>> - The tabs are often too small to show the title.
>> - The title is more important for the user than the URL. No?
>>
>> cheers,
>>
>>
>> m
>>
>> On Mon, Sep 24, 2012 at 5:09 PM, Manuel Kaufmann
>> wrote:
>> > The Title of the current page is no longer shown in the URL
>> > entry. Now, it's only shown in the tab and the current URL is visible
>> > all the time in the URL entry.
>> >
>> > Signed-off-by: Manuel Kaufmann
>> > ---
>> > browser.py| 13 +
>> > webtoolbar.py | 53
>> > +
>> > 2 files changed, 18 insertions(+), 48 deletions(-)
>> >
>> > diff --git a/browser.py b/browser.py
>> > index de546f2..1c67beb 100644
>> > --- a/browser.py
>> > +++ b/browser.py
>> > @@ -453,6 +453,10 @@ class Browser(WebKit.WebView):
>> > # Scale text and graphics:
>> > self.set_full_content_zoom(True)
>> >
>> > +# This property is used to set the title immediatly the user
>> > +# presses Enter on the URL Entry
>> > +self._loading_uri = None
>> > +
>> > # Reference to the global history and callbacks to handle it:
>> > self._global_history = globalhistory.get_global_history()
>> > self.connect('notify::load-status',
>> > self.__load_status_changed_cb)
>> > @@ -542,6 +546,15 @@ class Browser(WebKit.WebView):
>> > def open_new_tab(self, url):
>> > self.emit('new-tab', url)
>> >
>> > +def _set_loading_uri(self, uri):
>> > +self._loading_uri = uri
>> > +
>> > +def _get_loading_uri(self):
>> > +return self._loading_uri
>> > +
>> > +loading_uri = GObject.property(type=str, setter=_set_loading_uri,
>> > + getter=_get_loading_uri)
>> > +
>> > def __run_file_chooser(self, browser, request):
>> > picker = FilePicker(self)
>> > chosen = picker.run()
>> > diff --git a/webtoolbar.py b/webtoolbar.py
>> > index 28bc015..1d531bc 100644
>> > --- a/webtoolbar.py
>> > +++ b/webtoolbar.py
>> > @@ -47,7 +47,6 @@ class WebEntry(iconentry.IconEntry):
>> > GObject.GObject.__init__(self)
>> >
>> > self._address = None
>> > -self._title = None
>> > self._search_view = self._search_create_view()
>> >
>> > self._search_window = Gtk.Window(type=Gtk.WindowType.POPUP)
>> > @@ -57,8 +56,6 @@ class WebEntry(iconentry.IconEntry):
>> > self.connect('focus-in-event', self.__focus_in_event_cb)
>> > self.connect('populate-popup', self.__populate_popup_cb)
>> > self.connect('key-press-event', self.__key_press_event_cb)
>> > -self.connect('enter-notify-event',
>> > self.__enter_notify_event_cb)
>> > -self.connect('leave-notify-event',
>> > self.__leave_notify_event_cb)
>> > self._focus_out_hid = self.connect(
>> > 'focus-out-event', self.__focus_out_event_cb)
>> > self._change_hid = self.connect('changed', self.__changed_cb)
>> > @@ -79,18 +76,11 @@ class WebEntry(iconentry.IconEntry):
>> >
>> > def _set_address(self, address):
>> > self._address = address
>> > -if address is not None and self.props.has_focus:
>> > +if address is not None:
>> > self._set_text(address)
>> >
>> > address = GObject.property(type=str, setter=_set_address)
>> >
>> > -def _set_title(self, title):
>> > -self._title = title
>> > -if title is not None and not self.props.has_focus:
>> > -self._set_text(title)
>> > -
>> > -title = GObject.property(type=str, s
Re: [Sugar-devel] [PATCH Browse] Display only the URL in the URL entry SL #3553
Is this really an improvement in behaviour?
- The tabs are often too small to show the title.
- The title is more important for the user than the URL. No?
cheers,
m
On Mon, Sep 24, 2012 at 5:09 PM, Manuel Kaufmann wrote:
> The Title of the current page is no longer shown in the URL
> entry. Now, it's only shown in the tab and the current URL is visible
> all the time in the URL entry.
>
> Signed-off-by: Manuel Kaufmann
> ---
> browser.py| 13 +
> webtoolbar.py | 53 +
> 2 files changed, 18 insertions(+), 48 deletions(-)
>
> diff --git a/browser.py b/browser.py
> index de546f2..1c67beb 100644
> --- a/browser.py
> +++ b/browser.py
> @@ -453,6 +453,10 @@ class Browser(WebKit.WebView):
> # Scale text and graphics:
> self.set_full_content_zoom(True)
>
> +# This property is used to set the title immediatly the user
> +# presses Enter on the URL Entry
> +self._loading_uri = None
> +
> # Reference to the global history and callbacks to handle it:
> self._global_history = globalhistory.get_global_history()
> self.connect('notify::load-status', self.__load_status_changed_cb)
> @@ -542,6 +546,15 @@ class Browser(WebKit.WebView):
> def open_new_tab(self, url):
> self.emit('new-tab', url)
>
> +def _set_loading_uri(self, uri):
> +self._loading_uri = uri
> +
> +def _get_loading_uri(self):
> +return self._loading_uri
> +
> +loading_uri = GObject.property(type=str, setter=_set_loading_uri,
> + getter=_get_loading_uri)
> +
> def __run_file_chooser(self, browser, request):
> picker = FilePicker(self)
> chosen = picker.run()
> diff --git a/webtoolbar.py b/webtoolbar.py
> index 28bc015..1d531bc 100644
> --- a/webtoolbar.py
> +++ b/webtoolbar.py
> @@ -47,7 +47,6 @@ class WebEntry(iconentry.IconEntry):
> GObject.GObject.__init__(self)
>
> self._address = None
> -self._title = None
> self._search_view = self._search_create_view()
>
> self._search_window = Gtk.Window(type=Gtk.WindowType.POPUP)
> @@ -57,8 +56,6 @@ class WebEntry(iconentry.IconEntry):
> self.connect('focus-in-event', self.__focus_in_event_cb)
> self.connect('populate-popup', self.__populate_popup_cb)
> self.connect('key-press-event', self.__key_press_event_cb)
> -self.connect('enter-notify-event', self.__enter_notify_event_cb)
> -self.connect('leave-notify-event', self.__leave_notify_event_cb)
> self._focus_out_hid = self.connect(
> 'focus-out-event', self.__focus_out_event_cb)
> self._change_hid = self.connect('changed', self.__changed_cb)
> @@ -79,18 +76,11 @@ class WebEntry(iconentry.IconEntry):
>
> def _set_address(self, address):
> self._address = address
> -if address is not None and self.props.has_focus:
> +if address is not None:
> self._set_text(address)
>
> address = GObject.property(type=str, setter=_set_address)
>
> -def _set_title(self, title):
> -self._title = title
> -if title is not None and not self.props.has_focus:
> -self._set_text(title)
> -
> -title = GObject.property(type=str, setter=_set_title)
> -
> def _search_create_view(self):
> view = Gtk.TreeView()
> view.props.headers_visible = False
> @@ -146,21 +136,11 @@ class WebEntry(iconentry.IconEntry):
> self._search_window.hide()
>
> def __focus_in_event_cb(self, entry, event):
> -self._set_text(self._address)
> self._search_popdown()
>
> def __focus_out_event_cb(self, entry, event):
> -self._set_text(self._title)
> self._search_popdown()
>
> -def __enter_notify_event_cb(self, entry, event):
> -if not entry.props.has_focus:
> -self._set_text(self._address)
> -
> -def __leave_notify_event_cb(self, entry, event):
> -if not entry.props.has_focus:
> -self._set_text(self._title)
> -
> def __view_button_press_event_cb(self, view, event):
> model = view.get_model()
>
> @@ -241,7 +221,6 @@ class PrimaryToolbar(ToolbarBase):
> self._tabbed_view = tabbed_view
>
> self._loading = False
> -self._title = _('Untitled')
>
> toolbar = self.toolbar
> activity_button = ActivityToolbarButton(self._activity)
> @@ -310,7 +289,6 @@ class PrimaryToolbar(ToolbarBase):
> self._loading_changed_hid = None
> self._progress_changed_hid = None
> self._session_history_changed_hid = None
> -self._title_changed_hid = None
> self._uri_changed_hid = None
>
> if tabbed_view.get_n_pages():
> @@ -324,25 +302,19 @@ class PrimaryToolbar(ToolbarBase):
>
> def _connect_to_browser(self, browser):
> if self._browser is not None:
> -se
[Sugar-devel] using logging vs logger
Sugar{,-toolkit,toolkit3,datastore} sources have a mix of direct calls
to the logging module, and calls to it via the sugar logger instance
of the logging module, which is configured.
Early in sugar-session, I understand the use of logging before
sugar.logger is setup. But after that... is there any logic to the
direct use of logging?
"it's messy/inconsistent but it works either way" is an acceptable
answer :-) -- OTOH, if there are subtleties at play, I would really
like to understand them.
Logging functionality does _not_ work correctly in all cases for the
datastore, and I'd like to fix that. Some tracebacks are being written
to the dbus daemon -- and the caller at the other end of the dbus
conversation does not understand the response. Sometimes *that* gets
logged but it's very unwieldy to debug in these conditions.
cheers,
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH] RFC: Optimizer: simplify, avoid hashing files, remove 'checksums' dir
This patch changes the strategy used by optimize.
Instead of maintaining a 'checksum' field for every
file, and maintaining metadata about whether they are
linked in the 'checksums' dir, it will only ever hash
files that match size _exactly_ with a new file.
A DS that does not see 2 files of identical size will
never hash any file.
- Allows the datastore to handle large files (downloads,
video captures) without paying a high 'hashing' price.
- Removes the 'checksums' dirtree; which was redundant:
we can evaluate whether two files are hardlinked comparing
their inode.
- There are no users of 'checksum' metadata field out of
DS, so we remove it completely.
- Removal of checksum metadata and the 'checksums' dirtree
means less risk of out-of-date metadata.
Caveats:
- Assumes no program is producing large files of identical size.
On the XO, this is true. Some file formats with no compression
could trigger unwelcome hashing (ie: an activity that saves TIFF
or BMP files).
- The optimizer should run in a separate process, to avoid making the DS
unresponsive. Was written to avoid races, but would need to flock()
the file between hashing it and hardlinking it.
I need to read up on forking a long-running process from a dbus service.
---
Background: reviewing DS code and datastuctures, the checksums dirtree
and strategy in optimize.py looked problematic to me. If I am saving a large
video, I don't want it hashed (expensive op) if I can (cheaply) check that
no other file is the same size.
I could retain the 'checksum' metadata field -- and add some assurances
that it does not get stale. It would help with large and identical-sized
files, such as screenshots, which may otherwise end up hashed more than
once, perhaps many times.
---
src/carquinyol/datastore.py |4 -
src/carquinyol/filestore.py | 12 ---
src/carquinyol/layoutmanager.py |9 --
src/carquinyol/metadatastore.py |4 +-
src/carquinyol/optimizer.py | 225 ++-
5 files changed, 106 insertions(+), 148 deletions(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index a859dfe..0ee70d2 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -362,9 +362,6 @@ class DataStore(dbus.service.Object):
self._metadata_store.store(uid, props)
self._index_store.store(uid, props)
-if os.path.exists(self._file_store.get_file_path(uid)) and \
-(not file_path or os.path.exists(file_path)):
-self._optimizer.remove(uid)
self._file_store.store(uid, file_path, transfer_ownership,
lambda * args: self._update_completion_cb(async_cb,
async_err_cb,
@@ -489,7 +486,6 @@ class DataStore(dbus.service.Object):
self._mark_dirty()
try:
entry_path = layoutmanager.get_instance().get_entry_path(uid)
-self._optimizer.remove(uid)
self._index_store.delete(uid)
self._file_store.delete(uid)
self._metadata_store.delete(uid)
diff --git a/src/carquinyol/filestore.py b/src/carquinyol/filestore.py
index 0b34b69..38a4018 100644
--- a/src/carquinyol/filestore.py
+++ b/src/carquinyol/filestore.py
@@ -146,18 +146,6 @@ class FileStore(object):
if os.path.exists(file_path):
os.remove(file_path)
-def hard_link_entry(self, new_uid, existing_uid):
-existing_file = layoutmanager.get_instance().get_data_path(
-existing_uid)
-new_file = layoutmanager.get_instance().get_data_path(new_uid)
-
-logging.debug('removing %r', new_file)
-os.remove(new_file)
-
-logging.debug('hard linking %r -> %r', new_file, existing_file)
-os.link(existing_file, new_file)
-
-
class AsyncCopy(object):
"""Copy a file in chunks in the idle loop.
diff --git a/src/carquinyol/layoutmanager.py b/src/carquinyol/layoutmanager.py
index 3179a98..ee8270c 100644
--- a/src/carquinyol/layoutmanager.py
+++ b/src/carquinyol/layoutmanager.py
@@ -35,9 +35,6 @@ class LayoutManager(object):
if not os.path.exists(self._root_path):
os.makedirs(self._root_path)
-self._create_if_needed(self.get_checksums_dir())
-self._create_if_needed(self.get_queue_path())
-
def _create_if_needed(self, path):
if not os.path.exists(path):
os.makedirs(path)
@@ -74,12 +71,6 @@ class LayoutManager(object):
def get_index_path(self):
return os.path.join(self._root_path, 'index')
-def get_checksums_dir(self):
-return os.path.join(self._root_path, 'checksums')
-
-def get_queue_path(self):
-return os.path.join(self.get_checksums_dir(), 'queue')
-
def find_all(self):
uids = []
for f in os.listdir(self._root_path):
diff --git a/src/carquinyol/metadatastore.py b/src/carquinyol/metadatastore.py
index 52
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Sat, Sep 22, 2012 at 10:09 PM, Martin Langhoff wrote: > On Fri, Sep 21, 2012 at 11:36 AM, Martin Langhoff > wrote: >> Reviews are boring. Testing is more fun! rpms for ARM at Reviews are boring, but necessary. As these patches have seen some reviews, and the ml is a bit awkward for tracking those, I've pushed a 'pu' (proposed updates) branch at http://dev.laptop.org/git/users/martin/sugar-datastore/ I have posted all the patches to the mailing list, and will continue to do so. The pu branch will have the latest, freshest set of patches (so: it rewinds/rebases as needed). m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Error page SL #3500
On Tue, Sep 25, 2012 at 12:43 PM, Manuel Kaufmann wrote: > ** Message: console message: @0: Not allowed to load local resource: > file:///home/humitos/src/browse/browse.png > > What do you think? Should I go for the file:// approach anyway? Hmmm, security, tiny performance tweak, what should I choose? ;-) Good catch. I didn' t know it had been locked down this way. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Fri, Sep 21, 2012 at 11:36 AM, Martin Langhoff wrote: > Reviews are boring. Testing is more fun! rpms for ARM at > >http://dev.laptop.org/~martin/ds-enospc/ New rpms for testing, including "datastore: handle low-disk and ENOSPC conditions gracefully - v2". This is exactly what Sam is testing. > The datastore.org will be telling you whether we are using tmpfs for make that datastore.log :-) m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Error page SL #3500
On Fri, Sep 21, 2012 at 6:56 PM, Manuel Kaufmann wrote: > I'm not sure to understand what you are saying here. I'm not doing > this each time the function is called. Here, I just showed what I did > to get the base64 file encoded. I mean, I did this just once and I put > the result inside the HTML file. Epiphany does the same: it has > hardcoded the image inside the html. Is that a bad idea? Yeah, it's a wasteful approach. Probably doesn't matter much. You have the file on disk, correct? So you have the file path. You know that the webkit engine can read the file, too (it's on the same machine, in a readable directory, etc). So just put . That saves a some of pointless CPU cycles, some RAM. This old man doesn't like inefficient code, even in small details. Mostly because that means that when you code some that is hot (ie: is exercised a lot, so is a potential bottleneck or memory hog if inefficient) you have the right instincts. I've spent many years fixing performance issues in large systems where people coded without much care to efficiency (or error handling). So there's a gnome in the back of my mind that kicks me if I write careless code... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 7/7] datastore: handle low-disk and ENOSPC conditions gracefully
With this commit, the datastore comes up even on ENOSPC and very
tight disk conditions, and allows entry deletions even when at
ENOSPC.
- Be conservative
- ds or index flags are dirty -> rebuild
- less than 5MB available -> rebuild
- migrated or upgraded -> rebuild
- Only skip an index rebuild if things look very clean
and good. Skipping the index rebuild is an optimization.
- If a straight index open fails, we attempt a rebuild.
- Updating a partial index is unreliable, always rebuild
- When rebuilding the index, the new index is placed on a tmpdir
(on Fedora and OLPC builds, this is a tmpfs). It is only moved
to disk if we are not in low-disk-space-available conditions.
Signed-off-by: Martin Langhoff
---
This is v2 of the patch, making sure the move to internal disk
works in more cases, and that failure is recorded.
Thanks Sam for the report!
---
src/carquinyol/datastore.py | 89 +++
1 file changed, 73 insertions(+), 16 deletions(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 01d175e..a859dfe 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -23,6 +23,8 @@ import uuid
import time
import os
import shutil
+import subprocess
+import tempfile
import dbus
import dbus.service
@@ -44,6 +46,7 @@ DS_LOG_CHANNEL = 'org.laptop.sugar.DataStore'
DS_SERVICE = "org.laptop.sugar.DataStore"
DS_DBUS_INTERFACE = "org.laptop.sugar.DataStore"
DS_OBJECT_PATH = "/org/laptop/sugar/DataStore"
+MIN_INDEX_FREE_BYTES = 1024 * 1024 * 5
logger = logging.getLogger(DS_LOG_CHANNEL)
@@ -70,35 +73,53 @@ class DataStore(dbus.service.Object):
root_path = layoutmanager.get_instance().get_root_path()
self._cleanflag = os.path.join(root_path, 'ds_clean')
-if migrated:
+if initiated:
+logging.debug('Initiate datastore')
self._rebuild_index()
+self._index_store.flush()
+self._mark_clean()
return
-try:
-self._index_store.open_index()
-except Exception:
-logging.exception('Failed to open index, will rebuild')
+if migrated:
self._rebuild_index()
+self._mark_clean()
return
-if initiated:
-logging.debug('Initiate datastore')
-self._index_store.flush()
-elif not self._index_store.index_updated:
-logging.debug('Index is not up-to-date, will update')
-self._update_index()
+rebuild = False
+stat = os.statvfs(root_path)
+da = stat.f_bavail * stat.f_bsize
+
+if not self._index_store.index_updated:
+logging.warn('Index is not up-to-date')
+rebuild = True
elif not os.path.exists(self._cleanflag):
-logging.debug('DS state is not clean, will update')
-self._update_index()
-self._mark_clean()
+logging.warn('DS state is not clean')
+rebuild = True
+elif da < MIN_INDEX_FREE_BYTES:
+logging.warn('Disk space tight for index')
+rebuild = True
+
+if rebuild:
+logging.warn('Trigger index rebuild')
+self._rebuild_index()
+ else:
+# fast path
+try:
+self._index_store.open_index()
+except:
+logging.exception('Failed to open index')
+# try...
+self._rebuild_index()
+self._mark_clean()
+return
def _mark_clean(self):
try:
f = open(self._cleanflag, 'w')
os.fsync(f.fileno())
f.close()
-except Exception:
+except:
logging.exception("Could not mark the datastore clean")
def _mark_dirty(self):
@@ -135,8 +156,44 @@ class DataStore(dbus.service.Object):
"""Remove and recreate index."""
self._index_store.close_index()
self._index_store.remove_index()
-self._index_store.open_index()
+
+# rebuild the index in tmpfs to better handle ENOSPC
+temp_index_path = tempfile.mkdtemp(prefix='sugar-datastore-index-')
+logger.warn('Rebuilding index in %s' % temp_index_path)
+self._index_store.open_index(temp_path=temp_index_path)
self._update_index()
+self._index_store.close_index()
+
+on_disk=False
+
+# can we fit the index on disk? get disk usage in bytes...
+index_du = subprocess.check_output(['/usr/bin/du', '-bs',
+temp_index_path])
+index_du = int(index_du.split('\t')[0])
+#
Re: [Sugar-devel] [PATCH 3/7] metadatastore: store/change files on disk defensively #2317
On Thu, Sep 20, 2012 at 11:55 PM, Martin Langhoff wrote:
> -f = open(os.path.join(metadata_path, key), 'w')
> -try:
> -if isinstance(value, unicode):
> -value = value.encode('utf-8')
> -elif not isinstance(value, basestring):
> -value = str(value)
> -f.write(value)
> -finally:
> -f.close()
BTW, I simplified the structure of
f = open(path, 'w')
try:
f.write(value)
finally:
f.close()
because in practice, f.write() writes into a buffer, so it never
fails. It is open() and close() that throw exceptions. In fact, at
ENOSPC, close() was failing.
And the use of finally is misguided. In my code, which wraps
open/write/close in a single try, if f.write() does throw an
exception, Python will close the FH when it goes out of scope.
Simpler, saner.
Python won't leak FHs on you. (IOWs, it takes some work to leak FHs on
any modern scripting language).
cheers,
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Thu, Sep 20, 2012 at 11:55 PM, Martin Langhoff wrote: > In conditions of very low disk space available, or ENOSPC, the datastore > misbehaves grossly: Reviews are boring. Testing is more fun! rpms for ARM at http://dev.laptop.org/~martin/ds-enospc/ How to test: - Set Sugar to verbose debugging - Make sure sugar logs are not on /, add a file in /etc/statetab.d containing "/home/olpc/.sugar/default/logs" - use "dd if=/dev/zero of=/bigfile bs=1M count=" to fill up your disk I usually create one large file to put the system where it has little disk space remaining, then add/remove smaller files to tighten the free space further, or to push all the way to ENOSPC. This is faster than recreating a very large file everytime. The datastore.org will be telling you whether we are using tmpfs for the index, when a rebuild is triggered, ENOSPC is hit, etc. It is also interesting to monitor the presence of two "clean" markers in the datastore dir, ds_clean and index_updated. You can use inotifywatch -e create -e delete /home/olpc/.sugar/default/datastore cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 5/7] indexstore: exit on _flush() errors, work on tmpdir
Preparations for better index management.
- open_index() now can place the index on
a temporary location (ie: a tmpfs when
low on diskspace or at ENOSPC)
- don't set index_updated flag when on
a temporary location
- exit the process when hitting an error
in _flush(). This is normally caused by
running into ENOSPC during a session, and
is unrecoverable. dbus will respawn the
process, the new spawn has a chance at
cleanup, moving things to tmpfs.
Signed-off-by: Martin Langhoff
---
This is v2 of the patch, making sure that later sessions
rebuild the on-disk index.
src/carquinyol/indexstore.py | 57 +-
1 file changed, 45 insertions(+), 12 deletions(-)
diff --git a/src/carquinyol/indexstore.py b/src/carquinyol/indexstore.py
index dc721e7..0bbd93f 100644
--- a/src/carquinyol/indexstore.py
+++ b/src/carquinyol/indexstore.py
@@ -235,12 +235,32 @@ class IndexStore(object):
self._database = None
self._flush_timeout = None
self._pending_writes = 0
-self._index_updated_path = os.path.join(
-layoutmanager.get_instance().get_root_path(), 'index_updated')
-
-def open_index(self):
-index_path = layoutmanager.get_instance().get_index_path()
-self._database = WritableDatabase(index_path, xapian.DB_CREATE_OR_OPEN)
+root_path=layoutmanager.get_instance().get_root_path()
+self._index_updated_path = os.path.join(root_path,
+'index_updated')
+self._std_index_path = layoutmanager.get_instance().get_index_path()
+ self._index_path = self._std_index_path
+
+def open_index(self, temp_path=False):
+# callers to open_index must be able to
+# handle an exception -- usually caused by
+# IO errors such as ENOSPC and retry putting
+# the index on a temp_path
+if temp_path:
+try:
+# mark the on-disk index stale
+self._set_index_updated(False)
+except:
+pass
+self._index_path = temp_path
+else:
+ self._index_path = self._std_index_path
+try:
+ self._database = WritableDatabase(self._index_path,
+ xapian.DB_CREATE_OR_OPEN)
+except Exception as e:
+ logging.error('Exception opening database')
+ raise
def close_index(self):
"""Close index database if it is open."""
@@ -248,14 +268,18 @@ class IndexStore(object):
return
self._flush(True)
-self._database = None
+try:
+# does Xapian write in its destructors?
+self._database = None
+except Exception as e:
+logging.error('Exception tearing down database')
+raise
def remove_index(self):
-index_path = layoutmanager.get_instance().get_index_path()
-if not os.path.exists(index_path):
+if not os.path.exists(self._index_path):
return
-for f in os.listdir(index_path):
-os.remove(os.path.join(index_path, f))
+for f in os.listdir(self._index_path):
+os.remove(os.path.join(self._index_path, f))
def contains(self, uid):
postings = self._database.postlist(_PREFIX_FULL_VALUE + \
@@ -347,6 +371,9 @@ class IndexStore(object):
index_updated = property(get_index_updated)
def _set_index_updated(self, index_updated):
+if self._std_index_path != self._index_path:
+ # operating from tmpfs
+ return True
if index_updated != self.index_updated:
if index_updated:
index_updated_file = open(self._index_updated_path, 'w')
@@ -374,7 +401,13 @@ class IndexStore(object):
self._pending_writes += 1
if force or self._pending_writes > _FLUSH_THRESHOLD:
-self._database.flush()
+try:
+self._database.flush()
+except Exception, e:
+logging.exception(e)
+logging.error("Exception during database.flush()")
+# bail out to trigger a reindex
+sys.exit(1)
self._pending_writes = 0
self._set_index_updated(True)
else:
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 7/7] datastore: handle low-disk and ENOSPC conditions gracefully
With this commit, the datastore comes up even on ENOSPC and very
tight disk conditions, and allows entry deletions even when at
ENOSPC.
- Be conservative
- ds or index flags are dirty -> rebuild
- less than 5MB available -> rebuild
- migrated or upgraded -> rebuild
- Only skip an index rebuild if things look very clean
and good. Skipping the index rebuild is an optimization.
- If a straight index open fails, we attempt a rebuild.
- Updating a partial index is unreliable, always rebuild
- When rebuilding the index, the new index is placed on a tmpdir
(on Fedora and OLPC builds, this is a tmpfs). It is only moved
to disk if we are not in low-disk-space-available conditions.
Signed-off-by: Martin Langhoff
---
src/carquinyol/datastore.py | 87 +++
1 file changed, 71 insertions(+), 16 deletions(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 01d175e..67bda06 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -23,6 +23,8 @@ import uuid
import time
import os
import shutil
+import subprocess
+import tempfile
import dbus
import dbus.service
@@ -44,6 +46,7 @@ DS_LOG_CHANNEL = 'org.laptop.sugar.DataStore'
DS_SERVICE = "org.laptop.sugar.DataStore"
DS_DBUS_INTERFACE = "org.laptop.sugar.DataStore"
DS_OBJECT_PATH = "/org/laptop/sugar/DataStore"
+MIN_INDEX_FREE_BYTES = 1024 * 1024 * 5
logger = logging.getLogger(DS_LOG_CHANNEL)
@@ -70,35 +73,53 @@ class DataStore(dbus.service.Object):
root_path = layoutmanager.get_instance().get_root_path()
self._cleanflag = os.path.join(root_path, 'ds_clean')
-if migrated:
+if initiated:
+logging.debug('Initiate datastore')
self._rebuild_index()
+self._index_store.flush()
+self._mark_clean()
return
-try:
-self._index_store.open_index()
-except Exception:
-logging.exception('Failed to open index, will rebuild')
+if migrated:
self._rebuild_index()
+self._mark_clean()
return
-if initiated:
-logging.debug('Initiate datastore')
-self._index_store.flush()
-elif not self._index_store.index_updated:
-logging.debug('Index is not up-to-date, will update')
-self._update_index()
+rebuild = False
+stat = os.statvfs(root_path)
+da = stat.f_bavail * stat.f_bsize
+
+if not self._index_store.index_updated:
+logging.warn('Index is not up-to-date')
+rebuild = True
elif not os.path.exists(self._cleanflag):
-logging.debug('DS state is not clean, will update')
-self._update_index()
-self._mark_clean()
+logging.warn('DS state is not clean')
+rebuild = True
+elif da < MIN_INDEX_FREE_BYTES:
+logging.warn('Disk space tight for index')
+rebuild = True
+
+if rebuild:
+logging.warn('Trigger index rebuild')
+self._rebuild_index()
+ else:
+# fast path
+try:
+self._index_store.open_index()
+except:
+logging.exception('Failed to open index')
+# try...
+self._rebuild_index()
+self._mark_clean()
+return
def _mark_clean(self):
try:
f = open(self._cleanflag, 'w')
os.fsync(f.fileno())
f.close()
-except Exception:
+except:
logging.exception("Could not mark the datastore clean")
def _mark_dirty(self):
@@ -135,8 +156,42 @@ class DataStore(dbus.service.Object):
"""Remove and recreate index."""
self._index_store.close_index()
self._index_store.remove_index()
-self._index_store.open_index()
+
+# rebuild the index in tmpfs to better handle ENOSPC
+temp_index_path = tempfile.mkdtemp(prefix='sugar-datastore-index-')
+logger.warn('Rebuilding index in %s' % temp_index_path)
+self._index_store.open_index(temp_path=temp_index_path)
self._update_index()
+self._index_store.close_index()
+
+on_disk=False
+
+# can we fit the index on disk? get disk usage in bytes...
+index_du = subprocess.check_output(['/usr/bin/du', '-bs',
+temp_index_path])
+index_du = int(index_du.split('\t')[0])
+# disk available, in bytes
+index_path = layoutmanager.get_instance().get_index_path()
+stat = os.statvfs(in
[Sugar-devel] [PATCH 6/7] datastore: make delete() more reliable, log properly
- use try/except to catch and log errors in datastore.log
(instead of echoing them to callers over dbus)
- use rmtree() to more reliably remove the entry
- get_entry_path() before we delete all the related
metadata.
Signed-off-by: Martin Langhoff
---
src/carquinyol/datastore.py | 24
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 33c658d..01d175e 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -430,14 +430,22 @@ class DataStore(dbus.service.Object):
out_signature='')
def delete(self, uid):
self._mark_dirty()
-self._optimizer.remove(uid)
-
-self._index_store.delete(uid)
-self._file_store.delete(uid)
-self._metadata_store.delete(uid)
-
-entry_path = layoutmanager.get_instance().get_entry_path(uid)
-os.removedirs(entry_path)
+try:
+entry_path = layoutmanager.get_instance().get_entry_path(uid)
+self._optimizer.remove(uid)
+self._index_store.delete(uid)
+self._file_store.delete(uid)
+self._metadata_store.delete(uid)
+# remove the dirtree
+shutil.rmtree(entry_path)
+try:
+# will remove the hashed dir if nothing else is there
+os.removedirs(os.path.dirname(entry_path))
+except:
+pass
+except:
+logger.exception('Exception deleting entry')
+raise
self.Deleted(uid)
logger.debug('deleted %s', uid)
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 5/7] indexstore: exit on _flush() errors, work on tmpdir
Preparations for better index management.
- open_index() now can place the index on
a temporary location (ie: a tmpfs when
low on diskspace or at ENOSPC)
- don't set index_updated flag when on
a temporary location
- exit the process when hitting an error
in _flush(). This is normally caused by
running into ENOSPC during a session, and
is unrecoverable. dbus will respawn the
process, the new spawn has a chance at
cleanup, moving things to tmpfs.
Signed-off-by: Martin Langhoff
---
src/carquinyol/indexstore.py | 52 --
1 file changed, 40 insertions(+), 12 deletions(-)
diff --git a/src/carquinyol/indexstore.py b/src/carquinyol/indexstore.py
index dc721e7..a702014 100644
--- a/src/carquinyol/indexstore.py
+++ b/src/carquinyol/indexstore.py
@@ -235,12 +235,27 @@ class IndexStore(object):
self._database = None
self._flush_timeout = None
self._pending_writes = 0
-self._index_updated_path = os.path.join(
-layoutmanager.get_instance().get_root_path(), 'index_updated')
-
-def open_index(self):
-index_path = layoutmanager.get_instance().get_index_path()
-self._database = WritableDatabase(index_path, xapian.DB_CREATE_OR_OPEN)
+root_path=layoutmanager.get_instance().get_root_path()
+self._index_updated_path = os.path.join(root_path,
+'index_updated')
+self._std_index_path = layoutmanager.get_instance().get_index_path()
+ self._index_path = self._std_index_path
+
+def open_index(self, temp_path=False):
+# callers to open_index must be able to
+# handle an exception -- usually caused by
+# IO errors such as ENOSPC and retry putting
+# the index on a temp_path
+if temp_path:
+self._index_path = temp_path
+else:
+ self._index_path = self._std_index_path
+try:
+ self._database = WritableDatabase(self._index_path,
+ xapian.DB_CREATE_OR_OPEN)
+except Exception as e:
+ logging.error('Exception opening database')
+ raise
def close_index(self):
"""Close index database if it is open."""
@@ -248,14 +263,18 @@ class IndexStore(object):
return
self._flush(True)
-self._database = None
+try:
+# does Xapian write in its destructors?
+self._database = None
+except Exception as e:
+logging.error('Exception tearing down database')
+raise
def remove_index(self):
-index_path = layoutmanager.get_instance().get_index_path()
-if not os.path.exists(index_path):
+if not os.path.exists(self._index_path):
return
-for f in os.listdir(index_path):
-os.remove(os.path.join(index_path, f))
+for f in os.listdir(self._index_path):
+os.remove(os.path.join(self._index_path, f))
def contains(self, uid):
postings = self._database.postlist(_PREFIX_FULL_VALUE + \
@@ -347,6 +366,9 @@ class IndexStore(object):
index_updated = property(get_index_updated)
def _set_index_updated(self, index_updated):
+if self._std_index_path != self._index_path:
+ # operating from tmpfs
+ return True
if index_updated != self.index_updated:
if index_updated:
index_updated_file = open(self._index_updated_path, 'w')
@@ -374,7 +396,13 @@ class IndexStore(object):
self._pending_writes += 1
if force or self._pending_writes > _FLUSH_THRESHOLD:
-self._database.flush()
+try:
+self._database.flush()
+except Exception, e:
+logging.exception(e)
+logging.error("Exception during database.flush()")
+# bail out to trigger a reindex
+sys.exit(1)
self._pending_writes = 0
self._set_index_updated(True)
else:
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 3/7] metadatastore: store/change files on disk defensively #2317
- only delete metadata files for keys that are being removed
- only write files when the data changes
- write/replace metadata files atomically, to avoid corrupting
existing data in case of an error
With this patch, we no longer corrupt metadata when trying
to edit/update a ds entry with the system hitting ENOSPC.
Signed-off-by: Martin Langhoff
---
src/carquinyol/metadatastore.py | 51 ++-
1 file changed, 34 insertions(+), 17 deletions(-)
diff --git a/src/carquinyol/metadatastore.py b/src/carquinyol/metadatastore.py
index 5967017..52cc10f 100644
--- a/src/carquinyol/metadatastore.py
+++ b/src/carquinyol/metadatastore.py
@@ -14,27 +14,46 @@ class MetadataStore(object):
if not os.path.exists(metadata_path):
os.makedirs(metadata_path)
else:
+received_keys = metadata.keys()
for key in os.listdir(metadata_path):
-if key not in _INTERNAL_KEYS:
+if key not in _INTERNAL_KEYS and key not in received_keys:
os.remove(os.path.join(metadata_path, key))
metadata['uid'] = uid
for key, value in metadata.items():
+self._set_property(uid, key, value, md_path=metadata_path)
-# Hack to support activities that still pass properties named as
+def _set_property(self, uid, key, value, md_path=False):
+if not md_path:
+md_path = layoutmanager.get_instance().get_metadata_path(uid)
+# Hack to support activities that still pass properties named as
# for example title:text.
-if ':' in key:
-key = key.split(':', 1)[0]
-
-f = open(os.path.join(metadata_path, key), 'w')
-try:
-if isinstance(value, unicode):
-value = value.encode('utf-8')
-elif not isinstance(value, basestring):
-value = str(value)
-f.write(value)
-finally:
-f.close()
+if ':' in key:
+key = key.split(':', 1)[0]
+
+changed = True
+fpath = os.path.join(md_path, key)
+tpath = os.path.join(md_path, '.' + key)
+# FIXME: this codepath handles raw image data
+# str() is 8-bit clean right now, but
+# this won't last. We will need more explicit
+# handling of strings, int/floats vs raw data
+if isinstance(value, unicode):
+value = value.encode('utf-8')
+elif not isinstance(value, basestring):
+value = str(value)
+
+# avoid pointless writes; replace atomically
+if os.path.exists(fpath):
+stored_val = open(fpath, 'r').read()
+
+if stored_val == value:
+changed = False
+if changed:
+f = open(tpath, 'w')
+f.write(value)
+f.close()
+os.rename(tpath, fpath)
def retrieve(self, uid, properties=None):
metadata_path = layoutmanager.get_instance().get_metadata_path(uid)
@@ -55,6 +74,4 @@ class MetadataStore(object):
return None
def set_property(self, uid, key, value):
-metadata_path = layoutmanager.get_instance().get_metadata_path(uid)
-property_path = os.path.join(metadata_path, key)
-open(property_path, 'w').write(value)
+self._set_property(uid, key, value)
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 4/7] metadatareader: ignore ".hidden" files
Ignore any file with a filename starting with a '.' -- metadatastore
writes its tempfiles prefixed with a '.' .
Signed-off-by: Martin Langhoff
---
Note: this one needs a bit of testing.
---
src/carquinyol/metadatareader.c |7 ++-
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/src/carquinyol/metadatareader.c b/src/carquinyol/metadatareader.c
index d053a22..ee546f3 100644
--- a/src/carquinyol/metadatareader.c
+++ b/src/carquinyol/metadatareader.c
@@ -161,11 +161,8 @@ static PyObject *read_all_properties (const char
*metadata_path) {
dir_entry = readdir (dir_stream);
while (dir_entry != NULL) {
-// Skip . and ..
-if (dir_entry->d_name[0] == '.' &&
-(strlen (dir_entry->d_name) == 1 ||
-(dir_entry->d_name[1] == '.' &&
-strlen (dir_entry->d_name) == 2)))
+// Skip '.', '..', and any .hidden file
+if (dir_entry->d_name[0] == '.')
goto next_property;
if (add_property (metadata_path, dir_entry->d_name, dict, 1) == 0)
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 2/7] Remove invalid/corrupt on-disk entries #2317
When operating close to ENOSPC, we sometimes end up with
incomplete or invalid on-disk entries. So we prune these
during index rebuild.
Signed-off-by: Martin Langhoff
---
src/carquinyol/datastore.py |8
1 file changed, 8 insertions(+)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 1f098b4..33c658d 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -22,6 +22,7 @@ import logging
import uuid
import time
import os
+import shutil
import dbus
import dbus.service
@@ -181,6 +182,13 @@ class DataStore(dbus.service.Object):
self._index_store.store(uid, props)
except Exception:
logging.exception('Error processing %r', uid)
+logging.warn('Will attempt to delete corrupt entry %r',
uid)
+try:
+# self.delete(uid) only works on well-formed entries
:-/
+entry_path =
layoutmanager.get_instance().get_entry_path(uid)
+shutil.rmtree(entry_path)
+except Exception:
+logging.exception('Error deleting corrupt entry %r',
uid)
if not uids:
self._index_store.flush()
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 1/7] Add ds_clean flag to trigger index rebuilds #2095, #2317
This gives us more complete coverage of cases where ENOSPC or
other errors are hit when creating/updating datastore entries.
Without this patch, using the Journal on ENOSPC sometimes leads
to an "empty" Journal after restart. Datastore entries exist on
disk but are not in the index (Xapian DB).
With this patch, failure to complete create/update/delete forces
an index rebuild on restart.
Signed-off-by: Martin Langhoff
---
src/carquinyol/datastore.py | 31 ++-
1 file changed, 30 insertions(+), 1 deletion(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index de79500..1f098b4 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -66,6 +66,9 @@ class DataStore(dbus.service.Object):
self._index_store = IndexStore()
self._index_updating = False
+root_path = layoutmanager.get_instance().get_root_path()
+self._cleanflag = os.path.join(root_path, 'ds_clean')
+
if migrated:
self._rebuild_index()
return
@@ -83,6 +86,25 @@ class DataStore(dbus.service.Object):
elif not self._index_store.index_updated:
logging.debug('Index is not up-to-date, will update')
self._update_index()
+elif not os.path.exists(self._cleanflag):
+logging.debug('DS state is not clean, will update')
+self._update_index()
+self._mark_clean()
+
+
+def _mark_clean(self):
+try:
+ f = open(self._cleanflag, 'w')
+ os.fsync(f.fileno())
+ f.close()
+except Exception:
+ logging.exception("Could not mark the datastore clean")
+
+def _mark_dirty(self):
+try:
+os.remove(self._cleanflag)
+except:
+pass
def _open_layout(self):
"""Open layout manager, check version of data store on disk and
@@ -178,6 +200,7 @@ class DataStore(dbus.service.Object):
self.Created(uid)
self._optimizer.optimize(uid)
logger.debug('created %s', uid)
+self._mark_clean()
async_cb(uid)
@dbus.service.method(DS_DBUS_INTERFACE,
@@ -190,6 +213,8 @@ class DataStore(dbus.service.Object):
uid = str(uuid.uuid4())
logging.debug('datastore.create %r', uid)
+self._mark_dirty()
+
if not props.get('timestamp', ''):
props['timestamp'] = int(time.time())
@@ -232,6 +257,7 @@ class DataStore(dbus.service.Object):
self.Updated(uid)
self._optimizer.optimize(uid)
logger.debug('updated %s', uid)
+self._mark_clean()
async_cb()
@dbus.service.method(DS_DBUS_INTERFACE,
@@ -243,6 +269,8 @@ class DataStore(dbus.service.Object):
async_cb, async_err_cb):
logging.debug('datastore.update %r', uid)
+self._mark_dirty()
+
if not props.get('timestamp', ''):
props['timestamp'] = int(time.time())
@@ -393,6 +421,7 @@ class DataStore(dbus.service.Object):
in_signature='s',
out_signature='')
def delete(self, uid):
+self._mark_dirty()
self._optimizer.remove(uid)
self._index_store.delete(uid)
@@ -404,7 +433,7 @@ class DataStore(dbus.service.Object):
self.Deleted(uid)
logger.debug('deleted %s', uid)
-
+self._mark_clean()
@dbus.service.signal(DS_DBUS_INTERFACE, signature="s")
def Deleted(self, uid):
pass
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
In conditions of very low disk space available, or ENOSPC, the datastore misbehaves grossly: - sometimes corrupts the index, and won't rebuild it, "hiding" valid entries from the user - if you edit an entry metadata, it will corrupt the whole entry - attempts at deleting entries fail, so users cannot get out of ENOSPC This patchseries is an attempt at fixing these issues. The overall user experience is not perfect when we hit ENOSPC -- metadata edits are ignored, sometimes the datastore process goes away (so the Journal is not responsive until a new datastore process is spawned and ready). However, it behaves sanely: - testing this code I do not lose ds entries when editing them - I can remove entries correctly, even at ENOSPC, effectively freeing disk space - the index works even at ENOSPC, though it may take slower for the system to start in those cases where we do rebuild it - the index recovers gracefully on restart when there is free disk space Martin Langhoff (7): Add ds_clean flag to trigger index rebuilds #2095, #2317 Remove invalid/corrupt on-disk entries #2317 metadatastore: store/change files on disk defensively #2317 metadatareader: ignore ".hidden" files indexstore: exit on _flush() errors, work on tmpdir datastore: make delete() more reliable, log properly datastore: handle low-disk and ENOSPC conditions gracefully src/carquinyol/datastore.py | 142 +-- src/carquinyol/indexstore.py| 52 ++ src/carquinyol/metadatareader.c |7 +- src/carquinyol/metadatastore.py | 51 +- 4 files changed, 197 insertions(+), 55 deletions(-) -- 1.7.10.4 ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] Datastore vs ENOSPC patches, progress
Things I am working on - Add ds_clean flag to trigger index rebuilds - patch posted for review - Remove invalid corrupt disk entries - patch posted for review - When hitting ENOSPC, use a tmpfs for index database -- working on it... m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 2/2] Remove invalid/corrupt on-disk entries #2317
When operating close to ENOSPC, we sometimes end up with
incomplete or invalid on-disk entries. So we prune these
during index rebuild.
---
src/carquinyol/datastore.py |7 +++
1 file changed, 7 insertions(+)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index 6ede341..ee6bbb5 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -175,6 +175,13 @@ class DataStore(dbus.service.Object):
self._index_store.store(uid, props)
except Exception:
logging.exception('Error processing %r', uid)
+logging.warn('Will attempt to delete corrupt entry %r',
uid)
+try:
+# self.delete(uid) only works on well-formed entries
:-/
+entry_path =
layoutmanager.get_instance().get_entry_path(uid)
+os.removedirs(entry_path)
+except Exception:
+logging.exception('Error deleting corrupt entry %r',
uid)
if not uids:
self._index_store.flush()
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 1/2] Add ds_clean flag to trigger index rebuilds #2095, #2317
This gives us more complete coverage of cases where ENOSPC or
other errors are hit when creating/updating datastore entries.
Without this patch, using the Journal on ENOSPC sometimes leads
to an "empty" Journal after restart. Datastore entries exist on
disk but are not in the index (Xapian DB).
With this patch, failure to complete create/update/delete forces
an index rebuild on restart.
---
src/carquinyol/datastore.py | 25 -
1 file changed, 24 insertions(+), 1 deletion(-)
diff --git a/src/carquinyol/datastore.py b/src/carquinyol/datastore.py
index de79500..6ede341 100644
--- a/src/carquinyol/datastore.py
+++ b/src/carquinyol/datastore.py
@@ -77,12 +77,28 @@ class DataStore(dbus.service.Object):
self._rebuild_index()
return
+ds_root_path=layoutmanager.get_instance().get_root_path()
+self._cleanflag = os.path.join(ds_root_path, 'ds_clean')
+
if initiated:
logging.debug('Initiate datastore')
self._index_store.flush()
elif not self._index_store.index_updated:
logging.debug('Index is not up-to-date, will update')
self._update_index()
+elif not os.path.exists(self._cleanflag):
+logging.debug('DS state is not clean, will update')
+self._update_index()
+self._mark_clean()
+
+
+def _mark_clean(self):
+try:
+ f = open(self._cleanflag, 'w')
+ os.fsync(f.fileno())
+ f.close()
+except Exception:
+ logging.exception("Could not mark the datastore clean")
def _open_layout(self):
"""Open layout manager, check version of data store on disk and
@@ -178,6 +194,7 @@ class DataStore(dbus.service.Object):
self.Created(uid)
self._optimizer.optimize(uid)
logger.debug('created %s', uid)
+self._mark_clean()
async_cb(uid)
@dbus.service.method(DS_DBUS_INTERFACE,
@@ -190,6 +207,8 @@ class DataStore(dbus.service.Object):
uid = str(uuid.uuid4())
logging.debug('datastore.create %r', uid)
+os.remove(self._cleanflag)
+
if not props.get('timestamp', ''):
props['timestamp'] = int(time.time())
@@ -232,6 +251,7 @@ class DataStore(dbus.service.Object):
self.Updated(uid)
self._optimizer.optimize(uid)
logger.debug('updated %s', uid)
+self._mark_clean()
async_cb()
@dbus.service.method(DS_DBUS_INTERFACE,
@@ -243,6 +263,8 @@ class DataStore(dbus.service.Object):
async_cb, async_err_cb):
logging.debug('datastore.update %r', uid)
+os.remove(self._cleanflag)
+
if not props.get('timestamp', ''):
props['timestamp'] = int(time.time())
@@ -393,6 +415,7 @@ class DataStore(dbus.service.Object):
in_signature='s',
out_signature='')
def delete(self, uid):
+os.remove(self._cleanflag)
self._optimizer.remove(uid)
self._index_store.delete(uid)
@@ -404,7 +427,7 @@ class DataStore(dbus.service.Object):
self.Deleted(uid)
logger.debug('deleted %s', uid)
-
+self._mark_clean()
@dbus.service.signal(DS_DBUS_INTERFACE, signature="s")
def Deleted(self, uid):
pass
--
1.7.10.4
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Error page SL #3500
On Wed, Sep 19, 2012 at 6:54 AM, Manuel Kaufmann wrote:
> On Tue, Sep 18, 2012 at 10:58 PM, James Cameron wrote:
>> Neat. What method did you use to convert the image?
>
> Python... It is my best friend :)
>
import base64
base64.b64encode(open('browse-logo.png', 'r').read())
You have the file on-disk. It is inefficient to base64-encode it. Just
say img src="file://http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Debugging sugar-datastore (and similar dbus-connected services)
On Wed, Sep 19, 2012 at 4:54 PM, Gonzalo Odiard wrote: > You already have the line > export SUGAR_LOGGER_LEVEL=debug > uncommented in .sugar/debug, right? Yes. And with that, datastore.log is very chatty, which is good. But when the datastore code hits an unhandled exception, the exception does not get logged. DBus restarts the process transparently, but that isn't so useful :-/ In terms of progress, I have these tracks underway, - Add a "ds_clean" flag file, rm it on the start of every create/update/delete op, create it on completion. Complements "index_updated", and between the two we get good coverage. Missing either one, we get an index rebuild. This is done, and working well. - During index rebuild, delete incomplete entries. Not working well yet. With the two above, we can operate with the system at ENOSPC, and essentially we don't lose existing user data. New files fail to save, and metadata changes fail to get stored (so your edit, ie: a rename, is reverted in the journal, visibly). But we don't lose the existing Journal entries -- so behaviour is _almost_ civilized now :-) Behind the scenes, however, the datastore is dying and respawning. Which complicates things because what we are missing is to be able to delete entries while we are at ENOSPC. A delete is pretty straightforward, because we are removing files, and that succeeds even on ENOSPC. Except that we need to be able to maintain the xapian indexes. My plan right now is to catch the Xapian-is-hitting-ENOSPC exception, and trigger a Xapian index re-creation, putting the Xapian files in /tmp (which is a tmpfs in our builds and in Fedora upstream too). We'll make sure that the index_updated file on disk is removed so after a reboot the index is recreated once more. With Xapian index on /tmp, deleting Journal entries should just work. All dbus sockets are on tmpfs. However, to do all of the above with some sense that it's working correctly... yeah, I need to see the exceptions I hit. Right now I am guessing "what could be going wrong in this black box...?" which isn't a productive mode of life. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Debugging sugar-datastore (and similar dbus-connected services)
On Wed, Sep 19, 2012 at 4:15 PM, Martin Langhoff wrote: > I'm debugging odd situations w Sugar datastore, and I wonder whether > there is any tricks to debugging python programs that are run under > dbus. > > I can see the sugar-datastore PID growing steadily while I test, which > means that it's dying and respawning plenty, likely from unhandled > exceptions... Adding to /usr/bin/datastore-service # debugging logerr=open(os.path.join(log_dir, 'datastore.err'), 'w') sys.stderr=logerr does not help in the least. The file is created, but nothing is ever written there. Any hints? m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] Debugging sugar-datastore (and similar dbus-connected services)
I'm debugging odd situations w Sugar datastore, and I wonder whether there is any tricks to debugging python programs that are run under dbus. I can see the sugar-datastore PID growing steadily while I test, which means that it's dying and respawning plenty, likely from unhandled exceptions... thanks, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Adventures in the land of ENOSPC
On Sat, Sep 15, 2012 at 8:03 AM, Sascha Silbe wrote: > Martin Langhoff writes: > >> Manuel Kaufmann has been looking at SL#394, and looking at the bug >> report, it struck me that it was reported backwards. I would have >> written: "I filled up my disk and it knocked the system out, Sugar >> would not start again, etc. Oh, btw, it was with Browse". > > I agree. Every single mishandling of ENOSPC is a bug in the affected > piece of software and should be fixed. However, there's a lot of broken In our case, 99% of the time is the datastore. And when it borks, it messes up its indexes so it _has_ the files and the metadata, but will show you an empty listing for the Journal. > Similarly, the data store should take care to leave some free The datastore needs to handle ENOSPC correctly. That is much much deeper than what we can do activity-side. > Sounds like SL#2317 [1], as Gonzalo already pointed out. Indeed, and the problem is that index_updated remains on disk all the time. I am causing ENOSPC artificially, and creating or updating journal entries against ENOSPC and... index_updated is never removed. BTW, to test this I am putting sugar's own logs on tmpfs, which you can do with a file like this: # cat /etc/statetab.d/olpc.debug /home/olpc/.sugar/default/logs cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Not enough space adventure
On Tue, Sep 18, 2012 at 8:42 AM, Manuel Kaufmann wrote: > We where discussing about this last week[1] and we found the "root" > issue of this problem: Sugar is not handling ENOSPC error. This could > cause some problems at boot time when the XO is restarted, but as we > discussed[2], Linux has made some improvement on this side and it > seems that it recovers without problem (we need more testing here, I > think). Thanks for diving into this! > So, there are different problems to manage here: > 1. What are we going to do when ENOSPC is reached? I did some work lastweek, and plan to hack on it tomorrow. > 2. How are we going to avoid ENOSCP? ... > - remove the annoying check of free space I hope that's not completely removed (I see Gonzalo's email...). It is not very effective preventing you from using activities, but it does prompt users to do something (remove stuff?). BTW, 50MB may be too high, some units with 2GB storage, installing modern builds, end up with 150MB free total :-/ > - create a signal inside sugar-toolkit-gtk3 datastore that is emitted > when free space is behind > "sugar3.datastore.datastore.SPACE_THRESHOLD". This check is done every > time a model is updated or created by the datastore. A model? Maybe my lingo is a bit stale... a datastore entry you mean? > - this signal can be connected from every activity that wants to > handle this situation. For example in Browse we are cancelling the > download in progress when we get that signal So there is a bit of confusion here... - When we start the download, if we get Content-Length, we compare it with disk space available, or with disk space available minus datastore.SPACE_THRESHOLD? - Related: can we ask the datastore to tell us its SPACE_THRESHOLD, so we don't hardcode it? Ah, I see you have a helper function. Personally, I would prefer to preserve a relatively high threshold (lower than 50MB, but say, 10MB) for the _warnings_. But still allow a download to complete. I would only cancel a download in Browse if will leave us with <1MB disk space. To put it in other words, we are investing too much effort in... warnings. And we are mixing a high threshold that is good for a warning, with something that disrupts an operation (cancel download). You may be downloading an activity that allows you to delete stuff :-) Finally, we seem to be missing the tmpfile cleanup related to incomplete downloads. That can invisibly "eat" a ton of disk space in a way that the end user cannot cleanup... cheers, m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] Problem downloading a lease.sig file on an XO
On Wed, Sep 12, 2012 at 9:22 PM, Samuel Greenfeld wrote: >> 1. Is there a security problem/concern with having our project's lease.sig >> file publicly available? (we only generate activations for non-stolen XOs) Should be ok, as long as you're comfortable with the validity period (ie: not too long!). >> 2. Why is the XO adding this .asc extension or how can it be avoided? As Sam mentions, it is MIME-Types. Perhaps this helps: set the mime types in your webserver so that .sig files are served as application/octet-stream (something that I believe sugar will not mess with). You can check what mimetype is being set on a transfer with wget -- during download, it'll print a message about Length, that same line shows the mime-type. If that doesn't help, let us know. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] Adventures in the land of ENOSPC
Manuel Kaufmann has been looking at SL#394, and looking at the bug report, it struck me that it was reported backwards. I would have written: "I filled up my disk and it knocked the system out, Sugar would not start again, etc. Oh, btw, it was with Browse". Maybe I'm the only one who thinks this way. So I've gone and looked at what happens with a current Sugar and OS in that case (the original repot is very old, and so many things have changed...). I tested with 12.1.0. I artificially filled up / with dd until barely 1MB was remaining, then experimented with filling it up completely, then leaving a few K free and saving stuff. The good: - During the sugar session, activities politely complained that they could not save their session when closing, nothing crashed. - The Journal did sometimes pop up the warning, but it was trivial to click ok and continue. - I could perform Journal ops, deleting some existing Journal entries. - Shutdown/restart worked correctly - The OS and Sugar came up correctly on the next boot The bad: - Next boot, my Journal was gone. Actually, it's not gone, I can see the files in ~/.sugar/default/datastore, but it is corrupt or the Journal thread died, so it won't display in the Journal UI. If you are really out of disk space, you need to see them in the journal to delete them! - Freeing up some space, I can start using the journal again, and new entries survive a reboot. However, on-disk I see both old and new entries in the 'datastore' dir. I'll try to make time to understand wth is going on with the journal here. IIRC, we used to have a trigger to "reindex" the journal when the indexes seemed corrupt. That is not kicking in -- it would be helpful if it did :-) cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Wed, Sep 12, 2012 at 8:24 AM, Manuel Kaufmann wrote: > On Wed, Sep 12, 2012 at 9:21 AM, Martin Langhoff > wrote: >> That's very good handling on webkit's side. And does it remove the >> file? IOWs, when you get the signal, what do you see in the tmpfs? Is >> there a file filling it, or has the file been removed? > > Oh, sorry. I forgot to mention this. The file is still there with the > size that webkit could get (in my case 904kb). Webkit didn't remove > it. Hmmm. Ok, so it's leaving it up to the download manager code (our browse code) to handle the remains. AIUI, Webkit will be using the filename (or filename template) we passed it, so we know enough to find the file and remove it in that case. We should :-) cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Wed, Sep 12, 2012 at 8:20 AM, Manuel Kaufmann wrote: > OK. I was taking a look at the datastore source code and I understood > that the file is copied, using the traditional way and when the copy > finishes the source file is unlinked. Ouch! I haven't reviewed the code (have to run some personal errands today, can't dig into it as I'd like), but if you are correct, there are two worries - Downloading a 100MB file needs >200MB+ free. - All that passing around the file to update the percentage bar is causing pointless file copies -- if you are downloading a large file, it is pretty screwed. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Wed, Sep 12, 2012 at 7:50 AM, Manuel Kaufmann wrote: > I'm attaching an example that downloads a file with WebKit. > > I tried it setting up a tmpfs with 1Mb as you suggested and aftert > that, I ran this script. WebKit tells us about the insufficient space > on the disk by raising the "error" signal. I catched it and I printed > the arguments of the signal: That's very good handling on webkit's side. And does it remove the file? IOWs, when you get the signal, what do you see in the tmpfs? Is there a file filling it, or has the file been removed? cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Tue, Sep 11, 2012 at 9:07 PM, Manuel Kaufmann wrote:
> This is what I got:
>
> [olpc@xo-07-30-50 ~]$ mkdir /tmp/foo
> [olpc@xo-07-30-50 ~]$ sudo mount -t tmpfs -o size=1M none /tmp/foo
> [olpc@xo-07-30-50 ~]$ python
> Python 2.7.3 (default, Apr 30 2012, 23:07:00)
> [GCC 4.7.0 20120416 (Red Hat 4.7.0-2)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
f = open('/tmp/foo/cl.txt', 'w')
f.write('w' * 200 ** 2)
> (...)
f.write('w' * 2000)
> Traceback (most recent call last):
> File "", line 1, in
> IOError: [Errno 28] No space left on device
I guess that's what we'll want to grep for in the sugar datastore code ;-)
Hmmm, reading the code in download.py. AIUI Browse lets webkit run the
downloading (I cannot see any write() calls around). When we say
self._download.set_destination_uri('file://' + self._dest_path)
self._download.start()
we pass the ball to webkit. So in this case, we most of the writing is
being done in webkit. I do wonder what happens if webkit gets ENOSPC.
Does it manage to report it to Browse? Does it cleanup the partial
tmpfile? If not, should we file a bug upstream?
Also, IIRC, when you give a file to the datastore, the datastore does
not make a copy. It should make a hardlink to it. I hope the current
datastore is still sane in that regard. Still, if you are extremely
tight for space the datastore saves metadata, which could hit ENOSPC.
So I see 3 places where we could run out of space
- Empty tmpfile setup, extremely unlikely; you may want to implement
a try/except for consistency and elegance and good karma...
- During the webkit download -- most probable stage where you'll hit
it. How does Webkit behave? Does Browse need to do anything?
- During datastore.write() -- I truly believe that the
datastore.write() call doesn't literally make a copy of the file --
should just create/update the hardlink and associated metadata
(including progress bar). It is not very likely that we'll run out of
space during this call, but it should be handled, I believe.
Does that make sense? TBH, my datastore knowledge is extremely rusty.
It has been a long time...
m
--
martin.langh...@gmail.com
mar...@laptop.org -- Software Architect - OLPC
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Tue, Sep 11, 2012 at 4:32 PM, Manuel Kaufmann wrote: > I didn't find an exception raised by Sugar when this happens. The > check is done[1] when a new Model is created[2] or updated[3] inside > the Journal, and if there are less than 50Mb a ModelAlert is shown but > no exception is raised. Maybe we can write the portion of code here to > raise that exception before show the ModelAlert and if it's not > managed we can call the ModelAlert. > > [1] sugar/src/jarabe/journal/journalactivity.py L336 > [2] sugar/src/jarabe/journal/journalactivity.py L274 > [3] sugar/src/jarabe/journal/journalactivity.py L265 Forget about check_available_space() and pretty messages to the user. The real problem is hitting out-of-disk space. Can you run a test that bypasses this code so it really gets to ENOSPC? How does ENOSPC appear in Python? Py documentation is pretty incomplete: http://docs.python.org/library/exceptions.html#exceptions.IOError This says IOError 28 http://stackoverflow.com/questions/6998083/python-causing-ioerror-errno-28-no-space-left-on-device-results-32766-h I guess you'll have to try yourself. Try mkdir /tmp/foo sudo mount -t tmpfs -o size=1M none /tmp/foo Now write there to a file in /tmp/foo to see what happens, exactly. > The file is downloaded in a temporary directory and then copied to the > Journal: > > temp_path = os.path.join(activity.get_activity_root(), 'instance') > fd, self._dest_path = tempfile.mkstemp(dir=temp_path, > suffix=download.get_suggested_filename(), > prefix='tmp') I guess activity.get_activity_root() is in the main FS, not in a tmpfs. Luckily. > That is something like (I guess): > ~/.sugar/default/org.laptop.WebActivity/instance Ah, cool. So it's in the same FS. > > When the download finishes: > > (...) > self.dl_jobject = datastore.create() > (...) > datastore.write(self.dl_jobject, > transfer_ownership=True, > reply_handler=self.__internal_save_cb, > error_handler=self.__internal_error_cb, > timeout=360) > (...) Oh, that expects a callback. If there is an exception, maybe we don't get it on this side? :-( > def __progress_change_cb(self, download, something): > progress = self._download.get_progress() > self.dl_jobject.metadata['progress'] = str(int(progress * 100)) > datastore.write(self.dl_jobject) > > so, we are updating a model inside the datastore... If we had the > ENOSPC exception we could manage it there. I insist this goes in a > different ticket and a different patch. Being callback, its is probably going over dbus, so, ummm. m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Tue, Sep 11, 2012 at 9:01 AM, Manuel Kaufmann wrote: > If we are going to use this approach, I have to change the logic about > "where" I should check this. +200. I hadn't realized this. The main two things you absolutely need are - check at the start of the download that it'll fit (with X room to spare) - make damn sure that we handle ENOSPC if the disk does fill up Checking in the progress bar... it's optional. Up to you if you want to check there _as well_. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Mon, Sep 10, 2012 at 6:39 PM, James Cameron wrote: > Some users pay for data by volume. I don't think that cancelling a > download is the right approach. It should be paused, giving the user > a chance to clean up. Rather complex -- I haven't seen any webbrowser correctly implement that correctly. > Also, I agree with Martin that additional use of Content-Length > headers can be made, to warn the user against trying a download larger > than available space. Thanks > I disagree with Bernie in #394 comment 5 ... because the > Content-Length of all concurrent downloads can be used in a > calculation. We agree in disagreeing with Bernie, but no, trying to have a "download controller" pooling download information is another big ball of complexity I would not want to get into. Specially since it is ineffective -- _any other program in the system may be pushing you to ENOSPC_. The kernel acts as the final FS controller. What we _must_ have is solid recovery from ENOSPC. I think datastore has it (that's worth a test :-) ) and if datastore has it, then Browse can handle the exception gracefully with a small amount of coding effort... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Mon, Sep 10, 2012 at 6:02 PM, Manuel Kaufmann wrote: > On Mon, Sep 10, 2012 at 9:49 PM, Martin Langhoff > wrote: >> Perhaps it is a stupid question but... in many HTTP requests you get >> the size of the transfer in the HTTP headers, in the form of the >> Content-Length header. > > I didn't follow this approach because of this comment: > http://bugs.sugarlabs.org/ticket/394#comment:5 We address _that_ possibiliy (concurrent processes using up that disk space) with the error handling I was proposing later in my comments. So, putting that aside for a moment why a fixed guess is better than the actual content-length? It does make sense to declare a minimum buffer, but you have good reliably info in content-length, use it! >> And then, you need the most clear step: if the write fails, and the >> error / exception tells you it's ENOSPC, you delete the object. > > Will Sugar recover immediately after getting ENOSPC and deleting the object? Dunno. I hope so. Force the bug, see what you get. If things are reasonably ok, you'll get an exception in the Sugar code, hopefully the file is removed automatically from the Journal. So if all is good, you are expected to handle the exception in Browse. In terms of the Journal/datastore, you'd get the same situation copying from a USB stick -- try a file that is too large to fit in the free space you have... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [DESIGN] set zoom to initial value in Browse
On Mon, Sep 10, 2012 at 10:07 AM, Gonzalo Odiard wrote: > "Actual size" is used in Read too. http://imgs.xkcd.com/store/imgs/actual_size_200.png m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Mon, Sep 10, 2012 at 5:49 PM, Martin Langhoff wrote: > Perhaps it is a stupid question but... in many HTTP requests you get s/question/suggestion/ m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH Browse] Cancel a download if space is very tight SL #394
On Mon, Sep 10, 2012 at 5:33 PM, Manuel Kaufmann wrote: > If there are less than 50Mb free on the Hard Disk the downloading > process is canceled and this is informed to the user via an Alert. Perhaps it is a stupid question but... in many HTTP requests you get the size of the transfer in the HTTP headers, in the form of the Content-Length header. This is specially true with downloads of large files (less true for dynamic content like PHP-generated pages). > +# > http://stackoverflow.com/questions/787776/find-free-disk-space-in-python-on-os-x > +def free_space(path): Do you really need that comment? Doing s.f_bavail * s.f_frsize is just common sense. > +global k > +s = os.statvfs(path) > +return (s.f_bavail * s.f_frsize) / 1024 / 1024 Points for using f_bavail instead of bfree :-) Don't divide it yet. Return bytes -- only convert to MB if you are displaying to users. You are going to need bytes to compare with content-length which is in bytes too. > progress = self._download.get_progress() > -self.dl_jobject.metadata['progress'] = str(int(progress * 100)) Here, get the content-length... and compare with the free bytes (plus some padding, say, 15%). Maybe we still want to preserve a minimum disk space. For example: don't let a download put you below 50MB or 10MB free -- in that case do the multiplication in a constant definition. MIN_DISKFREE_AFTER_DOWNLOAD = 50 * 1024 * 1024 If the request does not have content-length, it is sane to assume it will be small (dynamically generated content) so that padding you preserved is very likely to fit it. And then, you need the most clear step: if the write fails, and the error / exception tells you it's ENOSPC, you delete the object. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
