Scaling issue
Hi everyone, I've just run into problems with fetching all articles belonging to a specific feed through the link class below. (class +ArFeLink +Entity) (rel article (+Aux +Ref +Link) (feed) NIL (+Article)) (rel feed (+Ref +Link) NIL (+Feed)) (collect 'feed '+ArFeLink Obj Obj 'article) takes forever (2 mins) I need it to take something like maximum 2 seconds... Can this be fixed by adding some index or key or do I need make this part of the DB distributed and chopped up so I can run this in parallel? /Henrik
Re: Changing db structures?
This can't be right, I must've meant (rebuild 29 'article '+ArFeLink) ? On Thu, Jan 21, 2010 at 4:49 PM, Henrik Sarvell wrote: > After some discussion on IRC I did the following: > > (rebuild 29 '+ArFeLink 'article) > > Where 29 is the number of the database file in question and 'article is the > relation. It had (+Ref +Link), it now has (+Aux +Ref +Link). > > So far I haven't noticed any problems. > > > > On Wed, Jan 20, 2010 at 8:13 AM, Alexander Burger wrote: > >> Hi Henrik, >> >> > and it was on IRC so better get it out here. What exactly happens if I >> for >> > instance add an +Aux relation, will the DB engine recognize that it's >> new >> > and create the index to include all concerned objects, or will it simply >> >> When the DB model (the "er.l" file) is changed, it is not automatically >> detected at runtime. This would be difficult, as it might require a scan >> of the whole database. Also, a program is allowed to create arbitrary >> data, possibly outside the DB model, so such an automatism would be >> dangerous. >> >> The programmer must understand the implications of a change, and run an >> explicit rebuild-script. Adding or changing an index (like adding an >> +Aux relation) is one of the most typical cases. >> >> >> > come in to effect on all objects created from now on or will it corrupt >> the >> > DB? If I recall correctly it will simply work from now on but existing >> > objects won't be included in the index. >> >> Yes. The DB is not really corrupted in such a case, but the affected >> objects may not be found at searches. >> >> >> > And if I recall correctly I believe there were also some tools to >> rebuild in >> > situations like this or? >> >> 'rebuild' (in "lib/too.l") can be used for that. >> >> Sometimes, such modifications can be rather complex, and involve changes >> in the structures or types of objects. I strongly recommend to test it >> first on a local copy of the database. >> >> It is tricky if not only an index, but the type of indexed data changes. >> For example, I had the case last week that the "Land" property of >> several object classes changed from a plain string >> >> (+Ref +String) >> >> to an entity >> >> (+Ref +Link) NIL (+Land)) >> >> Then simply changing the string in all objects to a '+Link' does not >> work, because the new index has a different structure. Therefore, >> removing the old string must happen in the old model, while setting the >> +Link to the land object must happen in the new model. I achieved this >> by switching the class of that relation daemon with 'bind' during each >> change: >> >> (for (This ...) # Iterate all involved objects >> (let Str (: lnd) # Keep the string value >> (bind # Temporarily set old relation classes >>(list >> (cons (get '+Cls1 'lnd) '(+Need +Sn +Idx +String)) >> (cons (get '+Cls2 'lnd) '(+Ref +String)) >> ... ) >>(put> This 'lnd NIL) ) # Clear old value and index >> (put> This 'lnd # Set new (entity) value and build index >>(request '(+Land) ... Str ..) ) ) ) >> >> This is just an example how complex such changes can be. But, on the >> other hand, there is almost no limit on what can be done. >> >> Cheers, >> - Alex >> -- >> UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe >> > >
Re: Wiki at home.picolisp.com
On Sat, Apr 10, 2010 at 01:09:28PM +0200, Alexander Burger wrote: > > And versioning? And maybe diff view? > > That is there already, if you go to the DB admin part. A little more explanation: As you see (if you log in), there are two "Roles" (objects holding access permissions). New users who create an account, and get their password mailed, have the "Member" role, with limited access rights. They can browse documents and histories, but can only change their own user data. A "Member" can be changed to "Admin" only manually by an existing admin. If you want that, please contact one of the admins (can be seen in the user management) via mail or IRC to do that for you. Cheers, Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Wiki at home.picolisp.com
Hi Tomas, > Hi Jon, > >> And, will it be a way to select a language other than English? > > you mean like this? > > http://translate.google.com/translate?hl=en&ie=UTF-8&sl=en&tl=cs&u=http://home.picolisp.com/&prev=_t > > :-) > > Tomas Not exactly ;-), but I'm quite impressed by the quality of Google's Norwegian translation. And I'm equally surprised that "Translation from English into English is not supported." ;-) One Lisp expression that Google didn't handle successfully, however, was "CAR part" ... As for support for other languages in picolisp.com, I think we can do very well without it. /Jon -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: VizReader
But how is it different or better compared to emacs/gnus with nnrss and > nnshimbun backends? > > And do you really find web interface more usable? > Probably not different or better but I want this system to be accessible irrespective of hardware and location, I want to be able to login on a library computer running Windows to check out what's happening. For me usability nowadays is mainly a function of response time and then the remote server is the important thing, not the client. What does special feeds mean? In what sense is Twitter special? Is it > that you need to scrape web pages and convert them to RSS yourself? > It requires special handling to get at the content, it might conform to the standards but it still requires special handling. It's also special in the sense that it will vary on every fetch if you follow enough people. I follow 1000+ persons and am now getting realtime coverage of the situation in Thailand from 50+ sources, VizReader updates the feed every 60 seconds and shows only new posts in the GUI, that's how Twitter could be viewed as a special feed. Does vizreader create these meta feeds or does it rely on existing meta > feeds like the yahoo link above? > It relies on them, they conform to the specs and can be read just like original feeds, and there are probably more meta/aggregational feeds than original feeds these days. I find that the biggest problem (aside from having enough time for > reading the feeds) is that lots of sites don't publish feeds. Solving > this somehow would be great but probably impossible. > Not really a problem, you could probably use Google (or some other entity that has a lot of resources) to generate a meta feed for you that will give you the updates to the site as soon as it changes even though it doesn't have its own feed. The problem is however a pretty hard one to solve otherwise as it would involve having VizReader poll the site in question on a pretty frequent basis to fetch new content, I need more hardware to do that kind of stuff :) On Sat, Apr 10, 2010 at 12:54 PM, Tomas Hlavaty wrote: > Hi Henrik, > > I like the ui, looks neat! > > > The main reason for doing this (as supposed to getting email alerts > > when a site has updated from for instance Google) is that I have > > everything in one place and I can see what I've already read and > > what's new. In short, it helps me make cut through to exactly the > > information I want and enables me to get to it effortlessly, I > > couldn't manage without an RSS reader anymore. > > I agree, I find RSS esential these days too. > > > Since the application is that important I felt like I needed to create > > it myself in order to have total control over it. > > I see;-) > > But how is it different or better compared to emacs/gnus with nnrss and > nnshimbun backends? > > And do you really find web interface more usable? > > > And then there are special feeds, like for instance Twitter where I > > get most of my real time information from various agents that tweet > > news, like the NY Times science twitter account for instance: > > http://twitter.com/nytimesscience > > What does special feeds mean? In what sense is Twitter special? Is it > that you need to scrape web pages and convert them to RSS yourself? > > > Finally there are the meta feeds which are aggregations, I have a few > > of these but not many, for instance the Yahoo LISP pipe: > > http://pipes.yahoo.com/pipes/pipe.info?_id=3PHwctj52xGg02vB6kjTQA > > Does vizreader create these meta feeds or does it rely on existing meta > feeds like the yahoo link above? > > > Finding the feeds are usually not hard, often sites have explicit > > icons or links you can click to go to the url of the feed/xml, when > > you have it you simply add it in your reader. Firefox also displays > > the feed icon in the address field if it detects a tag that for > > instance looks like this in the content: > > I find that the biggest problem (aside from having enough time for > reading the feeds) is that lots of sites don't publish feeds. Solving > this somehow would be great but probably impossible. > > Cheers, > > Tomas > -- > UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe >
Re: picolisp-mode download
Hi TC, > It should work now. However, you can grab the latest stable version > from the devel picoLisp.tgz too. It's located in the lib/el folder. found it, thanks! > I haven't touched it for a while due to exams and other duties, so I > still haven't implemented documentation lookup. I have some ongoing > work regarding autocompletion but it's far from complete (and to be > honest, I rarely use it). doc lookup would be great. And maybe something like eldoc-mode? > If you have any problems, let me know. I like the autoindenting/formatting feature a lot. It doesn't format exactly as Alex does, but it's very close and probably wouldn't be hard to fix. I can't remember where I saw that at the moment, but I think it was something with indenting 'case' or so. Thank you, Tomas -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Wiki at home.picolisp.com
Hi Tomas, > I am confused, we have several wikis now? Unfortunately (or fortunately?), yes. > - http://logand.com/picoWiki/ > - http://www.picolisp.org/ > - http://picolisp.com/5000/-2.html The official entry point for the latter is "home.picolisp.com", with a parallel direct entry to the DB maintenance at "wiki.picolisp.com". We (or at least I) were dissatisfied with picolisp.org (wikidot). It is not open, and too slow, and the advertisements are annoying. We discussed it here and in IRC, so picolisp.com is intended as a long term replacement of picolisp.org. > Can I wish for the RSS feature? Great idea. I have no idea, though. Perhaps some input from Henrik? > And versioning? And maybe diff view? That is there already, if you go to the DB admin part. You could also study the sources (wiki.tgz). > The new wikis seem very complex to me:-( After I wrote my original > picoWiki, I found that even that was way too complex and came to the > conclusion that a wiki for a small community should be something like > the ultimate and first wiki ever http://c2.com/cgi/wiki which I find > brilliant both regarding content and implementation;-) I have not much experience and overview. I just wanted something fully controllable (the picoLisp way :) and simple, which is usable without much trouble. > I must admit that I find this account feature unnecessary overkill:-( Hmm, perhaps, but I don't want to give full access to anybody. As we have so many wikis now, we can experiment with several strategies ;-) > Also markup seems too far from plain text. Armadillo and Javier seem to be of the opposite opinion. For them, my markup is too simple. Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: VizReader
Hi Henrik, I like the ui, looks neat! > The main reason for doing this (as supposed to getting email alerts > when a site has updated from for instance Google) is that I have > everything in one place and I can see what I've already read and > what's new. In short, it helps me make cut through to exactly the > information I want and enables me to get to it effortlessly, I > couldn't manage without an RSS reader anymore. I agree, I find RSS esential these days too. > Since the application is that important I felt like I needed to create > it myself in order to have total control over it. I see;-) But how is it different or better compared to emacs/gnus with nnrss and nnshimbun backends? And do you really find web interface more usable? > And then there are special feeds, like for instance Twitter where I > get most of my real time information from various agents that tweet > news, like the NY Times science twitter account for instance: > http://twitter.com/nytimesscience What does special feeds mean? In what sense is Twitter special? Is it that you need to scrape web pages and convert them to RSS yourself? > Finally there are the meta feeds which are aggregations, I have a few > of these but not many, for instance the Yahoo LISP pipe: > http://pipes.yahoo.com/pipes/pipe.info?_id=3PHwctj52xGg02vB6kjTQA Does vizreader create these meta feeds or does it rely on existing meta feeds like the yahoo link above? > Finding the feeds are usually not hard, often sites have explicit > icons or links you can click to go to the url of the feed/xml, when > you have it you simply add it in your reader. Firefox also displays > the feed icon in the address field if it detects a tag that for > instance looks like this in the content: I find that the biggest problem (aside from having enough time for reading the feeds) is that lots of sites don't publish feeds. Solving this somehow would be great but probably impossible. Cheers, Tomas -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Wiki at home.picolisp.com
Hi Tomas, > you mean like this? > http://translate.google.com/translate?hl=en&ie=UTF-8&sl=en&tl=cs&u=http://home.picolisp.com/&prev=_t Hm, well, I take this as another vote not to introduce language selection (I tried the german version of that ;-) Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Wiki at home.picolisp.com
Hi Alex, > since yesterday, the first version of the wiki is running. The entry > point is at "http://home.picolisp.com";. I am confused, we have several wikis now? - http://logand.com/picoWiki/ - http://www.picolisp.org/ - http://picolisp.com/5000/-2.html > It uses a minimal markup syntax so far, but TC and Javier are working > on an extended version. As you can see from the sources, the rest of > the system is also _really_ minimal. It provides just for the most > basic functionality. But I wanted to have something up and running. Can I wish for the RSS feature? And versioning? And maybe diff view? The new wikis seem very complex to me:-( After I wrote my original picoWiki, I found that even that was way too complex and came to the conclusion that a wiki for a small community should be something like the ultimate and first wiki ever http://c2.com/cgi/wiki which I find brilliant both regarding content and implementation;-) > All members who have a test login since the first experiments, can log > in with their previous name/password and edit changes or replies. New > users can click the "Log in" link, and then the "Create account" button. > They'll be mailed an automatically generated password. I must admit that I find this account feature unnecessary overkill:-( Also markup seems too far from plain text. On the positive side, it's great to see some community effort and movement in the right direction;-) Cheers, Tomas -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Wiki at home.picolisp.com
Hi Jon, > And, will it be a way to select a language other than English? you mean like this? http://translate.google.com/translate?hl=en&ie=UTF-8&sl=en&tl=cs&u=http://home.picolisp.com/&prev=_t :-) Tomas -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re[2]: Wiki at home.picolisp.com
Hi all! Alex, IMHO there is no need to switch on locale selection at the moment Best Regards, Mansur > > So what is the general opinion? Should I put back in the locale > selection? This would imply, however, that we also must prepare > translations in all languages for the wiki-specific texts. > -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
