Re: [wikireader] Rudimentary support for several wikis
Have you seen this on the git (http://wiki.github.com/wikireader/wikireader/structure-of-sd-card) Multiple Language Version * Programs are in the root (*.elf) * Forth related items (*.4th *.4mu forth.ini) are in the root * Fonts (*.bmf) are in the root * XXpedia subdirectories contain the data (wiki*.*) (XX = en, es, de, fr ... see wiki-app/wiki_info.c) * XXpedia/wiki.nls is the language specific messages file (plain text key=message) How already tested this ? -- Thomas HOCEDEZ ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: [wikireader] Rudimentary support for several wikis
Thomas HOCEDEZ wrote: Have you seen this on the git (http://wiki.github.com/wikireader/wikireader/structure-of-sd-card) Nope. I take it most of my work was useless … ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: [wikireader] Rudimentary support for several wikis
Actually that fits my needs very well. My support for several wikis was a rudimentary hack at best, to enable my real goal: using the wikireader as an ebook reader. *That* code was almost trivial to port, and can be found at git://gitorious.org/wikireader-ness/wikireader-ness2.git. I'll keep pushing there (mainly for backup), so if anyone is interested in having the entire project gutenberg library in their pocket … ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: [wikireader] Rudimentary support for several wikis
Le 26/01/2010 14:23, Tom Bachmann a écrit : Thomas HOCEDEZ wrote: Have you seen this on the git (http://wiki.github.com/wikireader/wikireader/structure-of-sd-card) Nope. I take it most of my work was useless … Did you think that maybe YOUR work causes this stuff to be developped ?! héhé ... OpenSource mysteries ;-) I'll keep watch your work. -- Thomas HOCEDEZ ___ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
Re: [wikireader] Rudimentary support for several wikis
Le 21/01/2010 19:07, Tom Bachmann a écrit :
Alright. The latest commit has a ChangeCollection script. Use it like this:
ChangeCollection.py --from=none --to=1 --prefix=/path/to/image/pedia
--dat-offset=${next free dat}
where ${next free dat} is the first unused number in the .dat namespace
of the english wiki. This will take a long while (it has to decompress
and recompress all articles!), but it is probably faster than
re-rendering everything (on my laptop it takes about 40 seconds to patch
1000 articles).
Next copy the pedia.idx, pedia.pfx, pedia.fnd, pedia.hsh, pedia?.dat of
the english wiki to your image, renaming to pedia0.idx, pedia0.pfx,
pedia0.fnd, pedia0.hsh (the pedia?.dat can keep their names). If you now
boot my kernel, you should be able to change between both wikis, as
described in my first post.
Please tell me if everything works as expected.
I have other Wikis to test with, (french + bosnian + lituanian) but I
still have two questions :
#1 : I can't find your ChangeCollection.py script : where did you hide
it ?
#2 : How can you find the last id in .dat file ? for me it is not
humanly readable...
#3 (extra bonus question) : Don't you think it would be possible to
automatize those steps ?
The last steps are very easy to do. ;-), I think I'll manage to do it.
Thanks for your nice job.
AstHrO
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community
Re: [wikireader] Rudimentary support for several wikis
Alright. The latest commit has a ChangeCollection script. Use it like this:
ChangeCollection.py --from=none --to=1 --prefix=/path/to/image/pedia
--dat-offset=${next free dat}
where ${next free dat} is the first unused number in the .dat namespace
of the english wiki. This will take a long while (it has to decompress
and recompress all articles!), but it is probably faster than
re-rendering everything (on my laptop it takes about 40 seconds to patch
1000 articles).
Next copy the pedia.idx, pedia.pfx, pedia.fnd, pedia.hsh, pedia?.dat of
the english wiki to your image, renaming to pedia0.idx, pedia0.pfx,
pedia0.fnd, pedia0.hsh (the pedia?.dat can keep their names). If you now
boot my kernel, you should be able to change between both wikis, as
described in my first post.
Please tell me if everything works as expected.
Thomas HOCEDEZ wrote:
It would be awesome !
I finished French Wiki last night, upload is in action. It will be
available before tonight on some mirors.
I'll post urls as soon as it is available.
Thomas
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community
Re: [wikireader] Rudimentary support for several wikis
Le 19/01/2010 16:33, Tom Bachmann a écrit :
I now registered to the list, since unregistered didn't seem to come
through and c...@thewikireader doesn't seem to respond. Possibly you
might recive this message more than once.
Original Message
Subject: [wikireader] Rudimentary support for several wikis
Date: Sun, 17 Jan 2010 00:56:53 +
From: Tom Bachmanntb...@cam.ac.uk
To: community@lists.openmoko.org
Hello,
first of all, please CC me since I'm not registered to the list.
Over the last few days I have been hacking together rudimentary support
for displaying several collections of data (e.g. wikis of different
languages) on the wikireader. This code is not yet ready to be
incorporated into the main repository (I think), and furthermore I don't
actually know if it complies with your ideas of simplicity.
HOWEVER, I would be very grateful to everyone who can test the code. I
don't yet have a real wikireader (i.e. I have been developing this on
the simulator; I will get one after sorting out my budget...) and I'm
worried that there might be problems related to e.g. the scarcity of
memory on the reader (how much ram has it installed?).
Here is what I did: basically, articles are now identified by their
index and by their collection id (the highest four bits of the 32bit
identifier). The .pfx, .fnd, .hsh and .idx files are replicated per
collection. The .dat files are just numbered consecutively (and
identified by the usual way). So if you have e.g. two collections, say
english and french wikipedia, then your image layout may look like this:
pedia0.idx pedia0.hsh pedia0.pfx pedia0.fnd
pedia1.idx pedia1.hsh pedia1.pfx pedia1.fnd
pedia0.dat pedia1.dat pedia2.dat pedia3.dat pedia4.dat
You cannot tell what articles are in what .dat files (in principle
articles from several wikis could be mixed in one file), but in practice
we might have pedia0-2.dat corresponding to the collection 0 (english
wiki) and pedia{3,4}.dat corresponding to collection 1 (french wiki).
The searching functionality etc is implemented in the wiki-app, the user
inteface is rather non-existent. As a hack for testing I'm statically
configuring the system to use two collections (identified 0 and 1) and I
added an invisible button to the upper right corner of the search menu
to switch between the collections (in the simulator you will see a
message). There seem to be some bugs in that button but it's really for
testing only.
In addition to implementing all that in the wiki-app, I modified the
render, index and combine programs. All take a new --coll-number
argument to identify the collection being worked on, and
ArticleRender.py has a new --dat-number argument to specify the .dat
file (--number only identifies the block for the .idx file).
The good news is, you can just re-use your primary collection (the one
identified by 0). The bad news is, all extra collections have to be
re-built. For a quick test, try
make DESTDIR=image WORKDIR=work \
XML_FILES=xml-file-samples/japanese_architects.xml \
COLL_NUMBER=1 DAT_NUMBER=${first unused index in .dat} iprch
make DESTDIR=image WORKDIR=work install
and then copy everything to your wikireader (or try sim4).
Again, it would be *greatly* appreciated if someone could build a large
second collection and try two real-life datasets on the wikireader.
All the code is at gitorious (just because I am already registered there
but not yet on github). To get it, do
git clone git://gitorious.org/wikireader-ness/wikireader-ness.git
Let me know what you think!
Thanks,
Tom
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community
It would be awesome !
I finished French Wiki last night, upload is in action. It will be
available before tonight on some mirors.
I'll post urls as soon as it is available.
Thomas
--
Thomas HOCEDEZ
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community
Re: [wikireader] Rudimentary support for several wikis
in the light of the awfully long render times for complete wikis, I
figure I should create a 'change collection number' script.
Thomas HOCEDEZ wrote:
Le 19/01/2010 16:33, Tom Bachmann a écrit :
I now registered to the list, since unregistered didn't seem to come
through and c...@thewikireader doesn't seem to respond. Possibly you
might recive this message more than once.
Original Message
Subject: [wikireader] Rudimentary support for several wikis
Date: Sun, 17 Jan 2010 00:56:53 +
From: Tom Bachmanntb...@cam.ac.uk
To: community@lists.openmoko.org
Hello,
first of all, please CC me since I'm not registered to the list.
Over the last few days I have been hacking together rudimentary support
for displaying several collections of data (e.g. wikis of different
languages) on the wikireader. This code is not yet ready to be
incorporated into the main repository (I think), and furthermore I don't
actually know if it complies with your ideas of simplicity.
HOWEVER, I would be very grateful to everyone who can test the code. I
don't yet have a real wikireader (i.e. I have been developing this on
the simulator; I will get one after sorting out my budget...) and I'm
worried that there might be problems related to e.g. the scarcity of
memory on the reader (how much ram has it installed?).
Here is what I did: basically, articles are now identified by their
index and by their collection id (the highest four bits of the 32bit
identifier). The .pfx, .fnd, .hsh and .idx files are replicated per
collection. The .dat files are just numbered consecutively (and
identified by the usual way). So if you have e.g. two collections, say
english and french wikipedia, then your image layout may look like this:
pedia0.idx pedia0.hsh pedia0.pfx pedia0.fnd
pedia1.idx pedia1.hsh pedia1.pfx pedia1.fnd
pedia0.dat pedia1.dat pedia2.dat pedia3.dat pedia4.dat
You cannot tell what articles are in what .dat files (in principle
articles from several wikis could be mixed in one file), but in practice
we might have pedia0-2.dat corresponding to the collection 0 (english
wiki) and pedia{3,4}.dat corresponding to collection 1 (french wiki).
The searching functionality etc is implemented in the wiki-app, the user
inteface is rather non-existent. As a hack for testing I'm statically
configuring the system to use two collections (identified 0 and 1) and I
added an invisible button to the upper right corner of the search menu
to switch between the collections (in the simulator you will see a
message). There seem to be some bugs in that button but it's really for
testing only.
In addition to implementing all that in the wiki-app, I modified the
render, index and combine programs. All take a new --coll-number
argument to identify the collection being worked on, and
ArticleRender.py has a new --dat-number argument to specify the .dat
file (--number only identifies the block for the .idx file).
The good news is, you can just re-use your primary collection (the one
identified by 0). The bad news is, all extra collections have to be
re-built. For a quick test, try
make DESTDIR=image WORKDIR=work \
XML_FILES=xml-file-samples/japanese_architects.xml \
COLL_NUMBER=1 DAT_NUMBER=${first unused index in .dat} iprch
make DESTDIR=image WORKDIR=work install
and then copy everything to your wikireader (or try sim4).
Again, it would be *greatly* appreciated if someone could build a large
second collection and try two real-life datasets on the wikireader.
All the code is at gitorious (just because I am already registered there
but not yet on github). To get it, do
git clone git://gitorious.org/wikireader-ness/wikireader-ness.git
Let me know what you think!
Thanks,
Tom
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community
It would be awesome !
I finished French Wiki last night, upload is in action. It will be
available before tonight on some mirors.
I'll post urls as soon as it is available.
Thomas
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community
[wikireader] Rudimentary support for several wikis
I now registered to the list, since unregistered didn't seem to come
through and c...@thewikireader doesn't seem to respond. Possibly you
might recive this message more than once.
Original Message
Subject: [wikireader] Rudimentary support for several wikis
Date: Sun, 17 Jan 2010 00:56:53 +
From: Tom Bachmann tb...@cam.ac.uk
To: community@lists.openmoko.org
Hello,
first of all, please CC me since I'm not registered to the list.
Over the last few days I have been hacking together rudimentary support
for displaying several collections of data (e.g. wikis of different
languages) on the wikireader. This code is not yet ready to be
incorporated into the main repository (I think), and furthermore I don't
actually know if it complies with your ideas of simplicity.
HOWEVER, I would be very grateful to everyone who can test the code. I
don't yet have a real wikireader (i.e. I have been developing this on
the simulator; I will get one after sorting out my budget...) and I'm
worried that there might be problems related to e.g. the scarcity of
memory on the reader (how much ram has it installed?).
Here is what I did: basically, articles are now identified by their
index and by their collection id (the highest four bits of the 32bit
identifier). The .pfx, .fnd, .hsh and .idx files are replicated per
collection. The .dat files are just numbered consecutively (and
identified by the usual way). So if you have e.g. two collections, say
english and french wikipedia, then your image layout may look like this:
pedia0.idx pedia0.hsh pedia0.pfx pedia0.fnd
pedia1.idx pedia1.hsh pedia1.pfx pedia1.fnd
pedia0.dat pedia1.dat pedia2.dat pedia3.dat pedia4.dat
You cannot tell what articles are in what .dat files (in principle
articles from several wikis could be mixed in one file), but in practice
we might have pedia0-2.dat corresponding to the collection 0 (english
wiki) and pedia{3,4}.dat corresponding to collection 1 (french wiki).
The searching functionality etc is implemented in the wiki-app, the user
inteface is rather non-existent. As a hack for testing I'm statically
configuring the system to use two collections (identified 0 and 1) and I
added an invisible button to the upper right corner of the search menu
to switch between the collections (in the simulator you will see a
message). There seem to be some bugs in that button but it's really for
testing only.
In addition to implementing all that in the wiki-app, I modified the
render, index and combine programs. All take a new --coll-number
argument to identify the collection being worked on, and
ArticleRender.py has a new --dat-number argument to specify the .dat
file (--number only identifies the block for the .idx file).
The good news is, you can just re-use your primary collection (the one
identified by 0). The bad news is, all extra collections have to be
re-built. For a quick test, try
make DESTDIR=image WORKDIR=work \
XML_FILES=xml-file-samples/japanese_architects.xml \
COLL_NUMBER=1 DAT_NUMBER=${first unused index in .dat} iprch
make DESTDIR=image WORKDIR=work install
and then copy everything to your wikireader (or try sim4).
Again, it would be *greatly* appreciated if someone could build a large
second collection and try two real-life datasets on the wikireader.
All the code is at gitorious (just because I am already registered there
but not yet on github). To get it, do
git clone git://gitorious.org/wikireader-ness/wikireader-ness.git
Let me know what you think!
Thanks,
Tom
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community
