[Offline-l] Re: Fwd: Affiliate notification about new user group - Kiwix Wikimedians

[Emmanuel Engelhart]

Hi Florence,

Thank you for the hint. Following yours and SJ remarks, I have put my 
own interrogations (and all of them go in the same direction).


We are missing the users of offline solutions in our UG offline, would 
be great if we achieve to change this at this occasion.


Regards
Kelson

On 20.12.23 01:25, Florence Devouard wrote:

Hello everyone

Dumi is seeking feedback from us regarding this new Nigerian based UG 
"Kiwix Wikimedians"


Any feedback from your side is welcome.

Florence



 Message transféré 
Sujet : Affiliate notification about new user group - Kiwix Wikimedians
Date :  Thu, 14 Dec 2023 11:25:28 +0200
De :Dumisani Ndubane 
Pour :  Florence Devouard , Sj 
Copie à : 	Affiliations Committee list for Wikimedia user group 
applications 




Dear Wikimedians for Offline Wikis User Group,

I'm writing on behalf of the Affiliations Committee to request your 
feedback on a proposed new user group.


As you probably know, Wikimedia user groups are intended to be simple 
and flexible affiliates which offer an initial recognition for 
qualifying groups to be able to operate with a brand name and logo use 
for local projects and events.[1]  User groups may come and go, or stay 
to thrive and grow as user groups, or sometimes develop into chapters or 
thematic organisations.


User groups do not have "ownership" or "exclusivity" of any territory, 
and we encourage many user groups to coexist within the same region. We 
aim to ensure peaceful coexistence by working with all new and existing 
groups to identify their planned activities and goals, to connect on 
shared movement goals, and to balance any potential overlaps with 
agreements for operating in shared spaces.


At this time, we are reaching out to let you know that there is a new 
group, Kiwix Wikimedians User Group [2], which has submitted an 
application for recognition as a Wikimedia User Group. We are requesting 
your feedback because this group seems to be working in some of the same 
community or thematic spaces as you are.  We want to hear any 
observations, concerns, or other thoughts that you might be able to 
offer regarding the group and its work so that the committee can 
consider them as it reviews this application.



Links:
[1] https://meta.wikimedia.org/wiki/Wikimedia_movement_affiliates
[2] https://meta.wikimedia.org/wiki/Kiwix_Wikimedians_User_Group


Regards,



Dumisani Ndubane

Snr Governance Committee Strategist

Wikimedia Foundation 





Dumisani Ndubane

Snr Governance Committee Strategist

Wikimedia Foundation 



___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org


--
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Re: Wikimania 2023 Program submission : Offline wiki

[Emmanuel Engelhart]

Hi Butch

This is meanwhile pretty on the TOP of my todo list :)

Kind regards
Kelson

On 20.03.23 19:15, Butch Bustria wrote:

My Fellow Wikimedians,

I highly encourage you to submit a program idea, such as demonstration, 
workshop for Wikimania 2023. It is expected that we will be having a 
higher attendance of emerging Wikimedia communities such as in 
Indonesia, Philippines and economically less developed Wikimedia 
communities such as in East Timor, Papua New Guinea, Pacific islands, as 
well as attendance of enthusiasts of the offline wiki in Singapore.


We also have the opportunity to connect to community members in remote 
locations as this catered to onsite and remote participation.


Program submission for Wikimania 2023 ends on *28 March*.

Information about this process is shown on the links below.

https://wikimania.wikimedia.org/wiki/2023:Program/Submissions 



https://wikimania.wikimedia.org/wiki/2023:Program/FAQ 



https://wikimania.wikimedia.org/wiki/2023:Program/Form_Questions 



If you have any questions, you can post a question on wiki 
 , telegram group 
(t.me/wikimaniachat ) or at 
wikima...@wikimedia.org 


We look forward to hearing from you on your participation in this event.

--
Sincerely,

*Butch Bustria*
Chair, Program Subcommittee
Event lead, ESEAP Wikimania 2023 Core Organizing Team


-
The information contained in this message is privileged and intended 
only for the recipients named. If the reader is not a representative of 
the intended recipient, any review, dissemination or copying of this 
message or the information it contains is prohibited. If you have 
received this message in error, please immediately notify the sender, 
and delete the original message and attachments.


Please consider the environment before printing this email.


___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org


--
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Mastodon: https://mastodon.social/@kiwix
* Wiki: https://wiki.kiwix.org/



OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Re: Fwd: Overlapping affiliate notification-Wikimedians of Nomads User Group

[Emmanuel Engelhart]

On 01.12.22 16:15, Florence Devouard wrote:

Well yes...

When they sent the announcement on the African Mailing List, several 
people answered the message as you may see here : 
https://lists.wikimedia.org/hyperkitty/list/african-wikimedi...@lists.wikimedia.org/thread/RENRV4UK3VKUHPQMVPG4TTFXQX5AMJWB/


But the original poster himself did not comment any further and did not 
enter into contact with us (either by email, or on this list, or on the 
wiki).


Would anyone volunteer posting a little something on their UG talk page 
to introduce us ? 
https://meta.wikimedia.org/wiki/Talk:Wikimedians_of_Nomads_User_Group


Looking at this wiki page, the purpose of UG seems slightly different. 
The word "offline" is not even used there. Maybe a sign that the project 
might not be really mature?


Kelson
--
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Re: Fwd: Overlapping affiliate notification-Wikimedians of Nomads User Group

[Emmanuel Engelhart]

On 01.12.22 04:09, Jorge Vargas wrote:
Would it be helpful to start a conversation with them and see if the 
overlap would rather be resolved by them joining this UG?


I think it would. IMO, we still lack a quality conversation between tech 
providers, integrators and end-users. Might be an opportunity here to 
help to fill the gap.


Kelson

--
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Re: offline wikipages within 2 degrees of separation

[Emmanuel Engelhart]

On 07.07.22 22:25, Željko Blaće wrote:

Firstly, thank you all for your fast and useful inputs!

Pardon me for answering all together in a compiled way,
with my comments below and inline yours...

On Thu, Jul 7, 2022 at 9:32 AM Emmanuel Engelhart <mailto:kel...@kiwix.org>> wrote:


Given a wiki and a list of article titles, we can make an offline
snapshot easily. To me the question is: What kind of selections? Based
on which approach? Using which data exactly?... and then probably find
to apply it and get the list of article titles.


I can easily say for a well covered monument it would be at least:
- Wikipedia pages on the Monument, Author, Location, Time of making
(ideally in at least 2 languages, possibly in two styles like EN & Simple)

>

- Commons page as a Gallery of related images, audio, video...3D model?
(ideally in higher and lower resolutions)
- Wikispore page with non-encyclopedic info on location and/or contexts
(ideally artistic, social and geographic focuses)


We can not - for now - create one snapshot with articles coming from 
multiple Mediawikis. This is an open challenge.


On Wed, Jul 6, 2022 at 5:16 PM Stephane Coillet-Matillon 
mailto:steph...@kiwix.org>> wrote:


Ok this sounds a lot like our Wikipedia-on-Demand project, 


What is the way to follow it? Public repository?


Project page https://meta.wikimedia.org/wiki/Kiwix/Wikipedia_on_demand.

Most of the work will happen (at least in a first stage) at 
https://github.com/openzim/wp1



ie enter a list of articles on one end and get the corresponding zim
file on the other end. Release is planned for this Fall, so we’ll
make sure to put out a call so people can test it.

Great! Happy to test.

Doing something that works cross-wiki, on the other hand, is another
ball game entirely and not on our roadmap at the moment (but make no
mistake, this would be a very cool feature).


Nice that you think so...happy to brainstorm more if there is good 
context for this.


If you are ready to invest a bit of time on this, I can send you an 
invitation to our Slack and put you in touch with the few people 
involved on this.

 Kelson

--
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/


OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Re: offline wikipages within 2 degrees of separation

[Emmanuel Engelhart]

On 11.07.22 22:37, Asaf Bartov wrote:
Yes, it sounds like the missing link here is a tool for creating the 
list of resources to offline.  Z made one particular specification, but 
I suppose it could be made a little more general, and potentially even 
leverage some existing general-purpose pageset curation tools, such as 
PetScan.  AFAIK, PetScan currently doesn't support the use-case of "get 
me N levels of pages linked from this first page (or from these P first 
pages)", but we can imagine (and advocate for) PetScan supporting it at 
some point.


Then, a PetScan query ID (which is enough to generate the page-set) can 
be an input to the Wikipedia-on-Demand tool, the problem is solved. 
(Well, almost: we'd still need to specify the logic for collecting 
related resources -- i.e. none/some/all images included in the pages, 
Wikidata items, etc.)


At a high level, this is indeed the kind of challenge we face now. This 
is a lack of tool which has been identified already a long time ago at 
Kiwix. I believe we are now ready to move forward on this because the 
underlying software pieces are ready.


The overall strategy is to extend wp1.openzim.org (API) to allow to 
implement sophisticated selection modules. So far, how these modules 
will look like at the end is really open and all the ideas are welcome 
(please open tickets at https://github.com/openzim/wp1). 
Collaborating/Relying with/on PetScan is an idea which should be assessed.


Once a selection done, our Zimfarm infrastructure is ready to build the 
snapshots (ZIM files). We will probably have to build (a) dedicated 
frontend(s) to bring these two tools together in a user friendly manner.


This is the goal of the Wikipeda-on-Demand project (WMCH granted) 
https://meta.wikimedia.org/wiki/Kiwix/Wikipedia_on_demand we have 
started to work on. We have an other project (related to the war in 
Ukraine) in the pipe which should even extend this tool.


Kelson

--
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/


OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Re: offline wikipages within 2 degrees of separation

[Emmanuel Engelhart]

Given a wiki and a list of article titles, we can make an offline 
snapshot easily. To me the question is: What kind of selections? Based 
on which approach? Using which data exactly?... and then probably find 
to apply it and get the list of article titles.


On 06.07.22 11:50, Željko Blaće wrote:

Dear fellows  - I would like to offline host
minimal set of Wiki pages (Wikipedias, Wikispore, Commons)
and minimal set of media (images, video and audio)
that all branch out from single Reasonator
(or Portal.toolforge.org ) search...
...and that keep global links beyond 2 degrees
(so that one can use them to continue online).

For now I think of just downloading pages offline
and hosting them as mobile wiki-to-static pages
using wget with manual corrections.

I would love to do this for monuments and
tiny libraries like this https://w.wiki/5QWn 
so maybe have library-like single-file wiki
(like https://tiddlywiki.com  system)
so that users could connect to it
and live notices.

Anyone has idea if and how to do this better
and package in the most elegant way?
Anyone interested in collaborating?

Best Z. Blace



___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org



--
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/


OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] [KIWIX] Release 3.3 of Kiwix JS

[Emmanuel Engelhart]

Hi

I remark that it seems it makes a year a half that we don't have post 
posted here news about Kiwix JS. It is a pity, because this is one of 
our most active code repository!


On the top of this, we just have released version 3.3.1 of all Web 
Browser extensions (Firefox, Chrome & Edge) and Kiwix JS for Windows.


You can find the latest version of Kiwix JS Browser extensions:
* Chrome: 
https://chrome.google.com/webstore/detail/kiwix/donaljnlmapmngakoipdmehbfcioahhk

* Firefox: https://addons.mozilla.org/en-US/firefox/addon/kiwix-offline/
* Edge: 
https://microsoftedge.microsoft.com/addons/detail/kiwix/jlepddlenlljlnnhjinfaciabanbnjbp


Here are the recent changelogs:

Kiwix-JS v3.3.0
* FIX: Improve packaging for the Ubuntu Touch app
* FEATURE: Provided a workaround to enable Service Worker mode in 
Firefox extensions (via a PWA)
* NEW: App now defaults to Service Worker mode if accessed directly as a 
PWA (does not apply to browser extensions)
* NEW: Support for non-http URLs in img and link tags (e.g. data: 
javascript:) in JQuery mode

* NEW: Added helpful tooltips to icons and settings in the app
* FIX: Scrolling to anchor targets in JQuery mode
* FIX: Caching of ZIM assets now includes any required query string in 
SW mode
* FIX: Any favicon declared by the ZIM is now extracted and attached in 
JQuery mode
* BUGFIX: Incorrect processing of titles with question marks or hashes 
prevented display of some articles


Kiwix-JS v3.2.0
* NEW: Support latest format of ZIM archives (with no namespace)
* NEW: Optimization of title search by eliminating redundancies
* NEW: Support new format of title listings (v1) in no-namespace ZIM 
archives

* NEW: Use fast binary WASM decoders with fallback to ASM if necessary
* UPDATE: Clearer and more extensive documentation for end users in About
* BUGFIX: Issue with calculation for selection of random articles

Kiwix-JS v3.1.0
* NEW: Low-level block cache significantly improves binary search speed, 
and can speed up resource loading in most browsers
* NEW: Archives with WebP-encoded images are now decoded in legacy 
browsers via a polyfill
* UPDATE: Images are now extracted sequentially in jQuery mode from the 
top of the DOM
* BUGFIX: Running binary searches are now cancelled completely if user 
enters new search term or navigates away


The full Changelog can be found here:
https://github.com/kiwix/kiwix-js/blob/master/CHANGELOG.md

We have already started to release ZIM files "without namesapces" and 
this version 3.2.0+ is mandatory if you want to enjoy them with Kiwix JS.


All our Windows apps based on Kiwix JS have been updated in Microsoft
App Store too. You can enjoy our apps on all Windows
based devices (mobiles, tablets, PC, Xbox, ...):
* Wikivoyage:
https://www.microsoft.com/fr-ch/p/wikivoyage-by-kiwix/9n5sb90q4jbj
* Wikimed: https://www.microsoft.com/fr-ch/p/wikimed/9phjsnp1cz8j
* Kiwix: https://www.microsoft.com/fr-ch/p/kiwix-js/9p8slz4j979j

Please report any problem. Work continues on our forge at
https://github.com/kiwix/kiwix-js and
https://github.com/kiwix/kiwix-js-windows.

Regards
Emmanuel

--
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication


OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Re: Message size beyond 250 ko ?

[Emmanuel Engelhart]

On 25.01.22 20:24, Florence Devouard wrote:
And yes... could perhaps the list moderator INCREASE a little bit the 
size of messages that could be sent to the list ? ;)


Not sure if this is doable, should be checked. But just out of 
curiosity, what have you sent to achieve to get a message over 250KB?


Kelson


--
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Maintenance release libzim 7.1.0 is out!

[Emmanuel Engelhart]

Hi,

We have released the first minor release of libzim7 branch: libzim 7.1.0.

Nothing really big, here is the changelog:
 * Fix dirent test on 32 bits architectures (@mgautierfr #632)
 * Fix compilation on Alpine - with musl (@amirouche #649)
 * Don't crash if ZIM without illustration nor X/W namespace
(@mgautierfr #641)
 * Switch default suggestion operator to AND (@maneeshpm #644)
 * Add a new method Archive::getMetadataItem (@mgautierfr #639)
 * Better indexion criterias (@mgautierfr #642)
 * Avoid duplicated archives in the searcher (@veloman-yunkan #648)
 * Fix random entry (@veloman-yunkan #650)
 * Various improvements.
   - CI @mgautierfr #640, @kelson42 #638, @legoktm #654
   - Doc @rgaudin #646

A new Python-libzim using it will be released soon and we are still
working on the libzim7 Node.js binding.

A small minor release 7.2.0 is already in the pipe and should be
released next week.

More info at: https://github.com/openzim/libzim

For the Kiwix team
Kelson
-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Zim-tools 3.0.0 has been released!

[Emmanuel Engelhart]

Hi,

After announcing the libzim7 release a month ago, we are proud to
release the version 3.0.0 of the Zim-tools. The Zim-tools is the
swiss-knife for manipulating ZIM files.

Actually this release has been made available already a few weeks ago
and here again this is major release. It makes almost a year we have
been working on this new release which mainly brings support of libzim7
and major improvements to the zimcheck tool.

Here is the complete changelog:
 * zimwriterfs: Do not allow zimwriterfs to overwrite a existing zim
file (@asashnov #193)
 * zimcheck: Make zimcheck print json output with option `--json`
(@veloman-yunkan #206)
 * zimcheck: Make zimcheck multithreaded (@veloman-yunkan #194)
 * zimcheck: Add an option to check for redirection loops (@shashwat1002
#218)
 * zimrecreate: Add an option `-j` to not create fulltext index
(@data-man #234)
 * zimrecreate: Add option `-J` (--threads) to set the number of threads
to use (@data-man #235)
 * remove zstd compression options. We are now always creating zim file
with zstd (@kelson42 #261, 264)
 * zimwriterfs: Do not use the extension of file as mimetype (@mgautier
#265)
 * zimcheck: Improve performance of detection of the kind of url
(@veloman-yunkan #179)
 * Support new protocols xmpp, news and urn (@kelson42 #205)
 * zimdump: zimdump now print the favicon and mainentry path instead of
the index (@kelson42 #220)
 * zimdump: Add a option to create html redirect insteal of symlink
 * zimdump: Fix issue where zimdump crash depending of the stored path
of redirection (@maneesh #211)
 * zimdump: Fix argument --ns not working (@maneesh #216)
 * Add zimcheck unit tests (@veloman-yunkan #215)
 * zimdump: Fix http redirect (@veloman-yunkan #225)
 * Adapt zim tools to new libzim api (@mgautier @veloman-yunkan @maneesh)
 * Compile zimsearch binary only if libzim is compiled with xapian
   (@mgautier, #201)
 * Docker : Better open zimtools image (@rgaudin #209)

This new release is an important milestone in our global QA effort
around ZIM files. Latest Zimcheck has been already integrated in our
Zimfarm and its new machine readable output can now be checked in our
(currently being build) CMS.

From a more general point of view, improving the Zim-tools is essential
to allow hackers to easily deal with the ZIM files and maybe even
produce their own. More infos at https://github.com/openzim/zim-tools.

Happy coding!

Kelson

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Python-libzim 1.0.0 has been released!

[Emmanuel Engelhart]

Hi,

After announcing the libzim7 release a few days ago, we are proud to
release the version 1.0.0 of Python-libzim. Python-libzim is our Python
binding for the libzim and the version 1.0.0 supports fully the libzim7.

Here is the complete changelog:
* [breaking] Uses new libzim 7-based API for both reader and writer
  * No more namespaces
  * Defaulting to zstd compression
* Python 3.9 support
* Uses libzim's own test ZIM files to test reader
* Code-coverage now includes Cython code as well
* macOS releases are signed and notarized
* Early-failure on invalid destination ZIM path

The Python-libzim is a pretty recent software library which has been
launched in 2020 and allows easily any (Linux/macOS) Python developer to
read and write ZIM files. You can easily install it with "pip", more
details at: https://pypi.org/project/libzim/

Almost all our openZIM scrapers (but not the Mediawiki one) are written
in Python and many of them rely already on the Python-libzim. If you are
a Python expert or beginner, your help is welcome to improve it... or
relying scrapers. Here is where to start:
https://github.com/openzim/python-libzim.

Happy coding!

Kelson

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Re: Libzim 7.0.0 released!!!

[Emmanuel Engelhart]

On 11.10.21 10:07, Matthieu Gautier wrote:
> Le 10/10/2021 à 21:38, Amirouche Boubekki a écrit :
>>
>> That is great ! I did not look at linked page yet. What are hints ?
> 
> Hi,
> 
> "Hints" is a generic system to allow the scraper to pass ... hints to
> the creator when we create a zim file.
> Hints are information for the creator only. They may change the way the
> entries are stored
> but they are not stored in the zim file and readers cannot access them.
> 
> For now we support only two hints:
> - COMPRESS : Force the creator to compress or not the content (if not
> given, use default heuristics based on mimetype)
> - FRONT_ARTICLE : Tell the item/redirection is a entry intended to be
> displayed to the user as a article (to oppose to resources as image/css/js)
>   Random articles or suggestion use front articles only (when possible).
> 
> COMPRESS is not really important but FRONT_ARTICLE is somehow mandatory
> as we have no other way to differentiate articles from resources
> (as we don't have namespace now)

Important other point: the documentation has been seriously improved.
There is even a migration documentation. Look at
https://libzim.readthedocs.io/

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Libzim 7.0.0 released!!!

[Emmanuel Engelhart]

Hi,

this is with a great joy that I can announce to you:
We have finally released the major version 7.0.0 of the libzim.

This is one of the biggest libzim releases we have ever made:
* Almost one year of effort
* ~100 tickets https://github.com/openzim/libzim/milestone/2?closed=1
* Code coverage increased from 50% to 85%
* 15K lines of code changed

This new release brings:
* A strongly rewritten new API: search engine, suggestions, creator, ...
* Most notable change is that now namespaces are "hidden" by the API
* Ability to add hints to articles
* New index with only the user visible/front articles
* Backward and forward compatibility is secured
* Other bug fixes and improvements, check
https://github.com/openzim/libzim/blob/master/ChangeLog

The ZIM file format specification has itself only changed a little, see:
https://wiki.openzim.org/w/index.php?title=ZIM_file_format=revision=31202=31166

A cascade of new releases will be done in the next weeks to bring all
the benefits of this latest libzim in both openZIM scrapers and Kiwix
readers.

I use the opportunity of this release to send a particular "Thank you!"
to the Wikimedia Foundation which has been supporting us for many years
now. This hard work on a low level library like the libzim would not
have been possible without its support.

For the Kiwix team
Kelson

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



OpenPGP_signature
Description: OpenPGP digital signature
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

[Offline-l] Kiwix Hotspot 2.4 released!

[Emmanuel Engelhart]

Hi,

We are proud to announce that Kiwix Hotspot v2.4 is out and ready to be
used on Windows/macOS/Linux:
https://download.kiwix.org/release/kiwix-hotspot/v2.4.5/

As I write this email, I realize that no release announcement, for Kiwix
Hotspot, has been made since Summer 2019! This is really not fair to the
product on which we have invested so much energy and which is developing
well its own audience.

As a reminder, the Kiwix Hotspot is a click-and-play solution to create
offline library on Raspberry-Pi hardware. Download and run the soft on
your computer, choose the works & softwares you love in the library and
Kiwix Hotspot will prepare automatically your micro-SD card. It has
never been that easy to create and offline library on Raspberry Pi.

So many things has been improved in this new milestone (and the other
one before), that I have no other choice than to just to share the
CHANGELOG here:
https://github.com/kiwix/kiwix-hotspot/blob/master/CHANGELOG

What is important to notice as well, is that we have a fully working
Software-As-A-Service (SAAS) version of Kiwix Hotspot. The so-called
"Kiwix Hotspot Cardshop" allows users to build there own custom RPI
image "in the Cloud". You can buy an access to it for a cheap price at
https://www.kiwix.org/en/cardshop-access/.

Regards
Kelson

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/
___
Offline-l mailing list -- offline-l@lists.wikimedia.org
To unsubscribe send an email to offline-l-le...@lists.wikimedia.org

Re: [Offline-l] New tool : Zimit

[Emmanuel Engelhart]

Hi Adam

On 22.12.20 15:20, Adam Holt wrote:
> 2) However for offline / low-income communities there's a fatal flaw
> with ZIM files that require https (TLS/SSL) which generally require web
> server certificates that renew every 90-days-or-so using a service like
> https://letsencrypt.com -- by definition this is inherently / tragically
> not possible in offline / low-income communities.  As Tim Moody alluded
> to below, and also at openzim/zimit#57
> 

It is possible to give access to these ZIM even with offline
kiwix-serve. This will just be not user friendly as the user will have
to go through the warning triggered by the browser because of the lack
of CA certified certificate. This is the way we will follow for the
moment in Kiwix-Hotspot to give access to these ZIM files. The
experience will show us how the users can (not?) deal with this.

> 3) Does anybody see any practical way forward, for us all to pull
> together here, towards giving offline / low-income communities a very
> pragmatic way forward?

This is probably possible with an additional serious software
engineering effort to replace the Service-Worker based wabac.js. This
has been discussed here: https://github.com/openzim/warc2zim/issues/48.

If this kind of ZIM has a lot of success and we find a financing to
that, this is not excluded we will work on this in the future, but not
for the moment.

Regards
Emmanuel


-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [NEW] first releases of openZIM tool warc2zim

[Emmanuel Engelhart]

Hi Amirouche

On 01.11.20 08:52, Amirouche Boubekki wrote:
> I am working on a search engine (unlike sphinx or elastic search, more
> like bing or google), I was planning to use .zim files to feed the
> index, the problem is there is no systematic way to find the original
> URL of the documents.

Yes, with Mediawiki based created ZIM, to an extrem large extened it
should be the same articleId.
> I am wondering whether one of the following will be possible for kiwix
> project to do:
> 
> A) Add a https://foobar;> in the html inside the .zim files,

Yes, but there is not always such link possible. Many ZIMs are mash-ups.
That said this is trivial to add such a meta node in in the HTML in
MWoffliner.

> A bis) Add a metadata field per document with the original url inside
> the .zim files,

Yes, I was sure to have such a ticket open at least for MWoffliner. Can
not find it anymore. Probably both approaches are doable, you should
open a ticket in MWoffliner repository.

> B) Publish .warc files of wikipedia, stackoverflow dumps etc... so
> that people like myself can re-use those. WARC files are more useful
> than .zim files but still less user friendly than the following
> proposal...

Why are WARC more useful, beside the fact that they have an exact copy
of of the original Web page?

> C) ... One last alternative, is to pivot the custom .zim file storage
> to an okvs [0] like rocksdb or sqlite lsm extension [1]. The idea is
> to make it very easy to access the kiwix dumps from many programming
> languages unlike the current approach that is limited to C++ and
> Python. Also, it will be easier to extend a given dump with custom
> fields, unlike the current .zim which seems to be read-only.

There is bindings for Go and Javascript as well. Which kind of
additional binding do you need? Creating a binding for libzim takes from
a few days to 2 weeks (depending how you want to make it). This is easy.
If you want to read the content of Wikipedia, this is the easiest
solution (if the current tools are not enough). zimdump allows you as
well to extract extremely efficiently  the content from the command line.

I personally think we should create a fuse driver.

Yes, the ZIM format is readonly. If you want to write content, then you
definitely need a rw DB, whatever its name. That does not mean you have
to replace the ZIM, this can be complementary. An other point is that if
you need a DB, that basically mean you have an additional source of
information you deal with. Something you don't have talked about.

Please open a ticket in a repo if you need anything from the libzim or
MWoffliner.

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [NEW] first releases of openZIM tool warc2zim

[Emmanuel Engelhart]

Hi

I'm very proud to announce the release of our new tool: warc2zim.

Warc2zim is a command line tool for GNU/Linux and macOS which allows to
convert a WARC file to a ZIM file. WARC being a widely used storage
format of the archive world, warc2zim offers new opportunities to reuse
WARC stored data and benefit of the whole feature set of the ZIM file
format and readers like Kiwix.

The tool has been achieved with the strong collaboration of the
Webrecorder team. It is one milestone of a bigger project called Zimit,
a project we run we the sponsoring of the Mozilla Foundation.

The ZIM created using that process works slightly differently than the
traditional ones (the ZIM specification is formally respected). We are
currently running an effort to update all the Kiwix readers, but it
already works well with Kiwix Serve.

The tool is distributed at:
https://pypi.org/project/warc2zim/

More news to come about warc2zim and Zimit in January 2020.

Happy scraping!
Happy coding!
Happy offline reading!

Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] Release 3.0.0 of Kiwix JS (with Zstandard support)

[Emmanuel Engelhart]

Hi

We just have released version 3.0.0 of Kiwix JS browser extensions for
Firefox, Chrome & Edge and Kiwix JS for Windows.

You can find the new browser extensions:
* Chrome:
https://chrome.google.com/webstore/detail/kiwix/donaljnlmapmngakoipdmehbfcioahhk
* Firefox: https://addons.mozilla.org/en-US/firefox/addon/kiwix-offline/
* Edge:
https://microsoftedge.microsoft.com/addons/detail/kiwix/jlepddlenlljlnnhjinfaciabanbnjbp

Kiwix JS 3.0.0 Changelog:
* NEW: Add support for reading ZIM archives compressed with the
zstandard compression algorithm
* UPDATE: Make Privacy Policy publicly accessible

We have already started to release Zstandard based ZIM files and this
new version is mandatory if you want to enjoy them with Kiwix JS.

All our Windows apps based on Kiwix JS have been updated in Microsoft
App Store. You can enjoy our apps on all Windows
based devices (mobiles, tablets, PC, Xbox, ...):
* Wikivoyage:
https://www.microsoft.com/fr-ch/p/wikivoyage-by-kiwix/9n5sb90q4jbj
* Wikimed: https://www.microsoft.com/fr-ch/p/wikimed/9phjsnp1cz8j
* Kiwix: https://www.microsoft.com/fr-ch/p/kiwix-js/9p8slz4j979j

Please report any problem. Work continues on our forge at
https://github.com/kiwix/kiwix-js and
https://github.com/kiwix/kiwix-js-windows.

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication











signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Kiwix/openZIM is moving to Zstandard compression

[Emmanuel Engelhart]

Hi

The ZIM format relies on third party compression algorithms to provide
the best user experience. This means, high compression level, quick
decompression, high portability, etc. Overtime the ZIM has used gzip,
bzip2 and since almost 10 years the lzma2 version of the xz library.

Time to time new modern compression algorithms are created and we use to
evaluate them to see if this would bring a significant improvement for
our users. Usually it would not be worth the effort of switching, but
this has been different in the case of Zstandard (zstd).

The key advantage is that zstd offers a higher decompression speed for a
similar (to lzma) compression ratio. That means articles are displayed
even quicker than before and the energy consumption is a lot less. You
can read the whole discussion at
https://github.com/openzim/libzim/issues/84.

The decision of switching from lzma to xz has been made earlier this
year and actually most of our Kiwix ports have been able to read zstd
based ZIM files since more than a quarter. We are approaching now the
end of the transition stage for Kiwix readers and therefore we will soon
(during October) start to publish ZIM files using the new compression zstd.

For 99% of the users this move will be totally transparent and they
should only notice the reading/searching speed-up. For the 1% left a few
things have to be considered:
* If you can not read a new ZIM file, please update to the latest
version of your (Kiwix) reader (this is always a good advice).
* For all non Kiwix readers, but still libzim based, these readers
should be re-compiled/re-released with the latest libzim.
* For users relying on ZIM files but not on the libzim/Kiwix software
stack, the software needs to be fixed.

We don't plan so far to discontinued the support of lzma compression
algorithm even if we discouraged to use it as zstd is simply always
better. Older ZIM files are still perfectly readable, and this won't
change in the next couple of years.

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [WIKIMEDIA TECH TALK] In 3 HOURS: openZIM/Kiwix ETL toolchain for Wikipedia dumping

[Emmanuel Engelhart]

Hi

In 3 hours I will make a Wikimedia Tech Talk to present our toolchain to
publish offline content:

Enjoying Wikipedia offline wherever, whenever is easy with Kiwix. But
behind the scenes, a bunch of tools are needed to make it work. From
article selection to dump publishing through scraping, optimisation and
packaging: here is a quick overview of how we do it.


Join at:
https://www.youtube.com/watch?v=9rFWaPSP3AM

Regards
Emmanuel
-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [OPENZIM] Introducing Python-libzim

[Emmanuel Engelhart]

Hi Tommi

We do our best, thank you for your legacy!

Emmmanuel

On 10.07.20 22:26, Tommi Mäkitalo wrote:
> Hi,
> 
> Really cool. Nice to hear, that libzim is in good shape. Thank you for
> your work.
> 
> 
> Tommi
> 
> Am 03.07.20 um 16:40 schrieb Emmanuel Engelhart:
>> Hi
>>
>> I'm happy to introduce you to Python-libzim.
>>
>> Python-libzim package allows you to read/write ZIM files in Python. It
>> provides a shallow Python interface on top of the libzim C++ library. It
>> supports out-of-the-box macOS and GNU/Linux. For the other OSes you will
>> have to compile the libzim manually.
>>
>> After Node.js, this is the second scripting language for which openZIM
>> proposes a binding of its famous reference implementation of the ZIM
>> open specification. This move is really important to allow more people
>> to benefit of the file format and ZIM files already published.
>>
>> On our side, Python-libzim was critical for a few other projects which
>> are currently running. In the next months a few critical scrapers will
>> be migrated from zimwriterfs to python-libzim and benefit of a sensitive
>> code simplification and speed-up.
>>
>> Install easily python-libzim with pip and give it a try:
>> https://pypi.org/project/libzim/
>>
>> Happy coding!
>>
>> Regards
>> Emmanuel
>>
>> ___
>> Offline-l mailing list
>> Offline-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/offline-l
> 
> ___
> Offline-l mailing list
> Offline-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/offline-l
> 


-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [OPENZIM] Introducing Python-libzim - Youtube2zim updated

[Emmanuel Engelhart]

Hi

We have now updated our first scraper to use Python-libzim: youtube2zim,
cf. https://pypi.org/project/youtube2zim/

See release 2.1.7 Changelog:
* using zimscraperlib 1.2.0
* replaced zimwriterfs with pylibzim
* tmp-dir now sanitized as build-dir
* fixed --debug not forwarded in playlists mode
* changed workdir to /output in docker image

As a consequence, anybody on macOS or GNU/Linux can now simply install
and then create ZIM files from a Youtube channel/user/playlist with
youtube2zim:
$sudo pip3 install -U youtube2zim
$youtube2zim --help

Other scrapers to follow in the next weeks.

Regards
Emmanuel

On 03.07.20 16:40, Emmanuel Engelhart wrote:
> Hi
> 
> I'm happy to introduce you to Python-libzim.
> 
> Python-libzim package allows you to read/write ZIM files in Python. It
> provides a shallow Python interface on top of the libzim C++ library. It
> supports out-of-the-box macOS and GNU/Linux. For the other OSes you will
> have to compile the libzim manually.
> 
> After Node.js, this is the second scripting language for which openZIM
> proposes a binding of its famous reference implementation of the ZIM
> open specification. This move is really important to allow more people
> to benefit of the file format and ZIM files already published.
> 
> On our side, Python-libzim was critical for a few other projects which
> are currently running. In the next months a few critical scrapers will
> be migrated from zimwriterfs to python-libzim and benefit of a sensitive
> code simplification and speed-up.
> 
> Install easily python-libzim with pip and give it a try:
> https://pypi.org/project/libzim/
> 
> Happy coding!
> 
> Regards
> Emmanuel
> 
> 
> ___
> Offline-l mailing list
> Offline-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/offline-l
> 


-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [OPENZIM] Introducing Python-libzim

[Emmanuel Engelhart]

Hi

I'm happy to introduce you to Python-libzim.

Python-libzim package allows you to read/write ZIM files in Python. It
provides a shallow Python interface on top of the libzim C++ library. It
supports out-of-the-box macOS and GNU/Linux. For the other OSes you will
have to compile the libzim manually.

After Node.js, this is the second scripting language for which openZIM
proposes a binding of its famous reference implementation of the ZIM
open specification. This move is really important to allow more people
to benefit of the file format and ZIM files already published.

On our side, Python-libzim was critical for a few other projects which
are currently running. In the next months a few critical scrapers will
be migrated from zimwriterfs to python-libzim and benefit of a sensitive
code simplification and speed-up.

Install easily python-libzim with pip and give it a try:
https://pypi.org/project/libzim/

Happy coding!

Regards
Emmanuel
-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] At long last, a new version of offline enwp

[Emmanuel Engelhart]

This was the last milestone on the way to generate a monthly fresh ZIM
snaspshot for each of our projects. Actually, many ZIM flavours for
Wikipedias and a few other projects.

This now happens 100% automatically with the help of the Zimfarm.

The work continues on MWoffliner, which is quite a delicate beast, to
better handle the many particularities/features Mediawiki is known for.

Our small typescript team welcomes any volunteer help. We have work to
do for junior and senior developers, just pick up your ticket at
https://github.com/openzim/mwoffliner/issues.

Emmanuel

On 03.07.20 15:43, Stephane Coillet-Matillon wrote:
> quick announcement for a major success on our side: we finally released
> late last night an updated version of the English Wikipedia[1]. The last
> full version (ie. with images) we had was from October 2018 (!), and
> since then we had been plagued by regressions, bugs, resource
> limitations and probably some very dark magic.
> 
> The new .zim file adds 900,000 articles (6.1 vs. 5.2 millions) and a
> healthy 11 Gb in size (89 vs. 78 Gb). The numbers are somewhat
> misleading because we need to include internal links and redirects,
> which brings the total to 100+ million interlinked items. Emmanuel will
> have more details on the hurdles that he had to deal with.
> 
> Updates will now run on a monthly basis, which is another major
> improvement: we had initially planned on bimonthly updates as a single
> run used to take up to three weeks. It can now be done in 5-6 days \o/
> 
> Congrats to everyone involved or who supported us one way or another,
> including the Foundation with the new & bigger servers they recently
> gave us access to. Hopefully now we can move on to newer problems.

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] HTML dumps

[Emmanuel Engelhart]

On 01.07.20 19:31, Samuel Klein wrote:
> There was a positive reply by RBrounley  from the WMF side, asking to
> learn more. Kelson, have you been in touch with them by chance?

Yes, I had a short meeting with Ryan & Nick. They are a new team. The
motto of this team seems to have a better offer around Wikimedia APIs.
They have chosen to start with this HTML dump topic as Phabricator had
quite a bit of tasks around this. They had a look a bit to what we do at
Kiwix... but so far this does not seem really clear what should be the
outcome of their effort. Therefore I was not really able to help at this
stage. Our toolchain might be a solution or not... depending which
problem they want to fix.

Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Rebranding: please give your opinion NOW

[Emmanuel Engelhart]

On 28.06.20 00:59, Florence Devouard wrote:
> Hello all
> 
> 
> Sj and I have discussed and decided the following.
> 
> 
> 1) We have decided that the offline wikimedians UG will not answer the
> WMF survey.
> Main argument is purely technical. Due to the complex design of the
> survey, we consider it too difficult to collect your collective opinions
> and draw relevant conclusions to inject into the survey.
> We could have made it simpler is filling a survey that would reflect Sj
> and I opinion, but we did not feel we should do that.
> So, if you want to provide your opinion and answer the WMF survey,
> please do so at the individual level (we strongly suggest you do)
> --->
> https://meta.wikimedia.org/wiki/Communications/Wikimedia_brands/2030_movement_brand_project/Naming_convention_proposals
> 
> 
> 
> 2) With regards to signing the COOL (Community Open Letter on renaming),
> we would like to ask you to please vote by saying :
> 
> either "Yes, we should sign the letter"
> or "No, we should not sign the letter"
> 
> Neutral opinion will be counted as No
> 
> 
> If we have at least 7 members voting and at least 2/3 YES, we will sign
> the letter.
> If we have at least 7 members voting and less than 2/3 YES, we will not
> sign the letter.
> If we have less than 7 members voting, we will not sign the letter.

Yes, please sign the Open Letter.

Emmanuel


-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [KIWIX] Our openZIM farm...

[Emmanuel Engelhart]

Not sure what you mean exactly with "coordinate". You can write a tool
which can do that because both expose everything via API.

On our side, with farm.openzim.org, we have already a few third party
orgs which use it for their own stuff.

Emmanuel

On 24.06.20 16:09, Samuel Klein wrote:
> Wow, this is fabulous.  If a new zimfarm starts up,  can it coordinate
> with existing ones? 
> 
> On Tue, Jun 23, 2020 at 3:23 AM Emmanuel Engelhart  <mailto:kel...@kiwix.org>> wrote:
> 
> Hi
> 
> There is a topic I wanted to talk about here for a long time and for
> which I never have achieved to take the time to write something. A few
> recent events have been a healthy remember that I should present one our
> most recent and most useful tool: Zimfarm.
> 
> The Zimfarm is the online tool which is in charge of building and
> publishing all our ZIM files. After years of creating ZIM files by
> launching scrapers more or less manually, we had to automatise the
> process to just be able to scale the operations, ie. publishing more and
> more often ZIM files.
> 
> The effort started 3 years ago with the support of the WMF but we use it
> only since Spring 2019 in production. The tool is now perfectly running
> and we fully rely on it now. If we can publish an update of all our
> wikis one time a month, this is thanks to this piece of software too.
> 
> The Zimfarm is a half-decentralized solution which has a central node
> (called "dispatcher") in charge of orchestrating the work to do and
> multiple decentralized nodes (called "workers") which run the scraping
> tasks.
> 
> The dispatcher provides an API to manage the ZIM recipes and tasks, have
> a look to https://api.farm.openzim.org/. We have setup a Web frontend on
> this API to allow easy mgmt through a Web browser. For a better
> transparency, even anonymous users can have a look and monitor what is
> going on. Look at https://farm.openzim.org/.
> 
> One important point is that, like all the rest of our infrastructure,
> the whole system is Dockerized. Which means, this is really easy to
> install a Zimfarm worker and we invite anybody having a spare server to
> help us to provide offline snapshots of the best of the Web. The
> procedure is documented and a few volunteers have already joined in.
> Look at https://farm.openzim.org/about for more details.
> 
> The development is fully transparent at
> https://github.com/openzim/zimfarm. We have a few things which are on
> the roadmap which would welcome volunteer Python developers. Look at the
> good first issues and make your first PR!
> 
> https://github.com/openzim/zimfarm/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
> 
> Regards
> Emmanuel
> 
> -- 
> Kiwix - Wikipedia Offline & more
> * Web: https://kiwix.org/
> * Twitter: https://twitter.com/KiwixOffline
> * Wiki: https://wiki.kiwix.org/
> 
> ___
> Offline-l mailing list
> Offline-l@lists.wikimedia.org <mailto:Offline-l@lists.wikimedia.org>
> https://lists.wikimedia.org/mailman/listinfo/offline-l
> 
> 
> 
> -- 
> Samuel Klein          @metasj           w:user:sj          +1 617 529 4266
> 
> ___
> Offline-l mailing list
> Offline-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/offline-l
> 


-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Wikimedia Brand Survey

[Emmanuel Engelhart]

Notice/Request/Movement_Brand_naming_proposals
> I am not sure the banner is live yet. At least, I see no banner myself.
> It should have gone live on the 16th
> 
> 8) Thus followed much discussion after the executive statement, on
> telegram and on meta.
> Probably central place is here :
> https://meta.wikimedia.org/wiki/Talk:Communications/Wikimedia_brands/2030_movement_brand_project/Executive_statement#A_call_for_Board_statements,_and_disappointment
> APPARENTLY, a statement from the board is expected. Unless wrong, it has
> not been published yet.
> 
> 9) There is a meeting TONIGHT (21h UTC+2), community organized, on the
> matter.
> https://meta.wikimedia.org/wiki/All-Affiliates_Brand_Meeting
> I'll attend and will try to summarize
> 
> 
> Should you invest more time on the process ? Good question. I am asking
> myself the same question. We have a few more days to reflect.
> 
> 
> Florence
> 
> 
> 
> Le 18/06/2020 à 09:24, Emmanuel Engelhart a écrit :
>> Hi Florence
>>
>> Thank you for keeping us up-to-date with this process.
>>
>> On 17.06.20 20:08, Florence Devouard wrote:
>>> So, the Brand Team is moving on and has announced a short list of
>>> proposals to rename Wikimedia Foundation, UG and such to remove the word
>>> "Wikimedia"
>> I hope I'm properly informed, but AFAIK there is an extremely strong
>> majority (almost a consensus) within the community against replacing
>> "Wikimedia" with "Wikipedia" in our orgs naming. cf.
>> https://meta.wikimedia.org/wiki/Requests_for_comment/Should_the_Foundation_call_itself_Wikipedia
>>
>> But it seems that the process continues without taking in account the
>> result of this survey. To me, this seems not fair.
>>
>> Considering that offline activities are booming, in particular because
>> of the Covid19 pandemic and therefore we have a lot to do anyway, I
>> wonder why exactly I should invest more time on this process?!
>>
>> Regards
>> Emmanuel
>>


-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] Our openZIM farm...

[Emmanuel Engelhart]

Hi

There is a topic I wanted to talk about here for a long time and for
which I never have achieved to take the time to write something. A few
recent events have been a healthy remember that I should present one our
most recent and most useful tool: Zimfarm.

The Zimfarm is the online tool which is in charge of building and
publishing all our ZIM files. After years of creating ZIM files by
launching scrapers more or less manually, we had to automatise the
process to just be able to scale the operations, ie. publishing more and
more often ZIM files.

The effort started 3 years ago with the support of the WMF but we use it
only since Spring 2019 in production. The tool is now perfectly running
and we fully rely on it now. If we can publish an update of all our
wikis one time a month, this is thanks to this piece of software too.

The Zimfarm is a half-decentralized solution which has a central node
(called "dispatcher") in charge of orchestrating the work to do and
multiple decentralized nodes (called "workers") which run the scraping
tasks.

The dispatcher provides an API to manage the ZIM recipes and tasks, have
a look to https://api.farm.openzim.org/. We have setup a Web frontend on
this API to allow easy mgmt through a Web browser. For a better
transparency, even anonymous users can have a look and monitor what is
going on. Look at https://farm.openzim.org/.

One important point is that, like all the rest of our infrastructure,
the whole system is Dockerized. Which means, this is really easy to
install a Zimfarm worker and we invite anybody having a spare server to
help us to provide offline snapshots of the best of the Web. The
procedure is documented and a few volunteers have already joined in.
Look at https://farm.openzim.org/about for more details.

The development is fully transparent at
https://github.com/openzim/zimfarm. We have a few things which are on
the roadmap which would welcome volunteer Python developers. Look at the
good first issues and make your first PR!
https://github.com/openzim/zimfarm/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] How libzim and zimwriterfs handle the bigger than memory nature of Wikipedia dumps?

[Emmanuel Engelhart]

Hi Amirouche

On 21.06.20 12:12, Amirouche Boubekki wrote:
> I am new to the mailing list. I used to work on sotoki.
> 
> My question is somewhat related to my failed attempt to store
> stackoverflow dumps inside wiredtiger. Eventually, I figured that
> wiredtiger could not keep up with the write load and that it is a pain
> point at least with wiredtiger (but also with sqlite lsm extension).
> The workaround is to have as much as RAM as the data (which is in my
> opinion not acceptable), and fine-tune eviction triggers et al.

Your initiative on StackExchange/Sotoki has not been forgotten, lost. We
maintain and develop the tools. We have really improved the scraper and
made many releases these last few months:
https://pypi.org/project/sotoki/#history

> My questions are about libzim, zimwriterfs and how full-text search is
> implemented:
> 
> 1) Why zimwriterfs or libzim succeed at putting together all the html
> dumps of wikipedia? Is it because they're a lot of RAM? Or is it a
> particular algorithm?

libzim does not use a lot of RAM, otherwise it would not be able to run
a smaller devices like RPIs or Low-end smartphones.

libzim succeeds to store huge amount of data and make it available on
really small devices, because the file format and the libzim have been
conceived for that purpose. I won't explain all the details here, but
everything need for the understanding is available here
https://openzim.org/.

> 2) Follow up question, how the full-text search is put together? Is
> the index written document by document then packed into the zim file?

The fulltext search engine has "nothing" to do with the ZIM format. We
use the Xapian engine for that optional feature. We keep only the key
words in the Xapian not the documents (they are already in the ZIM).
Since a few years, this index is embedded in the ZIM file itself for a
better UX. See https://xapian.org/ for more details.

> I am working on my free time on a search engine [1], my goal is to
> have my own search engine that I can use locally. That is why, I was
> thinking about kiwix, because kiwix via .zim files provide readily
> available dumps of many useful resources. The last question is:

If you deal with large amount of free text and want to do a fulltext
search engine. This might be a good choice indeed.

> 3) how can I read the content of .zim from C code? Are there C
> bindings of libzim?

The libzim is done in C++, you won't be able to deal properly in C with it!

Good luck for your project.

Regards
Emmanuel


-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Wikimedia Brand Survey

[Emmanuel Engelhart]

Hi Florence

Thank you for keeping us up-to-date with this process.

On 17.06.20 20:08, Florence Devouard wrote:
> So, the Brand Team is moving on and has announced a short list of
> proposals to rename Wikimedia Foundation, UG and such to remove the word
> "Wikimedia"

I hope I'm properly informed, but AFAIK there is an extremely strong
majority (almost a consensus) within the community against replacing
"Wikimedia" with "Wikipedia" in our orgs naming. cf.
https://meta.wikimedia.org/wiki/Requests_for_comment/Should_the_Foundation_call_itself_Wikipedia

But it seems that the process continues without taking in account the
result of this survey. To me, this seems not fair.

Considering that offline activities are booming, in particular because
of the Covid19 pandemic and therefore we have a lot to do anyway, I
wonder why exactly I should invest more time on this process?!

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] wikifundi synchronisation

[Emmanuel Engelhart]

Hi Manfred

On 11.02.20 17:01, Manfred Gruening wrote:
> Somewhere I read that wikifundi is working towards synchronizing with
> the online wikipedias. Is this something for the near future or very far
> out??? That would be REALLY exciting!

It is not really true. We have a ticket to simplify that task here:
https://github.com/openzim/wikifundi/issues/97

... but it won't be automatic and it is not implemented.

> Also, can wikifundi be used for a wikipedia / wiktionary in a newly
> setup language - or only for the few languages currently available for
> download???

It could, but a big part of the WikiFundi is editorial work to choose
which articles, help, content to bring in the local Mediawiki... This
work has to be done.

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Kiwix portable for Linux

[Emmanuel Engelhart]

Hi Wilfredo

On 12.09.19 20:37, Wilfredo Rodríguez wrote:
> I have received some messages from Cuba requesting help to install Kiwix
> and its wikipedia package in Spanish on Ubuntu.
> 
> The main problem comes when the user can download and burn a CD on a
> public machine but he does not have Internet on his computer, that is,
> carrying the ZIM file does not represent any problem, however,
> installing kiwix on linux offline It represents another level of
> complexity especially in the installation process during the
> installation when dependencies are being resolved (this could be a
> nightmare down dependencies manually).

The problem of a CD should not be different from an USB flash drive. We
have created a documentation about how to do that for GNU/Linux and
Windows a few days ago:
https://github.com/kiwix/kiwix-desktop/wiki/Kiwix-desktop-on-an-USB-Drive-(Portable-version)

Hope that helps

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] Release 2.6.0 of Kiwix JS

[Emmanuel Engelhart]

Hi

We just have released version 2.6.0 of Kiwix JS browser extensions for
Firefox & Chrome and Kiwix JS for Windows.

You can find the new browser extensions:
* Chrome:
https://chrome.google.com/webstore/detail/kiwix/donaljnlmapmngakoipdmehbfcioahhk
* Firefox: https://addons.mozilla.org/en-US/firefox/addon/kiwix-offline/

Kiwix JS 2.6.0 Changelog:
* Support for imagemaps (area tags) in jQuery mode, like for maps in
wikivoyage
* Enable keyboard selection in search results
* Drag and drop support of ZIM files (on desktop)
* Support download of epub files (for Gutenberg ZIM files) and some
other types of downloads
* Generic handling of MIME Types, instead of the previously hard-coded ones
* Add explanations and detect when there are CORS issues when running
kiwix-js through file://
* Fix for some cache issues in ServiceWorker mode
* Some fixes for filenames with question marks, hyperlinks with
hashtags, articles with no title, and articles with special characters
in their URL

Almost at the same time our Windows apps based on Kiwix JS have been
updated in Microsoft App Store. You can enjoy our apps on all Windows
based devices (mobiles, tablets, PC, Xbox, ...):
* Wikivoyage:
https://www.microsoft.com/fr-ch/p/wikivoyage-by-kiwix/9n5sb90q4jbj
* Wikimed: https://www.microsoft.com/fr-ch/p/wikimed/9phjsnp1cz8j
* Kiwix: https://www.microsoft.com/fr-ch/p/kiwix-js/9p8slz4j979j

Please report any problem. Work continues on our forge at
https://github.com/kiwix/kiwix-js and
https://github.com/kiwix/kiwix-js-windows.

Regards
Emmanuel
-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication







signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] 1.9.7 for iOS released

[Emmanuel Engelhart]

Hi

We have published a new maintenance release of Kiwix for iOS 1.9 branch.

Here is a short changelog:
* Fix memory usage issue when performing searches
* swift 5.2
* Version bump of libkiwix, realm and SwiftyUserDefaults
* Removed third party library ProcedureKit, now use
Foundation.OperationQueue to handle async tasks

Kiwix for iOS has been conceived for iPhone & iPad devices.

The app is available on iTunes at http://ios.kiwix.org.

Please share with us problems and remarks:
https://github.com/kiwix/apple/issues

Enjoy
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication















signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Kiwix Hotspot maintenance release 2.0.7

[Emmanuel Engelhart]

Hi

We have released a few days ago a new Kiwix Hotspot version called
2.0.7. Here is the changelog (since my last Kiwix Hostpot release
annoucement on this mailing list):

2.0.7
* fixed size evaluation for KA-lite
* removed leftovers files on cards after use (kalite, aflatoun lang packs)

2.0.6
* fixed regression (not launching) by reverting to previous pygobjects
version

2.0.5
* Added raspberryPi version to version. page
* Fixed KAlite installation not working (regression)
* KA lite doesn't require manual language selection anymore
* Aflatoun lite doesn't require manual language selection anymore
* Updated master (06/06/2019)

As a reminder, Kiwix Hotspot is probably the easiest solution to
create your offline WiFi library running on Raspberry Pi. Whatever
your operating system (Windows/MacOS/Linux) is, even if you have no
technical skills, you can in a few clicks create a ready-to-use micro SD
card for your Raspberry-Pi.

More details at https://www.kiwix.org/en/downloads/kiwix-hotspot/

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication





signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Kiwix Desktop 2.0 for Windows/Linux Beta5 released

[Emmanuel Engelhart]

Hi

Kiwix-Desktop is our windowed ZIM reader for Windows/MacOS/Linux. For
years, it has been our traditional and most famous solution... but it
has reached its end-of-life.

With the help of the WMF we have been working since months to provide a
replacement: Kiwix Desktop 2.0. End of last year we have proposed a
first replacement for MacOS: http://macos.kiwix.org/. Today we want to
introduce Kiwix Desktop 2.0 for Windows & Linux with the release of its
beta5 version.

Kiwix Desktop 2.0 main features are:
* Windows & Linux
* Support new ZIM files with embedded fulltext index
* Fulltext search
* Suggestion system based on titles
* Library with both local & remote content
* Ability to search/download/delete ZIM files
* Zoom
* Fullscreen
* Reader list
* Tabs
* Search in page
* ...

You can download it:
* Windows x86 64 bits:
https://download.kiwix.org/release/kiwix-desktop/kiwix-desktop_windows_x64.zip
* Linux x86 64 bits:
https://download.kiwix.org/release/kiwix-desktop/kiwix-desktop_x86_64.appimage
or
https://download.kiwix.org/release/kiwix-desktop/org.kiwix.desktop.flatpak.
Also available on Flathub https://flathub.org/apps/details/org.kiwix.desktop

The RC1 will be published in around 10 days and we plan to release the
official 2.0.0 during July. Please give it a try and share your
feedbacks at https://github.com/kiwix/kiwix-desktop/issues.

Developers, the whole open source project is handled at
https://github.com/kiwix/kiwix-desktop. If you have a look in detail you
will see that there is still quite a bit of work to do. We really seek
for C++ volunteers to help us improving Kiwix Desktop 2.0. One or two
motivated guys would be so welcome to help our professional main developer.

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] Release 2.5.0 of Kiwix JS

[Emmanuel Engelhart]

Hi

We just have released version 2.5.0 of Kiwix JS browser extensions for
Firefox & Chrome.

You can find the new browser extensions:
* Chrome:
https://chrome.google.com/webstore/detail/kiwix/donaljnlmapmngakoipdmehbfcioahhk
* Firefox: https://addons.mozilla.org/en-US/firefox/addon/kiwix-offline/

Kiwix JS 2.5.0 Changelog:
* Support for javascript content inside the ZIM file, only in
ServiceWorker mode. Handle noscript tags in jQuery mode, and display a
warning when some unhandled javascript is detected
* Preliminary support of video/audio content (including subtitles) :
these content types are now supported. You can read them for example in
TED/TEDx, dirtybiology, oer4schools etc. BUT the main page of these ZIM
files needs javascript to work properly, which is only enabled in
ServiceWorker mode. In jQuery mode, the user needs to reach the pages
through a search
* Some technical fixes : footnotes in wikimedia ZIM files (and, more
generally, anchors in hyperlinks), incomplete searches in some specific
ZIM files (PhET and TED), compatibility with URLs that contain an
apostrophe, remove debug logs in ServiceWorker mode
* In the UI, replace the spinner by a more modern one

Work continues on our forge at https://github.com/kiwix/kiwix-js and
https://github.com/kiwix/kiwix-js-windows.

Regards
Emmanuel
-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication





signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [OPENZIM] MWoffliner 1.7.0 released

[Emmanuel Engelhart]

Hi

We have published a new version of MWoffliner: the Mediawiki scraper.
Version 1.7.0 is - like always - available here:
https://www.npmjs.com/package/mwoffliner

This new release contains mainly fixing an code improvements, here is
the detailed changelog:
* UPDATE: Deprecated desktop scraping (not --mobileLayout)
* UPDATE: Improve the article list welcome page #449
* UPDATE: Add string translation mechanism #16
* UPDATE: Fetch hundreds of redirects at a time instead of one #325
* UPDATE: Integrated MCS alongside Parsoid #388 #347 #404
* UPDATE: Introduce automated testing #146
* UPDATE: Support remote articleList and favicon #457
* UPDATE: Added --useCache option #263

This is the second milestone of a few we have planned with the support
of the WMF. Next one on the list is 1.8 and is planned for begin of
March. With 1.8 will get rid of zimwriterfs and write directly ZIM files
without temporary files on the hard disk. This will allow substantial
speed-ups and need less mass storage during the scraping process.

Like always, PR and bug reports are welcome at:
https://github.com/openzim/mwoffliner

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication





signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Offline-l Digest, Vol 84, Issue 4

[Emmanuel Engelhart]

On 24.12.18 14:10, Michael Graaf wrote:
> But that's changing! The new generation of "smart feature phones" using
> KaiOS does have WiFi access, and
> has already taken off in India. Where I am, South Africa, will see its
> first KaiOS phone in the next few months. So let's hang in, and let's
> try to get suitable apps developed for these devices.

FYI, here a statement about porting Kiwix JS to KaiOS:
https://github.com/kiwix/kiwix-js/issues/396

Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [OPENZIM] MWoffliner 1.6 released

[Emmanuel Engelhart]

Hi

We have published a new version of MWoffliner: the Mediawiki scraper.
Version 1.6.0 is - like always - available here:
https://www.npmjs.com/package/mwoffliner

This new release contains mainly fixing an code improvements, in
particular for non-Wikimedia sites.

Here is the detailed changelog:
* FIX: Template referencing #421
* FIX: Generated files are more no-js friendly #324
* FIX: Added top level section when it doesn't exist in API Response #180
* FIX: Fixed --addNamespace #338
* FIX: CSS Image issue #418
* UPDATE: Implemented PDF downloads #338
* UPDATE: Added mwmatrixoffliner tests
* UPDATE: Added git revision to version log #426
* UPDATE: Normalised Resources and Templates
* UPDATE: Strip HTML from title tags #159
* UPDATE: Improved Error Handling
* UPDATE: Converted to TypeScript #386

This is the first milestone of a few we have planned with the support of
the WMF, here is the roadmap:
https://github.com/openzim/mwoffliner/milestones?direction=asc=title=open

Like always, PR and bug reports are welcome at:
https://github.com/openzim/mwoffliner

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [OPENZIM][KIWIX] New offline version of the Gutenberg project

[Emmanuel Engelhart]

Hi

We finally have generated new offline versions of the famous Gutenberg
project library which has almost 60.000 free ebooks. See
https://www.gutenberg.org/.

This version provides of course more books than ever before, but also
fixes many bugs which were impacting the scrapper or the user interface.
We have extended the number of UI supported languages and fulltext
search is now provided natively with Kiwix - no need to compute an
external fulltext index.

The offline versions are available like usual in the ZIM format. More
details about the ZIM format and the openZIM project at
http://www.openzim.org/

You can easily download one ZIM file with all the Gutenberg project
books in all languages:
* via HTTP: https://download.kiwix.org/zim/gutenberg_mul_all.zim
* via BitTorrent:
https://download.kiwix.org/zim/gutenberg_mul_all.zim.torrent
* Language specific files at https://download.kiwix.org/zim/
* rsync or FTP are also supported.

These ZIM files are readable wherever you want and on almost all kinds
of devices thanks the multiple Kiwix ports, pick yours at
http://www.kiwix.org/ or on your preferred app store.

If you are curious and want just to have a look, please visit your
library at https://library.kiwix.org/gutenberg_mul_all_2018-10/.

For tech people, all the scrapper is open source and maintained. Join us:
* https://github.com/openzim/gutenberg
* https://pypi.org/project/gutenberg2zim/
* https://hub.docker.com/r/openzim/gutenberg/

We hope you will enjoy it!

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] Kiwix JS and Kiwix JS Windows 2.3.1 are out!

[Emmanuel Engelhart]

Hi

We just have published a new maintenance release of our Javascript based
Kiwix ports for Chrome & Firefox browsers, Windows mobile and Xbox
devices as for UBports (Ubuntu Touch).

As a reminder, these ports are not full featured like native code
based Kiwix ports and might be a bit slower, but offer other nice
properties, like for example an excellent portability.

Kiwix JS 2.3.1 Changelog:
* Fix excessive memory consumption
* Fix SVG file handling
* Add favicon

You can find the new Browser extensions:
* Chrome:
https://chrome.google.com/webstore/detail/kiwix/donaljnlmapmngakoipdmehbfcioahhk
* Firefox: https://addons.mozilla.org/en-US/firefox/addon/kiwix-offline/

Update is also available on the Microsoft app store for the Kiwix JS
Windows reader and a few custom apps, see:
* https://www.microsoft.com/en-us/p/kiwix-js/9p8slz4j979j
* https://www.microsoft.com/en-us/p/wikimed/9phjsnp1cz8j
* https://www.microsoft.com/en-us/p/wikivoyage-by-kiwix/9n5sb90q4jbj

Work continues on our forge at https://github.com/kiwix/kiwix-js and
https://github.com/kiwix/kiwix-js-windows.

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Kiwix and openZIM wave of releases

[Emmanuel Engelhart]

Hi

Two weeks ago we have  made a new waves of releases of many core
projects for both openZIM and Kiwix.

Here are the bits of changelog:

libzim 4.0.0


 * [Writer] Remove a lot of memory copy.
 * [Writer] Add xapian indexing directly in libzim.
 * [Writer] Better API.
 * [Writer] Use multi-threading to write clusters.

zimwriterfs 1.2
===

 * Revert "Add leading '/' at indexer url.
 * Move to meson build system.
 * Addapt to new libzim's writer's API.
 * Add travic CI.
 * Add redirect articles after "normal" articles.
 * Explicitly use icu namespace to allow use of packaged icu lib.
 * Fix help/log typos.
 * Better README.

zim-tools 1.0.0
===

 * Remove warnings.
 * Compile zim-tools with travis.
 * Add zim-check tool.
 * Addapt zimDiff and zimPatch to new version of libzim writer.
 * [zimSplit] Do not create a new ofstream for each part. (Do not need gcc5)
* Do not use `std::to_string` (Allow compilation with gcc4.8)

kiwix-lib 2.0.1
===

 * Fix parsing of url.
 * Remove unused static resources.
 * Correctly decode reserved characters in URLs.
 * Explicitly use icu namespace to allow use of packaged icu lib.

kiwix-tools 0.6.0
=

 * remove kiwix-install tool.

kiwix-serve
---

 * Improved taskbar #160
 * Fix global page when using the option `--nosearchbar`
 * Return 404 for missing resources
 * Fix compilation for gcc 4.8.

kiwix-manage


 * Returns proper exit code (not always 0)

Preparation of next wave has already started :)

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Haiti trip - looking for help getting Kiwix working on OLPC XO-1.5

[Emmanuel Engelhart]

Hi

Just wanted to close that thread by sharing the information that we have
finally added the full support in libzim/kiwix-tools/kiwix-serve of
older 586 architecture two weeks ago. The nightly binaries have been
tested by the IIAB team and already rollouted with success in Haiti.

Here are the Kiwix (serve) nightlies:
https://download.kiwix.org/nightly/

This will be part of to-be-released son new version of the Kiwix tools:
https://download.kiwix.org/release/

Regards
Emmanuel

On 26.03.2018 10:01, Matthieu Gautier wrote:
> Hi Avni,
> 
> Do you know https://github.com/kiwix/kiwix-build ?
> It is the tool we used to compile all kiwix projects (and so kiwix-tools).
> It download all sources, and compile everything from dependencies to
> kiwix-tools.
> 
> Anyway, you can compile kiwix-tools without kiwix-build :
> 
> Meson need the sources of kiwix-tools to compile them. You will need to
> clone them with `git clone https://github.com/kiwix/kiwix-lib`
> Once they are cloned, you can use them. The command is `meson
>  `.
> So if you already are in the source directory and want to build in the
> subdirectory build, the command is `meson . build`
> 
> Once the build is configured (meson is the equivalent of ./configure),
> you can compile it with `ninja` (being in the build_directory)
> 
> Cheers,
> Matthieu Gautier.
> 
> Le 26/03/2018 à 08:05, Avni Khatri a écrit :
>> Hi -
>>
>> I am working on compiling kiwix-tools, and therefore kiwix-lib for the
>> XO-1.5 (fedora 18), in preparation for the trip to Haiti in April.
>> I am following the instructions here:
>> https://github.com/kiwix/kiwix-tools and here:
>> https://github.com/kiwix/kiwix-lib
>> With Jerry, Nathan, and Adam's help, I was able to install all
>> dependencies for kiwix-lib (icu, zim, pugixml, ctpp2 [had to compile
>> from source], xapian) and
>> the environment compilation tools (automake, libtool, pkg-config).
>> But I am having trouble with *meson*. Meson requires python 3.5+ so we
>> installed python 3.6 from source (via
>> https://danieleriksson.net/2017/02/08/how-to-install-latest-python-on-centos/
>>
>> ).
>> Meson (finally) was successfully installed that way, but now am
>> running into the following error when running "meson . build" as
>> instructed to do so here:
>> https://github.com/kiwix/kiwix-lib.
>> /*Error during basic setup: Neither directory contains a build file
>> meson.build*/
>> Based on searching the web, it seems like this command requires the
>> source existing thought it seems per here:
>> https://github.com/kiwix/kiwix-tools/issues/158
>> that it may not be the preferred route anymore. The solution in that
>> issue was to run this command:
>> //
>> */sudo -u kiwix /bin/bash -c "cd /opt/kiwix;
>> /usr/lib/kiwix/bin/kiwix-manage /opt/kiwix/data/library/library.xml
>> add
>> /media/esata/MEDIA/System/Kiwix/data/content/wikipedia_en_for-schools_2013-01.zim"/
>> *
>> Would appreciate any guidance on how to proceed. Many thanks.
>>
>> Avni
>>
>>
>> On Tue, Mar 6, 2018 at 7:00 PM, Adam Holt > > wrote:
>>
>>     On Mar 6, 2018 6:26 PM, "Avni Khatri" >     > wrote:
>>
>>     Hi all -
>>
>>     As part of the Haiti trip, we are looking for help getting
>>     Kiwix (kiwis-tools)
>>
>>
>>     kiwix-tools (enables fully searchable offline Wikipedia, across
>>     disconnected lands).
>>
>>     This is necessary to display the new ZIM files (these contain
>>     full-text search indexes, since about a year ago, consequently the
>>     old Kiwix on i686 no longer works!)
>>
>>      compiled and working on OLPC XO-1.5
>>     .
>>
>>     We believe this should be straightforward as it *was* running
>>     on i686 just over a year ago at the end of 2016:
>>     https://github.com/kiwix/kiwix-build/issues/94
>>     
>>
>>     Steps are outlined at the bottom of:
>>     https://github.com/kiwix/kiwix-tools
>>      and we think it's a
>>     matter of knowing which compiler flags to use.
>>
>>     Support by Emmanuel & the Kiwix team will be available via IRC
>>     channel #kiwix at http://webchat.freenode.net
>>      for questions. Adam (cc'ed) can
>>     also provide additional technical details as necessary.
>>
>>     We need this so kids in the two schools we are visiting can
>>     view the Wikipedia we all love so much offline and has the
>>     impact to help any kid (or anybody!) using 32-bit Linux PC's.
>>
>>
>>     If an actual XO-1.5 laptop (about 3X faster than the original One
>>     Laptop Per Child laptop, and a more standard i686 architecture) is
>>     needed to compile kiwix-tools please let us know.
>>
>>     "It should be almost trivial IMO to get the binary 

Re: [Offline-l] Creating an offline-wiki user group

[Emmanuel Engelhart]

On 20.04.2018 22:46, Samuel Klein wrote:
> Perhaps we can use the old 2012 portal on meta but update it - At least
> with start and end dates for projects no longer running.  Those are the
> most linked to from elsewhere.

+1

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [INFO] Few good news about mwoffliner

[Emmanuel Engelhart]

Hi

We rarely write here specifically about mwoffliner, even if this tool is
time to time named by-the-way in threads. But these last months we have
done many interesting improvements to this important tool and I thought
it might be valuable to report quickly about them.

As a reminder, mwoffliner is a script which is thought to build a ZIM
file from any (recent) online Mediawiki. It scraps a snapshot of the
online wiki (HTML/JS/pictures/...) on your local disk.

Here is the list of recent improvements:

* We have introduce Parsoid as a local dependence, which means that even
if a Mediawiki does not have Parsoid/Visual Editor installed, mwoffliner
should have a chance now to build the ZIM file of it by running Parsoid
locally.

* We have introduced the Parsoid mobile layout suppport which allows to
build ZIM file with a similar layout as Wikipedia Mobile version. This
is pretty much in beta and we plan first to use it only for Wikipedia.org.

* We have introduced the support of audio/video which means that now,
like the pictures, they are mirrored too. Our first tests show that for
Wikipedia it tends to multiply the size of the ZIM file by a factor
four. As a consequence we won't use it directly everywhere. That said
the feature is there and we will step-by-step introduce video in the ZIM
files we are generating with mwoffliner.

* We have published mwoffliner (and mwmatrixoffliner) to the npmjs
repository: https://www.npmjs.com/package/mwoffliner. Now everybody can
install it easily (but you still need to take care about the dependences).

* We have made the script a bit more modular: you can call it like any
other program but now you can also use it as a library in your own
Javascript/Node.js scripts.

* We have moved the git repository to the openZIM organization on
Github: https://github.com/openzim/mwoffliner. By moving all our scraper
to the openZIM organization we hope to bring a bit of clarity between
Kiwix and openZIM respective duties. Have a look to all other scrapers
we have migrated to openZIM: https://github.com/openzim

mwoffliner is not a tool for everybody but it is really important to
continue to improve it to provide quality ZIM files of Wikipedia,
Wiktionary, ... So if you have Javascript skills please come to help us
to prepare the next big steps forward
https://github.com/openzim/mwoffliner/issues

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [OPENZIM] Stack Exchange Network offline version

[Emmanuel Engelhart]

Hi

After a year a half of efforts, we are proud to announce that we have
done our first delivering of ZIM files of the Stack Exchange web sites.

All the ZIM files are freely available to download via the Kiwix
software or directly on the Kiwix download server:
http://download.kiwix.org/zim/stack_exchange/

Stack Exchange is a network of question-and-answer websites on topics in
varied fields, each site covering a specific topic, where questions,
answers, and users are subject to a reputation award process. This
include famous web sites like Stackoverflow.com, AskUbuntu.com or
Superuser.com. More information about Stack Exchange and its more than
100 web sites is available here:
https://en.wikipedia.org/wiki/Stack_Exchange.

This ZIM files are done thanks to regularly updated archives provided on
archives.org and an ad-hoc software our team has specially developed for
that purpose. This software is called "Sotoki" and his of course open
source. You can have a look to the source code here
https://github.com/openzim/sotoki or use it directly using Python pip
packager:
https://pypi.python.org/pypi/sotoki

We plan to release updates of these ZIM files each time new archives
will be published.

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] for iOS 1.8.1 released

[Emmanuel Engelhart]

Hi

We just have released a new version of Kiwix for iOS.

Here the list of improvements:
* NEW: Remember scroll positions
* FIX: Clear search & browsing history
* FIX: few other bugs

The app is available on iTunes at http://ios.kiwix.org.

Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication









signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] for iOS 1.8 released

[Emmanuel Engelhart]

Hi

We just have released a new version of Kiwix for iOS.

Here the list of improvements:
* NEW: Library has a new look
* NEW: Support zim files with build in index
* NEW: Performance optimization
* FIX: Download of big content on WIFI

The app is available on iTunes at http://ios.kiwix.org.

Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication







signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] for Android 2.1 released

[Emmanuel Engelhart]

Hi

We have released last week a new version of Kiwix for Android.

Here the list of improvements:
* FIX: Display of storage devices
* FIX: External SD cards detection
* FIX: ZIM files detection speed
* FIX: Core errors handling
* FIX: Save images
* FIX: ZIM file deletion
* FIX: Transparent action menu
* FIX: ZIM opening from external file browsers
* NEW: Pause/Resume TTS

The app is available on Google play at http://android.kiwix.org and the
APK can also be download at http://download.kiwix.org/bin/kiwix.apk

Happy new year
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication





signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [INFO] ZIM reference library 1.4 is out!

[Emmanuel Engelhart]

Hi

The new libzim 1.4 is out:
http://www.openzim.org/download/zimlib-1.4.tar.gz

This new minor release brings a few improvements and bugs fixes:
* NEW: speed-up article retrieval a bit
* NEW: read cluster content at the last moment to avoid unnecessary I/O
access
* NEW: primitive cache system for not-compressed clusters (multimedia
content)

More information about the project openZIM and the ZIM format at:
http://www.openzim.org/

Regards
Emmanuel Engelhart

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] 10th anniversary

[Emmanuel Engelhart]

Hi

10 years ago we have launched the Kiwix project to bring Wikipedia to
people with no or a bad access to Internet. Today, we are really proud
to see that we have achieved to do that, and even more.

All of this would not have been made possible without a committed team
of volunteers and the strong support of Wikimedia Switzerland. Thanks to
them (and a few other ones) we have achieved to:
* Make Wikipedia readable offline on (almost) all kind of devices
* Provide monthly updates of all Wikimedia projects
* Provide an open-source/standardised offline technology re-usable for
any kind of web content with openZIM
* Build offline versions of various projects like Gutenberg, TEDs,
Vikidia, Youtube, ...

With the additional help of dozens of partners, which spread content and
technology in the field, we can track around 1M of users a year... and
strongly believe this is the "tip of the iceberg".

Luckily, in the last 10 years, we have done a lot of progresses in the
Internet access. That said, around half of the world population still
does not have an access to Internet and censorship is a raising and
already extremely serious problem.

As a consequence, we will be pretty busy in the next decade and will
continue to improve our technology and offer always better solutions to
read and spread free knowledge - offline.

We have made a blog post about our anniversary on Wikimedia blog, please
share it on the social networks:
https://blog.wikimedia.org/2016/10/11/kiwix-ten-years/

Regards
Emmanuel Engelhart

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Fwd: [Wikimedia-l] Offline Medical Wikipedia in Persian

[Emmanuel Engelhart]

 Forwarded Message 
Subject: [Wikimedia-l] Offline Medical Wikipedia in Persian
Date: Sun, 14 Aug 2016 16:21:46 -0600
From: James Heilman 
Reply-To: Wikimedia Mailing List 
To: Wikimedia Mailing List 

Hey All

Today we launched an offline version of Wikipedia's medical content in
Persian. Thank's to the local community, Translators Without Borders,
Wikimedia CH, and my colleagues at Wiki Project Med for bringing this about.

https://play.google.com/store/apps/details?id=org.kiwix.kiwixcustomwikimedfa

More languages coming soon.
-- 
James Heilman
MD, CCFP-EM, Wikipedian

The Wikipedia Open Textbook of Medicine
www.opentextbookofmedicine.com
___
Wikimedia-l mailing list, guidelines at:
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
New messages to: wikimedi...@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,


___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] for iOS 1.7 released

[Emmanuel Engelhart]

Hi

Last week we have released the version 1.7!

What's new?
* Informative bookmarks
* New add / remove bookmarks interface
* Bookmarks Today Widget
* Bug fixes and performance improvements

Here is a small roadmap of the features we want to implement next
https://gist.github.com/automactic/53d62c09d56d0707936c1a82920c6b0a

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication





signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Fwd: [Wikimedia-l] Arabic version of Offline Medical App Launches

[Emmanuel Engelhart]

FYI


 Forwarded Message 
Subject: [Wikimedia-l] Arabic version of Offline Medical App Launches
Date: Tue, 2 Aug 2016 04:29:06 -0600
From: James Heilman 
Reply-To: Wikimedia Mailing List 
To: Wikimedia Mailing List 

Hey All

We have just launched the Arabic version of our offline medical app. It is
all of Arabic Wikipedia's healthcare, pharmaceutical, anatomy, and
sanitation articles.

https://play.google.com/store/apps/details?id=org.kiwix.kiwixcustomwikimedar

Only avaliable for Android at this point in time. Persian coming soon :-)

-- 
James Heilman
MD, CCFP-EM, Wikipedian

The Wikipedia Open Textbook of Medicine
www.opentextbookofmedicine.com
___
Wikimedia-l mailing list, guidelines at:
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
New messages to: wikimedi...@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,


___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] for iOS 1.6 released

[Emmanuel Engelhart]

Hi

Last week we have released the version 1.6!

Here the new fixes and features:
* NEW: Table of content
* NEW: Recent searches
* NEW: New (fulltext) search ranking system
* NEW: Enhanced layout javascript on iPhone
* NEW: Use SafariViewController to handle external links
* NEW: Access ZIM storage server using HTTPS
* NEW: Enhanced Search UI on iPad and iPhone 6/6s Plus (landscape)
* NEW: Clear search history in settings
* FIXED: Downloading / paused book no longer got purged when they are
removed from online library
* FIXED: Removed code that mistakenly indicates app use Wallet
* FIXED: Open main page when first book finish downloaded
* FIXED: Performance problem in book scan and search
* FIXED: App Launch speed
* FIXED: iPad recent search divider not drawn correctly when rotating
from portrait to landscape

Stay tuned, 1.7 is already in the pipe :)

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [NEW] Offline Phet Interfactive Simulations

[Emmanuel Engelhart]

Hi

I'm happy to announce that during our Kiwix hackathon @Wikimania Esino
Lario, we have achieved to build an offline version of the famous "PhET
Interactive Simulations for Science and Math":
https://phet.colorado.edu/

This is of course available in the ZIM format at this URL:
http://download.kiwix.org/zim/phet_en_all.zim

We will release monthly updates integrating that way the last published
new PhET simulations.

In addition, we have also released and will maintained a dedicated
Android app on the Google Play store, making its usage even easier:
https://play.google.com/store/apps/details?id=org.kiwix.kiwixcustomphet

All the source code is available in Kiwix source code repository at
https://github.com/kiwix/phet and the next version is already in the
pipeline.

Git it a try!

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [RELEASE] Kiwix for Android 1.97

[Emmanuel Engelhart]

Hi

The Kiwix dev. team is happy to announce the version 1.97 of Kiwix for
Android. Most of this release was coded by Google Code-in student
mhutti1, big thanks to him.

This release brings a few improvements:
* FIXED: Soft-keyboard opens on "find in page" click
* FIXED: "Back to Top" button fades out after 1.5 seconds
* NEW: App can be moved to SD card
* NEW: ARM64 support
* NEW: Beta full-text search (put the *.idx" directory beside the ZIM)
* NEW: Restart with last open article on exact same position
* NEW: Click on app title displays the search input box
* NEW: Remember fullscreen setting over app restarts
* NEW: Few additional languages supported for the UI
* NEW: Build against newer version of Android NDK/SDK

Kiwix for Android 1.97 is available on multiple app stores like:
* Google Play:
https://play.google.com/store/apps/details?id=org.kiwix.kiwixmobile
* Aptoid:
http://www.aptoide.com/search/view/apps?search=org.kiwix.kiwixmobile=0=0
* F-Droid:
https://f-droid.org/repository/browse/?fdfilter=kiwix=org.kiwix.kiwixmobile

... but also via direct download (interesting for people wanting old APK
versions) at http://download.kiwix.org/bin/android/

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Kiwix for iOS 1.2 released!

[Emmanuel Engelhart]

Hi

The Kiwix dev team is happy to announce a new release of Kiwix (1.2) for
iOS:
* Faster search (especially when having a lot of large ZIM files)
* Better iPad multitasking experience
* Bug fixes and other improvements

If you have a device with iOS8+, give a try to Kiwix for iOS! It's free
and probably already one of the best Wikipedia offline apps on iTunes.
http://ios.kiwix.org/

Enjoy
Kelson
-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [RELEASE] Kiwix for iOS v1.0

[Emmanuel Engelhart]

Hi Quim

Thank you for the proposition. We have meanwhile released the v1.1 and
the app has proven to "do the job". So, this is definitely a good idea.
I have myself pretty tight resources but have made a call on Twitter:
https://twitter.com/KiwixOffline/status/650697497451606017

Hopefully someone will be volunteer for this task.

Regards
Emmanuel

On 29.08.2015 17:18, Quim Gil wrote:
> Congratulations! Would you like to write a blog post about this milestone?
> 
> https://meta.wikimedia.org/wiki/Wikimedia_Blog
> 
> On Sat, Aug 29, 2015 at 2:49 PM, Emmanuel Engelhart <kel...@kiwix.org
> <mailto:kel...@kiwix.org>> wrote:
> 
> Hi
> 
> This was a long standing feature request and not the easiest one to
> implement, that's why the Kiwix development team is pretty proud to
> announce the release of Kiwix for iOS v1.0.
> 
> With Kiwix for iOS you can easily download/read recent snapshots (with
> or without pictures) of all Wikimedia projects with your iPhone/iPad.
> This first version of the app provides basic functionalities like:
> * ZIM reader
> * Navigation through articles
> * Search based on article title
> * Content library/downloader
> * Bookmarks
> * Browse history
> * iTunes sync (already in beta testing for v1.1)
> 
> The app has been developed for the last 18 months by Chris, a New-York
> based developer. He was (a bit) mentored by an older member of the Kiwix
> dev team and the project was supported by WikimediaCH for the
> administrative work with Apple and the coverage of mandatory
> hardware costs.
> 
> With this last stone, Kiwix which is already available for Windows, OSX,
> Linux and Android, has reached a milestone. 10 years after the project
> creation, Kiwix is a a cutting-edge solution to access Wikipedia (and a
> lot more) offline - with most of the computers - whatever the platform.
> 
> Download Kiwix for iOS on Itunes:
> https://itunes.apple.com/us/app/kiwix/id997079563
> 
> Have a look to the source code:
> https://github.com/kiwix/iOS
> 
> Regards
> Emmanuel
> 
> --
> Kiwix - Wikipedia Offline & more
> * Web: http://www.kiwix.org
> * Twitter: https://twitter.com/KiwixOffline
> * more: http://www.kiwix.org/wiki/Communication
> 
> 
> ___
> Offline-l mailing list
> Offline-l@lists.wikimedia.org <mailto:Offline-l@lists.wikimedia.org>
> https://lists.wikimedia.org/mailman/listinfo/offline-l
> 
> 
> 
> 
> -- 
> Quim Gil
> Engineering Community Manager @ Wikimedia Foundation
> http://www.mediawiki.org/wiki/User:Qgil
> 
> 
> ___
> Offline-l mailing list
> Offline-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/offline-l
> 


-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Kiwix 1.1 for iOS released

[Emmanuel Engelhart]

Hi

A few weeks after the Kiwix for iOS launch, here we come with the first
upgrade, Kiwix 1.1 for iOS:
https://itunes.apple.com/us/app/kiwix/id997079563

What's new in version 1.1
* IOS 9.0 support
* iPad support
* ZIM file transfer using iTunes File Sharing
* Many small UI improvements

Happy offline reading!

Emmanuel

-- 
Kiwix - Wikipedia Offline & more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [RELEASE] Kiwix for iOS v1.0

[Emmanuel Engelhart]

Hi

This was a long standing feature request and not the easiest one to
implement, that's why the Kiwix development team is pretty proud to
announce the release of Kiwix for iOS v1.0.

With Kiwix for iOS you can easily download/read recent snapshots (with
or without pictures) of all Wikimedia projects with your iPhone/iPad.
This first version of the app provides basic functionalities like:
* ZIM reader
* Navigation through articles
* Search based on article title
* Content library/downloader
* Bookmarks
* Browse history
* iTunes sync (already in beta testing for v1.1)

The app has been developed for the last 18 months by Chris, a New-York
based developer. He was (a bit) mentored by an older member of the Kiwix
dev team and the project was supported by WikimediaCH for the
administrative work with Apple and the coverage of mandatory hardware costs.

With this last stone, Kiwix which is already available for Windows, OSX,
Linux and Android, has reached a milestone. 10 years after the project
creation, Kiwix is a a cutting-edge solution to access Wikipedia (and a
lot more) offline - with most of the computers - whatever the platform.

Download Kiwix for iOS on Itunes:
https://itunes.apple.com/us/app/kiwix/id997079563

Have a look to the source code:
https://github.com/kiwix/iOS

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Last news about Kiwix for Android

[Emmanuel Engelhart]

Hi

Last June, the Kiwix team has organised a hackathon focused on mobile
technologies in general and Android in particular.
http://www.kiwix.org/wiki/Hackathon_Mobile_2015

Sponsored by Wikimedia Switzerland, this one week long event has been
the occasion to do a few steps forward thx to the hard work of the 4
volunteer developers.

As a result, we are proud to announce the release of Kiwix for Android
1.94/1.95 which brings a lot of improvements. Among others:
* NEW: Revamped tabs system
* NEW: Revamped zoom system
* NEW: Audio/Video suport
* NEW: Updated Ray Charles ZIM demo file
* FIXED: AutoCompleteTextView in landscape mode
* FIXED: PDF  EPUB viewing
* FIXED: Many bugs related to night mode
* REMOVED: Gingerbread support
* REMOVED: Share with friends
* REMOVED: Zoom button

Install in one click on your device:
https://play.google.com/store/apps/details?id=org.kiwix.kiwixmobile

We have also created a new set of tools allowing us to generate and
publish quickly Android Bundle Apps (app+content). These apps are really
useful for users because in one click in the Google Play Store they have
what they want. We have started to use these tools and have already
released two high value apps:
* Wikivoyage in English:
https://play.google.com/store/apps/details?id=org.kiwix.kiwixcustomwikivoyage
* Wikimed in English:
https://play.google.com/store/apps/details?id=org.kiwix.kiwixcustomwikimed

An other consequence of this bundling software solution is that it is
now trivial for NGO  Companies to distribute devices with free content
pre-packaged on it.

Regards
Emmanuel
-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



signature.asc
Description: OpenPGP digital signature
___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Monthly ZIM dumps?

[Emmanuel Engelhart]

Dear Gordon

On 25.07.2015 01:38, Gordon Mohr wrote:

The 2015-05 enwiki nopic dump is a great resource for getting bulk
article text – much better in my experience than using scripts that
try to strip it out of XML dumps, or wrestling with a full MW+Parsoid
system.


Thank you. You use it for a research purpose?


I see threads from earlier in the year that the goal is monthly ZIM dumps.

Any projections for when that might be achieved, or perhaps just when
the process that succeeded in creating the 2015-05 dump(s) might be
repeated as another one-off?


Fixing that problem is my top-priority and we are getting better and 
better. Something you can see by yourself if you look at 
http://download.kiwix.org/zim/. Unfortunately we deal with limited 
hardware resources and the software solution to do these snapshots 
(mwoffliner) is still a little bit buggy.


WPEN being the worse snapshot to generate, it is also the one which 
suffers the most of these problems.


That said, I think we will achieve full monthly updates in the next 
months and I plan a new snapshot of WPEN in August (anyway).


Kind regards
Emmanuel

--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Adding a Tags metadata to ZIM files

[Emmanuel Engelhart]

Hi,

We have more and more ZIM files available for download and we need way 
to sort/differentiate them.


One of the concrete problems we face for example is that we provide at 
Kiwix a full offline version *with pictures and one without pictures. 
We have adopted a filename scheme allowing to differentiate them but 
this does not appear transparently in the content managers. As a 
consequence people are puzzled to see the same content twice (with 
different file size of course). Adding a nopic tags would allow 
readers to react on it adequately.


That's why I'm currently considering the addition of a Tags metadata 
entry in the ZIM format specification. To be more specific, here:

http://www.openzim.org/wiki/Metadata

Do you think that the right approach?

Regards
Emmanuel
--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [youtube2zim] a new tool to create ZIM fiels

[Emmanuel Engelhart]

Hi

Dattaz, a volunteer developer, has developed a python script to quickly 
build a ZIM file from a Youtube Channel or Playlist. The code is here:

https://sourceforge.net/p/kiwix/other/ci/master/tree/youtube/

His work was inspired by our ZIM export of the TED talks and allows us 
to make off-line versions of a large collections of videos.


We have already created a few demo files with youtube2zim:
* http://download.kiwix.org/zim/other/bil-tunisia_ar_all_2015-03.zim
* http://download.kiwix.org/zim/other/deus-ex-silicium_fr_all_2015-03.zim
* http://download.kiwix.org/zim/other/universcience-tv_fr_all_2015-03.zim

We have also a few improvements in view: like adding the support of 
Dailymotion or recreate the playlists in case of a whole channel snasphost.


Wikimedia Commons is unfortunately still not well recognised as a 
platform to host videos. However, we have a project to make a ZIM file 
of the best Commons pictures. If you are interested please subscribe 
and/or award a token on this blocking task:

https://phabricator.wikimedia.org/T87571

Thanks again to Dattaz for this really useful work.

Kind regards
Emmanuel
--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Fwd: Reasons you use the XML dumps or want to, but can't?

[Emmanuel Engelhart]

Hi

Thank you Nemo for adverting that interesting page about how to improve 
Wikimedia dumping processes. This topic is of course a primary concern 
for the Kiwix developer team.


Here my contribution:
https://www.mediawiki.org/w/index.php?title=Wikimedia_MediaWiki_Core_Team%2FBacklog%2FImprove_dumpsdiff=1417187oldid=1415717

Hope to see things going forward on this, I will help as much as I can.

Regards
Emmanuel

On 21.02.2015 08:44, Federico Leva (Nemo) wrote:

FYI


 Messaggio inoltrato 
Oggetto: [Xmldatadumps-l] Your comments needed (long term dumps
rewrite?)
Data: Thu, 19 Feb 2015 12:30:01 +0200
Mittente: Ariel Glenn WMF ar...@wikimedia.org
A: xmldatadump...@lists.wikimedia.org



The MediaWiki Core team has opened a discussion about getting more
involved in and maybe redoing the dumps infrastructure.  A good starting
point is to understand how folks use the dumps already or want to use
them but can't, and some questions about that are listed here:
https://www.mediawiki.org/wiki/Wikimedia_MediaWiki_Core_Team/Backlog/Improve_dumps

I've added some notes but please go weigh in.  Don't be shy about what
you do/what you need, this is the time to get it all on the table.

Ariel




___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l




--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Kiwix for iOS developers/testers need your help!

[Emmanuel Engelhart]

Hi,

A few people are currently working hard to release a first version of
Kiwix for iOS. Most of the Kiwix users don't have this kind of devices
which is more dedicated to wealthy people, but this is important to
support iOS to have a complete portfolio.

Unfortunately, we currently suffer of a lack of Apple devices:
* devices (iPhone or iPad) with iOS7 or greater
* OSX computers (for development purpose) not more than 3 years old.

If you want to make an hardware donation or have a second hand devices 
you might sell for a cheap price, please let us - this would really help 
us a lot!


Thank you in advance for your help.

Kind regards
Emmanuel
--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Quick Kiwix stats

[Emmanuel Engelhart]

Hi Nemo, Hi Wilfredo

The numbers on http://stats.kiwix.org/ are good. The peaks are the 
consequences of advertisement campaigns on third part web sites, like 
wikipedia.org but not only.


Regards
Emmanuel

On 24.11.2014 19:23, Wilfredor wrote:

Really nice stats, with a number of views, hard to believe in November 2012.

2014-11-24 15:47 GMT-02:00 Federico Leva (Nemo) nemow...@gmail.com
mailto:nemow...@gmail.com:

As a reminder, a lot of stats are accessible at stats.kiwix.org
http://stats.kiwix.org.
I made a short summary at
https://meta.wikimedia.org/?__oldid=10630028#Offline___Dissemination_Program
https://meta.wikimedia.org/?oldid=10630028#Offline_Dissemination_Program
See also two quick pre-made charts at
*

https://upload.wikimedia.org/__wikipedia/commons/b/b4/Kiwix___downloads_map_2014-10.png

https://upload.wikimedia.org/wikipedia/commons/b/b4/Kiwix_downloads_map_2014-10.png
*

https://upload.wikimedia.org/__wikipedia/commons/3/36/Kiwix___downloads_2012-2014.png

https://upload.wikimedia.org/wikipedia/commons/3/36/Kiwix_downloads_2012-2014.png

Nemo

_
Offline-l mailing list
Offline-l@lists.wikimedia.org mailto:Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/__mailman/listinfo/offline-l
https://lists.wikimedia.org/mailman/listinfo/offline-l




___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l




--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX][GUTENBERG] 50.000 public domain books available to everybody, everywhere, offline

[Emmanuel Engelhart]

Hi,

The Kiwix team is happy to release the whole Project Gutenberg 
(http://www.gutenberg.org/) library in a ZIM format: 
http://download.kiwix.org/zim/gutenberg/gutenberg_mul_all_2014-11.zim.torrent. 
We also provide a few language specific versions here 
http://download.kiwix.org/zim/gutenberg.


This file is dedicated to an offline usage (no connection to Internet) 
and it readable with Kiwix (http://www.kiwix.org). This allows anybody 
with a computer or a smartphone to own his own copy of this 50.000 books 
big library. You can also make it available for read to other people on 
your network, they only need a web browser.


In this ZIM file, you will find all the books available in HTML 
(directly readable), but also in EPUB (and time to time in PDF). We have 
created a custom user interface which is really simple to use: in a few 
clicks you can find your book, read it or download it. What is also 
unique is that Kiwix proposes a fulltext search engine over all books 
content. You can see by yourself using this demonstration web site: 
http://library.kiwix.org/gutenberg_mul_all_2014-11/


Most of the work was done during a week long hackathon in Lyon, France 
by four Kiwix volunteer developers. This hackathon was funded by the 
Fondation Orange with the administrative help of Framasoft and Wikimedia 
CH. The Fondation Orange is the first beneficiary of this work and use 
it already for its own deployments in Africa.


The solution to build this ZIM file is 100% free software and is 
available here https://github.com/kiwix/gutenberg. This solution allows 
to release easily new up2date versions. This is not a one shot project 
and we will release periodically new version of this offline version of 
the Project Gutenberg.


We also plan to use this code base to aggregate other online PD/free 
books libraries. Wikisource is one of the first we would love to add, 
this might be done pretty easily as soon as an OPDS feed is available.


We hope to see this work deployed by other third part organisation which 
are on place where Internet is not available, expensive or censored. We 
also need more developer (mostly Python) workforce for the next steps, a 
hackathon with this purpose will hopefully be organised in 2015 (sponsor 
needed). Last but not least: users, please report any problem here: 
https://github.com/kiwix/gutenberg/issues


Regards
Emmanuel
--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [KIWIX] New Version 1.91 for Android available

[Emmanuel Engelhart]

Dear Kiwix users

A new version of Kiwix for Android is available:
https://play.google.com/apps/testing/org.kiwix.kiwixmobile

You can also download it directly here:
http://download.kiwix.org/bin/android/kiwix-1.91.apk

This is a minor release, here is the changelog:
* FIXED: Better support of Indian languages
* FIXED: Many crash scenarios
* NEW: Scan the external SD card
* NEW: Bookmarking feature
* KNOWN BUG: Text rewrapping problem with 4.4

Please report any problem at:
http://reportabug.kiwix.org

Regards
Emmanuel
--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Bug day: Book tool/Collection/PDF, 2014-10-08, 14–22 UTC

[Emmanuel Engelhart]

On 09.10.2014 00:35, Federico Leva (Nemo) wrote:

And it's over! We reached our immediate goal, closing all the lost
PediaPress tickets (80 before the bug day); and about 40 new bugzilla
reports were filed, including some tricky ones about language support.

https://www.mediawiki.org/wiki/Bug_management/Triage/201410 has a dump
from the etherpad and will be updated with more precise information.
Give a look at the bugzilla reports!

Thanks to all the participants: Andre Klapper, Anomie, Azitrex, bawolff,
Helder, John Vandenberg, Kelson, Nemo, Raymond, Reedy, Rupert, saper,
Steinsplitter, csteipp, ebraminio, jem, josu, kepper, valhallasw`cloud.
(And probably someone else in bugzilla at the same time.)


Thank you Nemo for having organised this!

Emmanuel

--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [WikimediaMobile] Thinking about offline search within our native apps

[Emmanuel Engelhart]

On 01.10.2014 10:44, renaud gaudin wrote:

Full text search engine was not left out for performance reasons but for
practical ones.
Yes, we don't want people to generate the index on their phone. Except
for tiny tiny zim files, it would be too much CPU, battery and time
consuming.
But we do want to add full text search to Android.
It could work today but the search index is a (large) folder so it would
be a pain to setup.
As soon as we integrate both the ZIM file and the index in a single
file, we'll enable full text search on the Android App.


To complete Renaud's comment, here is the dev strategy to get it done:

1 - Modify the libxapian code to make it able to pack the index in one 
file and open it with a filehandle  offset. Here is a small technical 
discussion about this: 
http://lists.xapian.org/pipermail/xapian-discuss/2013-November/009039.html


2 - Modify current solution to integrate the search index in a ZIM file.

3 - Introduce the libxapian (Kiwix fulltext search engine) to the binary 
lib (libkiwix). Here is the compilation script:

https://sourceforge.net/p/kiwix/kiwix/ci/master/tree/android/build-android-with-native.py

4 - Update the JNI interface ans the Android UI.

Like told by Charles, this is on in the 2015 Roamap:
https://meta.wikimedia.org/wiki/Wikimedia_CH/2015_annual_plan#Offline_Dissemination_program

Emmanuel
--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [WikimediaMobile] Thinking about offline search within our native apps

[Emmanuel Engelhart]

On 10/01/2014 10:56 AM, rupert THURNER wrote:

how is this done in epub format?


The EPUB format does not contain any fulltext search index.

Emmanuel


___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Kiwix in Brasil

[Emmanuel Engelhart]

Hi Wilfredo

Great!
https://twitter.com/KiwixOffline/status/493879409137680384

Thank you!
Emmanuel

On 06.07.2014 22:21, Wilfredor wrote:

I would like to know that I am working with a project in Brazil to
provide Kiwix in schools. Finding a way to expand the knowledge gained
in the program for schools Venezuela.

I'm doing activism together with the WMF. Yesterday, a query to
strengthen Wikimedia Brazil was done and Kiwix was one of the topics of
interest
https://commons.wikimedia.org/wiki/File:Consulta_2014_sobre_como_fortalecer_a_Wikimedia_no_Brasil,_S%C3%A3o_Paulo_02.jpgwith
others.

I leave the link of the program,
https://br.wikimedia.org/wiki/Wikipedia_para_pueblos_de_Brasil




___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l




--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Posters about Kiwix by Wi-Fi

[Emmanuel Engelhart]

Hi

On 26.06.2014 18:00, Seb35 wrote:

After porting Kiwixi on Raspberry Pi, I presented the whole system during
the inauguration of the new WMFR’s offices two weeks ago, and I present it
this Saturday on a small hackfest for general public in my local
hackerspace.

For these two events I’ve created a small poster mainly to appeal the
public to this small device and explain the thing.

You can see on the photo that the system is entirely autonomous with the
battery (about 7 hours of functionning with Wi-Fi), which is kind to
walk with it even without power plug.

Poster (2xA5, in French, if you want the sources in SVG ask me):
https://commons.wikimedia.org/wiki/File:Wikipedia_hors-ligne_2xA5.pdf

Photo of the device:
https://commons.wikimedia.org/wiki/File:Raspberry_Pi_with_a_battery.JPG


This move is a pretty important one for Kiwix :

* It allows to make Wikipedia kiosk wireless/wireline with a full copy 
of Wikipedia for less than 100 USD (HW costs). This can be done with 
only a small amount of tech. knowledge: have a look to the kiwix-plug page:

http://www.kiwix.org/wiki/Kiwix-plug

* This is the first step to make fully autonomous (disconnected of the 
power grid) WIFI hotspots. Development of a standardized solar powered 
solution will start soon under the name of kiwix-solar.


So, thanks a lot to Seb35 for all his good work.

Emmanuel

--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [Xmldatadumps-l] Help for dump

[Emmanuel Engelhart]

On 06/02/2014 02:18 PM, Federico Leva (Nemo) wrote:

Yannick Guigui, 02/06/2014 10:59:

It’s about Dump of Wikipedia. I’m working in a project which uses
Wikipedia Database to display articles offline.


Nice! But please don't use the raw database + images. What you need is
already available: http://kiwix.org
If it's a problem for you to get about 42+22 GiB (en+fr) I or someone
can mail you a couple SDs at a cost of ~25 €. (If interested reply
offlist to organise details.)


This is becoming obvious to me: we have easy and efficient solutions 
with Kiwix but we fail to advert them correctly. Thank Nemo for trying 
to fill the gap.


Emmanuel

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] All TED talks available offline on your computer

[Emmanuel Engelhart]

Hi,

For the first time, we release a copy of all TED videos in the ZIM 
format, thus available for reading with Kiwix and any other ZIM reader. 
We have created 6 ZIM files, one per topic:

* http://download.kiwix.org/zim/ted/

These files provide from 250 to 500 videos each. It's also possible to 
enjoy the videos with subtitles in many languages. You can see how it 
looks like on our demo server:

* http://library.kiwix.org/ted_business_04_2014/
* http://library.kiwix.org/ted_design_05_2014/
* http://library.kiwix.org/ted_global_issues_05_2014/
* http://library.kiwix.org/ted_technology_05_2014/
* http://library.kiwix.org/ted_science_05_2014/
* http://library.kiwix.org/ted_entertainment_05_2014/

This is still a little bit experimental, there is work left to achieve 
to have a perfect support of this kind of multimedia content:
* Add support of HTML5 videos on Android (Java, 
https://sourceforge.net/p/kiwix/bugs/700/)
* Add HTTP byte range requests to http-serve (C++, 
https://bugzilla.wikimedia.org/show_bug.cgi?id=47405)
* Allow XMLHttpRequest() with ZIM files in Kiwix (javascript, 
https://sourceforge.net/p/kiwix/bugs/700/)


But, please test it and report any probalem at:
http://reportabug.kiwix.org

Most of the work was done by Ahmad (which is the Kiwix for Android main 
developer too) during a two weeks long internship at WikimediaCH.


The code to create this ZIM files is opensource and it's easy to prepare 
new version, so we will release time to time updates:

https://sourceforge.net/p/kiwix/other/ci/master/tree/TED/

We want now to prepare something similar for all the TEDx talks, any 
developer with Python skills is welcome to help.


Regards
Emmanuel
--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [OPENZIM] libzim 1.2 is out!

[Emmanuel Engelhart]

Hi,

We have released the version 1.2 of the zimlib
http://openzim.org/download/zimlib-1.2.tar.gz

This is a minor release, here comes the changelog:
* Do not try to remap mime types for redirects or other types (#52324)
* Remove duplicate decl. of Dirent:: redirectMimeType  linktarget
* Make it compiles with Microsoft VS2010 environment
* Make it compiles with Apple OSX 10.9 environment

The zimlib is the standard implementation of the ZIM specification. It 
is a library which implements the read and write method for ZIM files. 
Use the zimlib in your own software - like reader applications - to make 
them ZIM-capable without the need having to dig too much into the ZIM 
file format.


Regards
Emmanuel
--
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] The Whole Wikipedia in English with pictures in one 40GB big file

[Emmanuel Engelhart]

Le 07/03/2014 19:25, Asaf Bartov a écrit :
 btw, are these new improved tools documented anywhere? 
 http://kiwix.org/wiki/Development does not seem to point in the right
 direction.

The usage is pretty straightforward (for IT people) and IMO everything
necessary is explained in the READMEs:
* mwoffliner:
https://sourceforge.net/p/kiwix/other/ci/master/tree/mwoffliner/
* zimwriterfs:
https://sourceforge.net/p/kiwix/other/ci/master/tree/zimwriterfs/

NB: The goal is not that everybody creates its own full wikipedia ZIM
file. The goal is that we (Wikimedia) provide these files, often enough
to always have up2date ZIM information (so at least one time per month).
Thus, the challenge is now to setup an infrastructure similar to the one
which creates the XML dumps.

Emmanuel

PS: We really want to make a post @blog.wikimedia.org (so in English).
If someone is volunteer to write this, I would really appreciate his help.
-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] The Whole Wikipedia in English with pictures in one 40GB big file

[Emmanuel Engelhart]

Hi

For the first time, we have achieved to release a complete dump of all
encyclopedic articles of the Wikipedia in English, *with thumbnails*.

This ZIM file is 40 GB big and contains the current 4.5 million articles
with their 3.5 millions pictures:
http://download.kiwix.org/zim/wikipedia_en_all.zim.torrent

This ZIM file is directly and easily usable on many types of devices
like Android smartphones and Win/OSX/Linux PCs with Kiwix, or Symbian
with Wikionboard.

You don't need modern computers with big CPUs. You can for example
create a (read-only) Wikipedia mirror on a RaspberryPi for ~100USD by
using our ZIM dedicated Web server called kiwix-serve. A demo is
available here: http://library.kiwix.org/wikipedia_en_all/

Like always, we also provide a packaged version (for the main PC
systems) which includes fulltext search index+ZIM file+binaries:
http://download.kiwix.org/portable/wikipedia_en_all.zip.torrent

What is interesting too: This file was generated in less than 2 weeks
thanks to multiples recent innovations:
* The Parsoid (cluster), which gives us an HTML output with additional
semantic RDF tags
* mwoffliner, a nodejs script able to dumps pages based on the Mediawiki
API (and Parsoid API)
* zimwriterfs, a solution able to compile any local HTML directory to a
ZIM file

We have now an efficient way to generate new ZIM files. Consequently, we
will work to industrialize and automatize the ZIM file generation
process, one thing which is probably the most oldest and important
problem we still face at Kiwix.

All this would not have been possible without the support:
* Wikimedia CH and the ZIM autobuild project
* Wikimedia France and the Afripedia project
* Gwicke from the WMF Parsoid dev team.

BTW, we need additional developer helps with javascript/nodejs skills to
fix a few issues on mwoffliner:
* Recreate the table of content based on the HTML DOM (*)
* Scrape Mediawiki Resourceloader in a manner it will continue to work
offline (***)
* Scrape categories (**)
* Localized the script (*)
* Improve the global performance by introducing usage of workers (**)
* Create nodezim, the libzim nodejs binding and use it (***, need also
compilation and C++ skills)
* Evaluate necessary work to merge mwoffliner and new WMF PDF Renderer (***)

Emmanuel
-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [Wikitech-l] The Whole Wikipedia in English with pictures in one 40GB big file

[Emmanuel Engelhart]

Le 01/03/2014 19:26, James Forrester a écrit :
 On Saturday, March 1, 2014, Emmanuel Engelhart kel...@kiwix.org wrote:
 fix a few issues on mwoffliner:
 * Recreate the table of content based on the HTML DOM (*)
 
 We are currently working on doing similar work to this in VisualEditor (to
 provide for a Table of Contents that can change 'live' as the document is
 edited); this code may ultimately be used to generate the real Tables of
 Contents for the reading HTML, as part of the plans to replace the output
 of the PHP parser with Parsoid everywhere.
 
 It should be possible for Kiwix to re-use this in some way (rather than
 have to re-implement it!). We hope to have something to show in the next
 few weeks, if that's helpful.

Nice news!

Yes, indeed, this would be great to be able to re-use your work.

I have subscribed this Bugzilla entry, which is guess what you mean:
https://bugzilla.wikimedia.org/show_bug.cgi?id=49224

Of course, for our usage, the best solution would be to have the initial
TOC rendering done on Parsoid side, which would also offer the advantage
of speeding-up initial VE rendering.

Emmanuel

-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] kiwix beta

[Emmanuel Engelhart]

I have started to release time to time (when new features are
implemented) beta versions of Kiwix for Android on Google Play.

It's still an experiment to see if this bring any additional value to
the development process.

To benefit from this beta versions, you need to join the Google+
kiwix-testing group (sorry for that, but this is Google politics):
https://plus.google.com/communities/106410746144853739922

Le 30/12/2013 18:44, rupert THURNER a écrit :
 that is excellent news, emmanuel, you ment 1.9 coming now?
 
 rupert.
 
 On Fri, Dec 20, 2013 at 1:27 AM, Emmanuel Engelhart kel...@kiwix.org wrote:
 Hi Rupert,

 You are right, we will use this feature of the Android Play store for
 release 1.8.

 Regards
 Emmanuel

 Le 17/12/2013 10:15, rupert THURNER a écrit :
 hi,

 would it be possible to have a second upload to google play store
 called kiwix beta to allow easier testing of new features.
 advantages:
 * no relaxing of the android phone security to install an apk from
 untrusted source
 * easy state the new features on the play page
 * automatic update notification to people interested in testing the new 
 version

 rupert.

 ___
 Offline-l mailing list
 Offline-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/offline-l



 --
 Kiwix - Wikipedia Offline  more
 * Web: http://www.kiwix.org
 * Twitter: https://twitter.com/KiwixOffline
 * more: http://www.kiwix.org/wiki/Communication

 ___
 Offline-l mailing list
 Offline-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/offline-l
 
 ___
 Offline-l mailing list
 Offline-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/offline-l
 


-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Testers needed for new Kiwix 0.9rc3 (Windows/OSX/Linux/Sugar) release

[Emmanuel Engelhart]

Hi

We are almost ready to release a new version of Kiwix (for 
Windows/OSX/Linux/Sugar). This release will be called 0.9rc3 and 
provides a few fixes/improvements to the current 0.9rc2.


The most important fixes/improvements are:
* Global improvement of kiwix-serve
* Better data structure (easier to move data files)
... complete changelog is available here: http://changelog.kiwix.org

We really need testers to verify that we don't have some regressions and 
that new features work correctly. You can find nightly builds at:

* http://download.kiwix.org/nightly/bin/latest/
* http://download.kiwix.org/nightly/bin/

Kind regards
Emmanuel

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] download or create a book on android-4.1, chrome, not possible

[Emmanuel Engelhart]

Le 17/12/2013 23:48, rupert THURNER a écrit :
 hi, sorry for crossposting this, but i am a little lost whom to ask.
 erik mentioned some time ago how important zim support is in
 collections, and mentioned that there are barely downloads. very
 rarely i get my hands on a new enought android version where kiwix
 runs. and i tried to (1) create a book, and (2) download a book. both
 failed.
 
 1. create a book
 the mobile skin is lacking the book or add page to book link.
 calling http://en.wikipedia.org/wiki/Special:Book does not help
 either, there is no obvious way to add pages to a book.
 
 2. download a book
 go to a book like
 https://en.wikipedia.org/wiki/Wikipedia:Books/Derivatives. the mobile
 skin again is broken and the download link is crippled. if one is
 lucky to hit the zim download link anyway, it switches over to a
 non-mobile link to generate the book. it really seems to generate the
 book, and displays a link. clicking it then downloads a 6.5 mb big
 index.php and says failed.

The collection extension UI is not adapted for mobile usage. The best to
do now is to open a feature request at http://bugzilla.wikimedia.org

Emmanuel

-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Release of Kiwix for Android v.16

[Emmanuel Engelhart]

Hi,

last week we have released a new version of Kiwix for Android which
includes a few new features. Most of them were coded by young students
participating to the Google Code-in program:

* New share with friends button
* New fullscreen feature
* New 'back to top button
* Better search suggestions
* Small UI fixes

Enjoy:
https://play.google.com/store/apps/details?id=org.kiwix.kiwixmobile

Regards
Emmanuel
-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Best script for static html of a mediawiki

[Emmanuel Engelhart]

Le 19/11/2013 19:13, Bjoern Hassler a écrit :
 I'd like to ask a follow-up question to this. What do you guys use for
 converting html to pdf (on the command line)? Let's say once Parsoid
 has generated html5, how do you get it into pdf? I've used (and
 struggled) a bit with wkpdf and enscript to generate page numbers - is
 that the best solution around?

phantomjs PDF rasteriser
https://coderwall.com/p/5vmo1g

-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Get images of article.

[Emmanuel Engelhart]

Hi Yannick

Le 08/11/2013 13:01, Federico Leva (Nemo) a écrit :
 Yannick Guigui, 08/11/2013 12:22:
 I'm camerounian I built a webapp whose allows students to consult
 wikipedia articles

I was in Cameroun 3 weeks ago to promote Kiwix technologies. I can give
you some local contacts which should be able to deliver you
software/content and probably some help. Where are you located exactly?

Regards
Emmanuel

-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [HELP NEEDED] Javascript (beginner level) developer for a small feature request

[Emmanuel Engelhart]

Wilfredo has fixed everything... and more.

The result is there:
http://www.kiwix.org/m/

Let me know if you still see a bug.

Emmanuel

Le 09/10/2013 13:38, Emmanuel Engelhart a écrit :
  Message original 
 Sujet: [HELP NEEDED] Javascript (beginner level) developer for a small
 feature request
 Date : Wed, 09 Oct 2013 12:47:00 +0200
 De : Emmanuel Engelhart kel...@kiwix.org
 Pour : kiwix-develo...@lists.sourceforge.net
 kiwix-develo...@lists.sourceforge.net
 
 Hi,
 
 I would appreciate any help for a small javscript developement:
 https://sourceforge.net/p/kiwix/feature-requests/767/
 
 This is a feature request gathering many small improvements which should
 be done on http://www.kiwix.org/m/, a download page which is proposed to
 Android users to download contents.
 
 Here is the list of improvements:
 -- Replace Kiwix ZIM Contents and Content Downloads Filter with only
 one title like Download content for Kiwix
 -- Remove Content Type select box (we don't have enough content for
 now to justify such type of filtering)
 -- Language selectbox should be automatically filled with the list of
 languages available at http://www.kiwix.org/wiki/Template:ZIMdumps
 (first column)
 -- Language select box entry Auto-detect should be removed and
 autodetect should be applied at loading base on browser agent language
 -- Replace Download Link label by Download method
 -- Replace Download link select box by two radio buttons-- Remove
 Filter button, each change on Download method or Language should
 automatically apply the filter and reload the results
 -- Block the page during each loading of the result (modal dialog box)
 or darken the whole page
 -- Make the filtering working for all languages, Hindi for example
 does not work.
 
 The source code is available at:
 https://sourceforge.net/p/kiwix/web/ci/master/tree/mobile-folder/
 
 Any volunteer?
 
 Kind regards
 Emmanuel
 


-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Google Code-in: are you in?

[Emmanuel Engelhart]

Dear Quim

Le 22/10/2013 20:04, Quim Gil a écrit :
 Google Code-In is a contest to introduce pre-university students (ages
 13-17) to the many kinds of contributions that make open source software
 development possible. Students must complete tasks, one at a time. The
 Google Code-in 2013 contest runs from November 18, 2013 to January 6, 2014.
 
 https://www.mediawiki.org/wiki/Google_Code-In
 
 Do you have small tasks sitting in your backlogs? For instance (and
 pointing to the Kiwix project), we have been told that Android
 applications is an area where a lot of contributions from skilled
 students are almost granted.

I have tried to gather a few tasks:
* Share button, to help happy users to spread the word using
chat/email/social networks.
* Share this text selection, to allow to share a pre-selected text with
friends using chat/email/social networks.
* Fullscreen mode, to be able to fully enjoy Wikipedia articles.
* Go back to top, an overlay button displayed a the end of any article
to be able to quickly come back to it's top.
* Add a simply history viewer, to allow in two clicks to travel back to
a random article.
* Add tabs, to allow to open many articles at the same time.

Detailed view is here:
https://www.mediawiki.org/wiki/Google_Code-In#Kiwix_for_Android

Thank you for think to Kiwix for this and I whish us all a good luck for
the next stage ;)

Regards
Emmanuel
-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Fwd: Wikipedia for Schools new version

[Emmanuel Engelhart]

In addition: Wikipedia for Schools is already packaged with Kiwix:
* Portable version (Kiwix+ZIM+fulltext index): 
http://download.kiwix.org/portable/wikipedia_en_for_schools.zip.torrent
* ZIM file: 
http://download.kiwix.org/zim/wikipedia_en_for_schools.zim.torrent

* Online demo: http://library.kiwix.org/wikipedia_en_for_schools/

Emmanuel

On 10/15/2013 06:28 PM, Jessie Wild wrote:

See below regarding the updated Wikipedia for Schools. This is an
important tool for offline use, particularly in schools.

If you have any use cases and want to share a quote or story, please
pass them along to Jamie! I know there are some good ones out there!

-- Forwarded message --
From: *Jamie Goodland* jamie.goodl...@soschildrensvillages.org.uk
mailto:jamie.goodl...@soschildrensvillages.org.uk
Date: Tue, Oct 15, 2013 at 10:51 AM
Subject: Wikipedia for Schools new version
To:


Hello all

I'm emailing you today because you've expressed interest in Wikipedia
for Schools in the past. Please forgive me if I've already been in touch
individually - I'm emailing to the full mailing list today.

I'm very excited to announce that we've just updated Schools Wikipedia
for 2013. The new version is much expanded, containing roughly 6000
articles, 50,000 images and consisting of 26 million words! And although
it's still organised around the UK national curriculum, and still
hand-checked for suitability, it boasts a brand new look which we're
really rather proud of.

There are two ways you can access it:

  * Browse the online version http://schools-wikipedia.org/
  * Alternatively, you can download a copy

http://www.sos-schools.org/wikipedia-for-schools/wikipedia-schools-3.0.0.torrent
for use offline.

You can also find more information about the project, including its
purpose and history, here
http://www.sos-schools.org/wikipedia-for-schools on our schools website.

If anyone has anything nice to say - about this version or a previous
edition - that we can quote in press, I'd be really grateful - as much
or as little as you have time for.

Please do pass circulate this email to anyone you feel may benefit. And,
of course, if you have any questions whatsoever, please don't hesitate
to contact me.

Many thanks
--
Jamie Goodland
Website Editor
SOS Children

A loving home for every child
Registered Charity Number: 1069204
Tel: +44 (0) 1223 365589 tel:%2B44%20%280%29%201223%20365589


http://www.soschildrensvillages.org.uk/images/email/current-footer-link

Find out more about SOS Children
http://www.soschildrensvillages.org.uk/images/email/current-footer-link




--
/Jessie Wild
Grantmaking Learning  Evaluation /
/Wikimedia Foundation/
/
/
Imagine a world in which every single human being can freely share in
the sum of all knowledge.  Help us make it a reality!
Donate to Wikimedia https://donate.wikimedia.org/


___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l




___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Fwd: [HELP NEEDED] Javascript (beginner level) developer for a small feature request

[Emmanuel Engelhart]

 Message original 
Sujet: [HELP NEEDED] Javascript (beginner level) developer for a small
feature request
Date : Wed, 09 Oct 2013 12:47:00 +0200
De : Emmanuel Engelhart kel...@kiwix.org
Pour : kiwix-develo...@lists.sourceforge.net
kiwix-develo...@lists.sourceforge.net

Hi,

I would appreciate any help for a small javscript developement:
https://sourceforge.net/p/kiwix/feature-requests/767/

This is a feature request gathering many small improvements which should
be done on http://www.kiwix.org/m/, a download page which is proposed to
Android users to download contents.

Here is the list of improvements:
-- Replace Kiwix ZIM Contents and Content Downloads Filter with only
one title like Download content for Kiwix
-- Remove Content Type select box (we don't have enough content for
now to justify such type of filtering)
-- Language selectbox should be automatically filled with the list of
languages available at http://www.kiwix.org/wiki/Template:ZIMdumps
(first column)
-- Language select box entry Auto-detect should be removed and
autodetect should be applied at loading base on browser agent language
-- Replace Download Link label by Download method
-- Replace Download link select box by two radio buttons-- Remove
Filter button, each change on Download method or Language should
automatically apply the filter and reload the results
-- Block the page during each loading of the result (modal dialog box)
or darken the whole page
-- Make the filtering working for all languages, Hindi for example
does not work.

The source code is available at:
https://sourceforge.net/p/kiwix/web/ci/master/tree/mobile-folder/

Any volunteer?

Kind regards
Emmanuel
-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication



___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] Fwd: [HELP NEEDED] Javascript (beginner level) developer for a small feature request

[Emmanuel Engelhart]

Great!
I have assigned the ticket to you.
Thank you
Emmanuel

Le 09/10/2013 13:59, Wilfredor a écrit :
 Hi Emmannuel,
 
 I want to do this request.
 
 Thanks
 
 
 2013/10/9 Emmanuel Engelhart kel...@kiwix.org mailto:kel...@kiwix.org
 
  Message original 
 Sujet: [HELP NEEDED] Javascript (beginner level) developer for a small
 feature request
 Date : Wed, 09 Oct 2013 12:47:00 +0200
 De : Emmanuel Engelhart kel...@kiwix.org mailto:kel...@kiwix.org
 Pour : kiwix-develo...@lists.sourceforge.net
 mailto:kiwix-develo...@lists.sourceforge.net
 kiwix-develo...@lists.sourceforge.net
 mailto:kiwix-develo...@lists.sourceforge.net
 
 Hi,
 
 I would appreciate any help for a small javscript developement:
 https://sourceforge.net/p/kiwix/feature-requests/767/
 
 This is a feature request gathering many small improvements which should
 be done on http://www.kiwix.org/m/, a download page which is proposed to
 Android users to download contents.
 
 Here is the list of improvements:
 -- Replace Kiwix ZIM Contents and Content Downloads Filter with only
 one title like Download content for Kiwix
 -- Remove Content Type select box (we don't have enough content for
 now to justify such type of filtering)
 -- Language selectbox should be automatically filled with the list of
 languages available at http://www.kiwix.org/wiki/Template:ZIMdumps
 (first column)
 -- Language select box entry Auto-detect should be removed and
 autodetect should be applied at loading base on browser agent language
 -- Replace Download Link label by Download method
 -- Replace Download link select box by two radio buttons-- Remove
 Filter button, each change on Download method or Language should
 automatically apply the filter and reload the results
 -- Block the page during each loading of the result (modal dialog box)
 or darken the whole page
 -- Make the filtering working for all languages, Hindi for example
 does not work.
 
 The source code is available at:
 https://sourceforge.net/p/kiwix/web/ci/master/tree/mobile-folder/
 
 Any volunteer?
 
 Kind regards
 Emmanuel
 --
 Kiwix - Wikipedia Offline  more
 * Web: http://www.kiwix.org
 * Twitter: https://twitter.com/KiwixOffline
 * more: http://www.kiwix.org/wiki/Communication
 
 
 
 ___
 Offline-l mailing list
 Offline-l@lists.wikimedia.org mailto:Offline-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/offline-l
 
 
 
 
 ___
 Offline-l mailing list
 Offline-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/offline-l
 


-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] [RELEASE] Kiwix for Android v1.3

[Emmanuel Engelhart]

Hi,

we have released a new version of our ZIM reader for Android.

This release provides a few new things:
* New Night mode (reverse colors)
* New Exit menu entry
* Zoom controls now optional
* Fixed duplicate suggestions bug

Try it:
https://play.google.com/store/apps/details?id=org.kiwix.kiwixmobile

... and report any problem:
http://reportabug.kiwix.org

Emmanuel
-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] First Google Summer of Code for Kiwix and openZIM

[Emmanuel Engelhart]

Hi

We have had our first Google Summer of Code proposal finally accepted:
*
http://www.google-melange.com/gsoc/project/google/gsoc2013/kiranmathewkoshy/24001
* https://www.mediawiki.org/wiki/User:Kiran_mathew_1993
/ZIM_incremental_updates_for_Kiwix
* https://www.mediawiki.org/wiki/User:Kiran_mathew_1993

This project should really help people to keep their offline version of
Wikipedia up-to-date without having to re-download everything each time.

The student and the two mentors are really happy about this decision of
Google and Wikimedia. We take it as a big opportunity and will do our
best to succeed.

Kind regards
Emmanuel
-- 
Kiwix - Wikipedia Offline  more
* Web: http://www.kiwix.org
* Twitter: https://twitter.com/KiwixOffline
* more: http://www.kiwix.org/wiki/Communication

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

Re: [Offline-l] [Wikitech-l] GSoC Project

[Emmanuel Engelhart]

Dear Kiran

Before commenting your proposal, let me thank:
* Quim for having renamed this thread... I wouldn't have got a chance to
read it otherwise.
* Gnosygnu and Sumana for their previous answers.

Your emails points three problems:
(1) The size of the offline dumps
(2) Server mode of the offline solution
(3) The need of incremental updates

Regarding (1), I disagree. We have the ZIM format which is open, has an
extremly efficient standard implementation, provides high compression
rates and fast random access: http://www.openzim.org

Regarding (2), Kiwix, which is a ZIM reader, already does it: you can
either share Kiwix on a network disk or use Kiwix HTTP compatible daemon
called kiwix-serve: http://www.kiwix.org/wiki/Kiwix-serve

Regarding (3), I agree. This is an old feature request in the openZIM
project. It's both on the roadmap and in the bug tracker:
* http://www.openzim.org/wiki/Roadmap
* https://bugzilla.wikimedia.org/show_bug.cgi?id=47406

But, I also think the solution you propose isn't adapted to the problem.
Setting up a Mediawiki is not easy, it's resource intensive and you
don't need all this power (of the software setup) for the usage you want
to do.

On the other side, with ZIM you have a format which provides all what
you need, runs on devices which costs only a few dozens of USD and we
will make this incremental update trivial for the final user (it's just
a matter of time ;).

So to fix that problem, there is my approach: we should implement two
tools I call zimdiff and zimpatch:
* zimdiff is a tool able to compute the difference between two ZIM files
* zimpatch is a tool able to patch a ZIM file with a ZIM diff file

The incrementation process would be:
* Compute a ZIM diff file (done by the ZIM provider)
* Download and path the old ZIM file with the ZIM diff file (done by
the user)

We could implement two modes for zimpatch, leasy and normal:
* leasy mode: simple merge of the file and rewriting of the index (fast
but need a lot of mass storage)
* normal mode: recompute a new file (slow but need less mass storage)

Regarding the ZIM diff file format... the discussion is open, but it
looks like we could simply reuse the ZIM format and zimpatch would work
like a zimmerge (does not exist, it's just for the explanation).

Everything could be done IMO in only a few hundreds of smart lines of
C++. I would be really surprised if this need more than 2000 lines. But,
to do that, we need a pretty talentuous C++ developer, maybe you?

If your or someone else is interested we would probably be able to find
a tutor.

Kind regards
Emmanuel

PS: Wikimedia has an offline centric mailing list, let me add it in CC:
https://lists.wikimedia.org/mailman/listinfo/offline-l

Le 26/04/2013 22:27, Kiran Mathew Koshy a écrit :
 Hi guys,
 
 I have an own idea  for my GSoC project that I'd like to share with you.
 Its not a perfect one, so please forgive any mistakes.
 
 The project is related to the existing GSoC project *Incremental Data dumps
 * , but is in no way a replacement for it.
 
 
 *Offline Wikipedia*
 
 For a long time, a lot of offline solutions for Wikipedia have sprung up on
 the internet. All of these have been unofficial solutions, and  have
 limitations. A major problem is the* increasing size of  the data dumps*,
 and the problem of *updating the local content. *
 
 Consider the situation in a place where internet is costly/
 unavailable.(For the purpose of discussion, lets consider a school in a 3rd
 world country.) Internet speeds are extremely slow, and accessing Wikipedia
 directly from the web is out of the question.
 Such a school would greatly benefit from an instance of Wikipedia on  a
 local server. Now up to here, the school can use any of the freely
 available offline Wikipedia solutions to make a local instance. The problem
 arises when the database in the local instance becomes obsolete. The client
 is then required to download an entire new dump(approx. 10 GB in size) and
 load it into the database.
 Another problem that arises is that most 3rd part programs *do not allow
 network access*, and a new instance of the database is required(approx. 40
 GB) on each installation.For instance, in a school with around 50 desktops,
 each desktop would require a 40 GB  database. Plus, *updating* them becomes
 even more difficult.
 
 So here's my *idea*:
 Modify the existing MediaWiki software and to add a few PHP/Python scripts
 which will automatically update the database and will run in the
 background.(Details on how the update is done is described later).
 Initially, the MediaWiki(modified) will take an XML dump/ SQL dump (SQL
 dump preferred) as input and will create the local instance of Wikipedia.
 Later on, the updates will be added to the database automatically by the
 script.
 
 The installation process is extremely easy, it just requires a server
 package like XAMPP and the MediaWiki bundle.
 
 
 Process of updating:
 
 There will be two methods of updating the server. Both 

[Offline-l] [OPENZIM] zimlib 1.0 code tarball released

[Emmanuel Engelhart]

Hi

The openZIM project finally released a first version of its standard
implementation code, the zimlib:
http://www.openzim.org/download/zimlib-1.0.tar.gz

This tarball contains the C++ dev files:
* to read a ZIM file,
* to write a ZIM file,
* a few tools
* a few example programs.

Releasing this was mandatory to:
* help GNU/Linux packagers.
* have a stable code base for new bindings, like phpzim.

More information about the openZIM project at:
http://www.openzim.org

Regards
Emmanuel

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l

[Offline-l] Kiwix for Android released!

[Emmanuel Engelhart]

Hi,

Wikimedia CH has sponsored a long Kiwix Hackathon (and the WMF two
tablets) and there is the result, the first version of Kiwix (and also
the first ZIM reader) for Android:
https://play.google.com/store/apps/details?id=org.kiwix.kiwixmobile

It’s very easy to use:
* Download the app from the Google Play Store, and launch it;
* Click on the “Open” button, select a ZIM file from the list (from your
device or SD card);
* That’s it! You’re already browsing offline content.

If you don’t already have a ZIM file of Wikipedia, Kiwix leads you to
its online repository so you can download one:
http://kiwix.org/m/

You can also use any of the Books created on Wikipedia:
https://en.wikipedia.org/wiki/Special:Book

Being a mobile version of a feature-rich desktop software, the Android
version is lightweight. It provides access to the most essential
features: opening and reading a ZIM file, search with auto-completion on
article titles, in-page search, random navigation, zoom in  out.

More details in the Wikimedia Blog post:
https://blog.wikimedia.org/2013/04/17/carry-the-entirety-of-wikipedia-in-your-pocket-with-kiwix-for-android/

Emmanuel

___
Offline-l mailing list
Offline-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/offline-l