Re: [Wikitech-l] Idea for new desktop / mobile kiwix like application
On 26.01.2015 13:09, Petr Bena wrote: 1) There may be no ZIM's for the wiki they want to use and they have no idea how to create one. They won't be able to use kiwix here. We propose a ZIM file for most of the important projects. The problem is that we still don't have the resources to generate all of them one time a month. But, this problem is going to be fixed, we are currently building a farm in wmflabs to provide one time a month a fresh ZIM file of all projects. 2) There may be some ZIM's but these don't contain pages that user is interested in. 3) There may be some ZIM's but these contain too many pages that user doesn't need and thus are too large. I am trying to tell you that ZIM is a very limited solution, I don't want to say that kiwix is unusable for everyone, it's a very useful solution for many use cases (for example some school computers / libraries with no internet access could have kiwix with full encyclopedia available to everyone), but for USER's (eg. individuals with limited internet access or no computing knowledge) is pretty much unusable. I myself am having troubles getting kiwix even to try it out. I have SSD on my work PC with less than 2gb of free space and at home, with large HDD I have so slow internet that I would be downloading that ZIM file few weeks. And I don't even live in some underdeveloped country, try to imagine how hard this must be for people with really slow internet ( 20kbs etc). What you describe here is related to the selection tool, not to Kiwix, neither to the storage format. Extension:Collection was able to generate ZIM files before OCG project. Help-us to bring this feature back. IMHO best solution for this use-case would be to maintain local DB that would contain only data for pages selected by user, which would be downloaded using mediawiki API's, so that there would be no need for any ZIM packaging whatsoever. Downloading each page separately using the API will be for sure slower than downloading the corresponding data in ZIM format... probably even if you re-download all articles (so without any incremental update process). Having this option in Kiwix would be cool, but I can't see anything else what could be reused than the reader itself (the part of kiwix that turns wikitext into html page and display it on screen). Kiwix has nothing to do with wikitext and never turns it in HTML. Emmanuel -- Kiwix - Wikipedia Offline more * Web: http://www.kiwix.org * Twitter: https://twitter.com/KiwixOffline * more: http://www.kiwix.org/wiki/Communication ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] All Hands On Deck!
Le 23/01/2015 09:11, littlewmfb...@yandex.com a écrit : Oh what a day! Which began when perforce a visitor from afar began to exhort snip https://lists.wikimedia.org/pipermail/wikitech-l/2015-January/080300.html Hello, Can we get a simple English version of the poetry there? Seems it is related to last week Wikimedia all hands meeting, but I have hard time understand the text, the point of it or what should be done :-] cheers, -- Antoine hashar Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Current status of IPv6 connectivity?
Hello, Are there any reports that could allow one to check the status of IPv6 connectivity in the WMF cluster? If not, could you let me know if IPv6 is deployed on all servers and if you consider it stable? IPv6 is mostly deployed on a per server/service basis right now, but we do consider it stable. Unfortunately there is no way right now to check the status of IPv6 connectivity but there is a Tech Ops goal of better monitoring which includes plans to make IPv6 connectivity checking a first class citizen (same as IPv4) I'm seeing huge variations in IPv6 performance when connecting to WMF servers, while the rest of the internet seems to work and I'm trying to determine if the problem is on my side, somewhere on the way, or in the WMF network. Well tools like mtr or even IPv6 enabled traceroute could be of immense help there. I must say, I 've not found a performance problem in the IPv6 connectivity yet in the WMF cluster. Feel free to provide us with more information, though if you have persistent problems ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Thank you for listening (@ the Developer Summit)
Today I had the *fun* job of presenting and answering questions for 2 hours and 15 minutes straight. I didn't actually realize this would turn out to be the marathon that it did, but I feel like it turned out better than expected. I was especially impressed with the positive attitude that everyone had as they challenged my viewpoints and patiently heard me out. Thank you to those willing to spar with me from across the room over microphones. I really value people who question authority in search for the right answer. I felt proud today to work at a place where rigorous public debate is still alive and well. Also thank you to Jarred, Roan, Ed and Timo for taking some of the questions, allowing me shut up for a bit. - Trevor ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Github access?
Turns out I actually do have rights to do this, just took me a while to figure out how to find it in github interface. :) Ok you should have rights on apps-ios-wikipedia and should be able to manip the pull reqs. Don't merge them directly though -- merge them manually via gerrit and close them out manually if github doesn't pick them up after (which it might not since the mirroring is broken atm). -- brion On Mon, Jan 26, 2015 at 3:46 PM, Corey Floyd cfl...@wikimedia.org wrote: Does anyone know who has the administrative privlidges to add me to the github wikimedia organization? I need to cleanup/maintain our external pull requests. Thanks! -- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Github access?
Can you add me to the Wikimedia org as well? On Monday, January 26, 2015, Brion Vibber bvib...@wikimedia.org wrote: Turns out I actually do have rights to do this, just took me a while to figure out how to find it in github interface. :) Ok you should have rights on apps-ios-wikipedia and should be able to manip the pull reqs. Don't merge them directly though -- merge them manually via gerrit and close them out manually if github doesn't pick them up after (which it might not since the mirroring is broken atm). -- brion On Mon, Jan 26, 2015 at 3:46 PM, Corey Floyd cfl...@wikimedia.org javascript:; wrote: Does anyone know who has the administrative privlidges to add me to the github wikimedia organization? I need to cleanup/maintain our external pull requests. Thanks! -- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Github access?
Nevermind I found it. Sorry for the list spam everyone. :) -- brion On Mon, Jan 26, 2015 at 3:57 PM, Brion Vibber bvib...@wikimedia.org wrote: On Mon, Jan 26, 2015 at 3:54 PM, Brian Gerstle bgers...@wikimedia.org wrote: Can you add me to the Wikimedia org as well? Remind me your github username? (btw I'm adding folks to a 'mobile' team group within the org) -- brion On Monday, January 26, 2015, Brion Vibber bvib...@wikimedia.org wrote: Turns out I actually do have rights to do this, just took me a while to figure out how to find it in github interface. :) Ok you should have rights on apps-ios-wikipedia and should be able to manip the pull reqs. Don't merge them directly though -- merge them manually via gerrit and close them out manually if github doesn't pick them up after (which it might not since the mirroring is broken atm). -- brion On Mon, Jan 26, 2015 at 3:46 PM, Corey Floyd cfl...@wikimedia.org javascript:; wrote: Does anyone know who has the administrative privlidges to add me to the github wikimedia organization? I need to cleanup/maintain our external pull requests. Thanks! -- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] All Hands On Deck!
- Original Message - From: Antoine Musso hashar+...@free.fr Le 23/01/2015 09:11, littlewmfb...@yandex.com a écrit : Oh what a day! Which began when perforce a visitor from afar began to exhort snip https://lists.wikimedia.org/pipermail/wikitech-l/2015-January/080300.html Hello, Can we get a simple English version of the poetry there? Seems it is related to last week Wikimedia all hands meeting, but I have hard time understand the text, the point of it or what should be done :-] Poetry doesn't lend itself to translation well, but I read it as a scoff, myself. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Idea for new desktop / mobile kiwix like application
Dear Petr On 23.01.2015 11:59, Petr Bena wrote: Some of you probably know kiwix - kiwix.org which is offline wikipedia reader. I think the idea of this reader is cool, most of you probably sometimes wanted to access wikipedia while being offline somewhere, but couldn't. Kiwix can help with this, however it has one big problem and solution for it is so complex that it would basically need a rewrite of whole thing. That problem is that you need to download pretty huge file (40+GB) in order to use it for en wikipedia for example. We provide smaller ZIM files providing selection of articles (have a look on http://download.kiwix.org/zim/). AFAIK, for the same amount of information, we provide the best compression ratio (and consequently potential update speed). And if you wanted to update those few wikipages you are interested in, to a latest revision, then you again need to download that huge file. That's a true, openZIM was thought, implemented and optimized for read-only. But the sentence solution for it is so complex that it would basically need a rewrite of whole thing is wrong. Without repeating what Federico already said (later in this thread but earlier last week), we have almost a solution (implemented by a GSOC student) to make incremental upgrades. This solution works well but is a little bit CPU intensive (due to the need to recompress clusters). A lazy version of the patch functionality could fix that quickly (and for small amounts of articles this would not make a big difference). An intelligent version of diff/patch might avoid both the cluster re-compression step and the continuous ZIM file growing might also be implemented. All of this might be implement in a few weeks by a skilled C++ developer without the rewrite of whole thing, just adding new functionnalities. I have been working on this kind of stuff since almost ten years and have seen a lot of different offline softwares proposed and part of them published. None of the ones which brought a real added value to the end-user was developed quicker than in a few weeks. Regards Emmanuel -- Kiwix - Wikipedia Offline more * Web: http://www.kiwix.org * Twitter: https://twitter.com/KiwixOffline * more: http://www.kiwix.org/wiki/Communication ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Idea for new desktop / mobile kiwix like application
That is still IMHO highly unusable, let me explain why: Let's define USER as a computer illiterate that has absolutely no knowledge of how computers work. They got their PC (or mobile) and want to run offline reader of either wikipedia or their favorite wiki, which can be non-wmf thing. Problems: 1) There may be no ZIM's for the wiki they want to use and they have no idea how to create one. They won't be able to use kiwix here. 2) There may be some ZIM's but these don't contain pages that user is interested in. 3) There may be some ZIM's but these contain too many pages that user doesn't need and thus are too large. I am trying to tell you that ZIM is a very limited solution, I don't want to say that kiwix is unusable for everyone, it's a very useful solution for many use cases (for example some school computers / libraries with no internet access could have kiwix with full encyclopedia available to everyone), but for USER's (eg. individuals with limited internet access or no computing knowledge) is pretty much unusable. I myself am having troubles getting kiwix even to try it out. I have SSD on my work PC with less than 2gb of free space and at home, with large HDD I have so slow internet that I would be downloading that ZIM file few weeks. And I don't even live in some underdeveloped country, try to imagine how hard this must be for people with really slow internet ( 20kbs etc). IMHO best solution for this use-case would be to maintain local DB that would contain only data for pages selected by user, which would be downloaded using mediawiki API's, so that there would be no need for any ZIM packaging whatsoever. The local DB might take more space than ZIM, it's possible that compression wouldn't be so effective. But given that user would only have those pages they are interested in, it would be likely much smaller than whole collection. They would make their own collection, easily with few clicks. For any wiki. Having this option in Kiwix would be cool, but I can't see anything else what could be reused than the reader itself (the part of kiwix that turns wikitext into html page and display it on screen). On Mon, Jan 26, 2015 at 12:30 PM, Emmanuel Engelhart kel...@kiwix.org wrote: Dear Petr On 23.01.2015 11:59, Petr Bena wrote: Some of you probably know kiwix - kiwix.org which is offline wikipedia reader. I think the idea of this reader is cool, most of you probably sometimes wanted to access wikipedia while being offline somewhere, but couldn't. Kiwix can help with this, however it has one big problem and solution for it is so complex that it would basically need a rewrite of whole thing. That problem is that you need to download pretty huge file (40+GB) in order to use it for en wikipedia for example. We provide smaller ZIM files providing selection of articles (have a look on http://download.kiwix.org/zim/). AFAIK, for the same amount of information, we provide the best compression ratio (and consequently potential update speed). And if you wanted to update those few wikipages you are interested in, to a latest revision, then you again need to download that huge file. That's a true, openZIM was thought, implemented and optimized for read-only. But the sentence solution for it is so complex that it would basically need a rewrite of whole thing is wrong. Without repeating what Federico already said (later in this thread but earlier last week), we have almost a solution (implemented by a GSOC student) to make incremental upgrades. This solution works well but is a little bit CPU intensive (due to the need to recompress clusters). A lazy version of the patch functionality could fix that quickly (and for small amounts of articles this would not make a big difference). An intelligent version of diff/patch might avoid both the cluster re-compression step and the continuous ZIM file growing might also be implemented. All of this might be implement in a few weeks by a skilled C++ developer without the rewrite of whole thing, just adding new functionnalities. I have been working on this kind of stuff since almost ten years and have seen a lot of different offline softwares proposed and part of them published. None of the ones which brought a real added value to the end-user was developed quicker than in a few weeks. Regards Emmanuel -- Kiwix - Wikipedia Offline more * Web: http://www.kiwix.org * Twitter: https://twitter.com/KiwixOffline * more: http://www.kiwix.org/wiki/Communication ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Github access?
Does anyone know who has the administrative privlidges to add me to the github wikimedia organization? I need to cleanup/maintain our external pull requests. Thanks! -- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] MediaWiki Developer Summit: Future of the Architecture Committee
Hi all, A late notice for people at MWDS, and my apologies for everyone else. We (ArchCom + me) have belatedly added a session called Future of the Architecture Committee after the How MediaWiki Slows Us Down plenary, which will be at 1:45pm PST on Tuesday (tomorrow) Task associated with this: https://phabricator.wikimedia.org/T87605 Here's the list of questions we currently plan to tackle: 1. Should the ArchCom continue to exist as a committee with regular meetings? 2. Who takes care of RFCs? 3. Who drives paying down technical debt? 4. How does ArchCom/whatever integrate with quarterly planning at WMF and in the MediaWiki dev community in general? I've included my earlier email below titled No more Architecture Committee? as refresher prereading for this (and ignore my rampant transposition of the middle letters of BDFL, though in my defense, the BDFL issue is a BFD) :-) Rob -- Forwarded message -- From: Rob Lanphier ro...@wikimedia.org Date: Thu, Jan 15, 2015 at 8:04 PM Subject: Fwd: No more Architecture Committee? To: Wikimedia developers wikitech-l@lists.wikimedia.org (Alright...let's try this again!) Hi everyone, The current MediaWiki Architecture Committee[1] has its roots in a 2013 Amsterdam Hackathon session[2], where we had a pair of sessions to try to establish our architectural guidelines[3]. It was there that we agreed that it would be good to revive our then moribund process for reviewing RFCs[4]. Since no one there really knew whose job it was to review these things, I believe I said how about we start with everyone with 'architect' in their title at WMF?, which was met with uncomfortable shrugging that I interpreted as consensus!, and no one corrected me. Thus Brion Vibber, Mark Bergsma, and Tim Starling became the founding members of the Arch Committee. Subsequent to that meeting, I pretended to proceed as though a decision was made. However, over the past year and half since then, there's been much more uncomfortable shrugging. Even Brion, Mark, and Tim have not seemed entirely comfortable with the idea. It was widely acknowledged that the group was heavily biased toward the lower parts of our server software stack. The committee agreed to add Roan Kattouw and Daniel Kinzler to the group as a means of providing a wider perspective, with the added bonus of adding at least one person who isn't a WMF employee. So, here we are today. I believe no one would dispute the credentials of every member of the group. Brion, Tim, and Mark have an extremely long history with the project, being employees #1, #2, and #3 of the WMF respectively, and all having contributed massively to the success of Wikipedia and to MediaWiki as general purpose wiki software. In most open source projects, one of them would probably be BFDL[5]. Roan and Daniel are more recent, but only in relative terms, and also have very significant contributions to their name. All have the widespread respect of pretty much everyone in the MediaWiki developer community. Additionally, I hear quite a bit of relief that the previously moribund RFC process is doing much better now. Things are moving, and if you know how to work the process and aren't proposing anything too wild, you can get an RFC approved pretty quickly. The committee has made a lap through the entire backlog of RFCs. Still, the uncomfortable shrugging continues. The group is broader, but still lacks the breadth, particularly in front end and in the development of newer services such as Parsoid and RESTBase. This aspect is pretty obviously something that can be fixed. Another problem is that the scope of the group isn't clear to everyone. Is this group responsible for leading, or merely responsible for reviewing big ideas from others to ensure continuity and sanity? How big does an idea need to be before an RFC needs to be written (as opposed to just dropping a patch in Gerrit)? Defining the scope of the group is also a fixable problem. However, I don't sense much of a desire to fix things. The dominant meme that I hear is that we should go back to the day before uncomfortable shrugging led to a committee becoming BFDL. What I fear, though, is that we will develop a system lacking in conceptual integrity[6], as individual warring fiefdoms emerge. It's quite simple to argue this is already happening. So, where does that leave us? Do we need a BFDL? If so, who should pick? Should it be someone in the project? Should the WMF hire someone to lead this? If not, do we keep the committee? Do we just let this be consensus based? On the leadership front, let me throw out a hypothetical: should we have MediaWiki 2.0, where we start with an empty repository and build up? If so, who makes that decision? If not, what is our alternative vision? Who is going to define it? Is what we have good enough? In general, I feel a sense of urgency that seems lacking in the status quo. We've made progress over
Re: [Wikitech-l] Github access?
On Mon, Jan 26, 2015 at 3:54 PM, Brian Gerstle bgers...@wikimedia.org wrote: Can you add me to the Wikimedia org as well? Remind me your github username? (btw I'm adding folks to a 'mobile' team group within the org) -- brion On Monday, January 26, 2015, Brion Vibber bvib...@wikimedia.org wrote: Turns out I actually do have rights to do this, just took me a while to figure out how to find it in github interface. :) Ok you should have rights on apps-ios-wikipedia and should be able to manip the pull reqs. Don't merge them directly though -- merge them manually via gerrit and close them out manually if github doesn't pick them up after (which it might not since the mirroring is broken atm). -- brion On Mon, Jan 26, 2015 at 3:46 PM, Corey Floyd cfl...@wikimedia.org javascript:; wrote: Does anyone know who has the administrative privlidges to add me to the github wikimedia organization? I need to cleanup/maintain our external pull requests. Thanks! -- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
