Re: [Wikitech-l] FLIF for Wikimedia

2017-12-04 Thread Vi to 
Changing a crucial element of the fifth most popular website on the
Internet may be a good example for an article about [[PITA]] :D

While I'm open to new formats/technologies I think a fundamental
prerequisite is a widespread support among browsers/clients. JS-only
support will severely weaken user experience for high-latency users,
destroying it for non-JS users.

Also, we currently have not enough resources to pioneer and, furthermore,
we are bound -per mission- to serve our content in the most accessible way.

Vito

2017-12-04 23:09 GMT+01:00 Brian Wolff :

> An encode latency of 7 seconds and decode latency of 1 second arent what I
> would call "very fast" (the decode latency measurement probably isnt
> realistic as decode and encode to png is different from just display in
> browser. Then again all these measurements are going to vary by filesize,
> not to mention low power devices like phones)
>
> If it indeed takes 1 second to decode, thats probably more time than the
> space savings would save time wise on a moderate speed internet connection.
>
> And time is only one metric. Often image encoding is limitted by memory.
>
>
> Anyways i'd be opposed to this unless the bandwidth saving was extreme.
> Wikipedia is not the place to be a testing ground for experimental image
> formats that browsers arent even supporting.
>
>
> --
> bawolff
>
> p.s.lossless rotatio /cropping of jpegs is actually very common due to
> rotatebot
>
> On Monday, December 4, 2017, Ruben Kelevra  wrote:
> > Hey Thiemo,
> >
> > On 04.12.2017 10:43, Thiemo Kreuz wrote:> I consider myself an image
> > file format nerd, so thanks a lot for
> >> sharing this! FLIF was new to me.Don't mind it! :)
> >
> >> I would like to share two important notes:
> >>
> >> 1. Unfortunately the flif.info website does not say a word about the
> >> CPU resources their current implementation burns when converting a,
> >> let's say, PNG to FLIF.Well, currently it's single core and not
> > optimized at all. You should know CABAC encoding from x264/x265 so it's
> > slow, but not dead slow.
> >
> > The website doesn't mention it because it highly depends on the subject
> > as well as the setting on encoding named effort.
> >
> > Currently, effort is default 60, I tried 100 a lot, but there's nearly
> > nothing to gain. So I assume we always want to use the good default. :)
> >
> > PNG Encoding of the current featured picture of today[1] at a medium
> > image size for the web, 1.280×868 Pixel, opened in Gimp, converted to
> > sRGB and exported as maximum compression without any checkbox done in
> > Gimp ... takes Gimp 3 seconds to write it to the Harddisk.
> >
> > Transcoding this PNG to FLIF takes on my machine (i3-2330M@2.20GHz; 12
> > GB RAM):
> > real 0m7,405s
> > user 0m7,320s
> > sys 0m0,053s
> >
> > decoding the file again to PNG on the same machine (with FLIF)
> > real 0m1,077s
> > user 0m1,067s
> > sys 0m0,007s
> >
> > For comparison, we save 708 KByte in comparison to PNG in this case, and
> > the PNG exported by FLIF is just 14 KByte bigger than the one created by
> > Gimp.
> >
> >> It's pretty important to realize that CPU
> >> resources are even more valuable than storage space and network
> >> bandwidth. Sure, It's not really possible to come up with an exact
> >> threshold. But if, let's say, converting a PNG to FLIF saves 100 KiB,
> >> but takes a minute, this minute will never pay off.
> > So my Point was more: Get rid of this bloody Download a JPG, do some
> > Stuff & Upload a 3 Times locally saved JPG again, calling it an
> improvement.
> >
> >> If you follow the discussions on Wikimedia Commons, you will find this
> >> argument quite often: Downloading PNGs, optimizing them, and uploading
> >> them again is practically never worth it. Think of all the resources
> >> that are burned to do this: CPU time, download and upload, database
> >> storage and time, disk storage for the new revision, and not to forget
> >> the user doing all this.
> > Yep, but enabling FLIF for new files would do nothing to the old Data at
> > all, I don't want to start a discussion about converting petabytes of
> > Data, but all new revisions should be done in FLIF, if you ask me.
> >> Sure, your suggestion avoids a lot of this. But the CPUs involved will
> >> experience heavy load, both on the server as well as clients that need
> >> to recode FLIF files via a JavaScript library first.
> > FLIF is very fast to decode via JavaScript, and in general, as the
> > example shown above, it takes just 1 second to decode and encode a
> > medium size image as PNG with just one thread on a pretty outdated
> > notebook with an unoptimized decoder and encoder. :)
> >
> > Try adding a FLIF to a website and test out if the website load anywhat
> > slower with the FLIF ... at the small image sizes you get on articles,
> > the performance impact is neglectable and comparable to loading a font
> > file to the browser.
> >> 2. Lossy file formats

Re: [Wikitech-l] FLIF for Wikimedia

2017-12-04 Thread Brian Wolff 
An encode latency of 7 seconds and decode latency of 1 second arent what I
would call "very fast" (the decode latency measurement probably isnt
realistic as decode and encode to png is different from just display in
browser. Then again all these measurements are going to vary by filesize,
not to mention low power devices like phones)

If it indeed takes 1 second to decode, thats probably more time than the
space savings would save time wise on a moderate speed internet connection.

And time is only one metric. Often image encoding is limitted by memory.


Anyways i'd be opposed to this unless the bandwidth saving was extreme.
Wikipedia is not the place to be a testing ground for experimental image
formats that browsers arent even supporting.


--
bawolff

p.s.lossless rotatio /cropping of jpegs is actually very common due to
rotatebot

On Monday, December 4, 2017, Ruben Kelevra  wrote:
> Hey Thiemo,
>
> On 04.12.2017 10:43, Thiemo Kreuz wrote:> I consider myself an image
> file format nerd, so thanks a lot for
>> sharing this! FLIF was new to me.Don't mind it! :)
>
>> I would like to share two important notes:
>>
>> 1. Unfortunately the flif.info website does not say a word about the
>> CPU resources their current implementation burns when converting a,
>> let's say, PNG to FLIF.Well, currently it's single core and not
> optimized at all. You should know CABAC encoding from x264/x265 so it's
> slow, but not dead slow.
>
> The website doesn't mention it because it highly depends on the subject
> as well as the setting on encoding named effort.
>
> Currently, effort is default 60, I tried 100 a lot, but there's nearly
> nothing to gain. So I assume we always want to use the good default. :)
>
> PNG Encoding of the current featured picture of today[1] at a medium
> image size for the web, 1.280×868 Pixel, opened in Gimp, converted to
> sRGB and exported as maximum compression without any checkbox done in
> Gimp ... takes Gimp 3 seconds to write it to the Harddisk.
>
> Transcoding this PNG to FLIF takes on my machine (i3-2330M@2.20GHz; 12
> GB RAM):
> real 0m7,405s
> user 0m7,320s
> sys 0m0,053s
>
> decoding the file again to PNG on the same machine (with FLIF)
> real 0m1,077s
> user 0m1,067s
> sys 0m0,007s
>
> For comparison, we save 708 KByte in comparison to PNG in this case, and
> the PNG exported by FLIF is just 14 KByte bigger than the one created by
> Gimp.
>
>> It's pretty important to realize that CPU
>> resources are even more valuable than storage space and network
>> bandwidth. Sure, It's not really possible to come up with an exact
>> threshold. But if, let's say, converting a PNG to FLIF saves 100 KiB,
>> but takes a minute, this minute will never pay off.
> So my Point was more: Get rid of this bloody Download a JPG, do some
> Stuff & Upload a 3 Times locally saved JPG again, calling it an
improvement.
>
>> If you follow the discussions on Wikimedia Commons, you will find this
>> argument quite often: Downloading PNGs, optimizing them, and uploading
>> them again is practically never worth it. Think of all the resources
>> that are burned to do this: CPU time, download and upload, database
>> storage and time, disk storage for the new revision, and not to forget
>> the user doing all this.
> Yep, but enabling FLIF for new files would do nothing to the old Data at
> all, I don't want to start a discussion about converting petabytes of
> Data, but all new revisions should be done in FLIF, if you ask me.
>> Sure, your suggestion avoids a lot of this. But the CPUs involved will
>> experience heavy load, both on the server as well as clients that need
>> to recode FLIF files via a JavaScript library first.
> FLIF is very fast to decode via JavaScript, and in general, as the
> example shown above, it takes just 1 second to decode and encode a
> medium size image as PNG with just one thread on a pretty outdated
> notebook with an unoptimized decoder and encoder. :)
>
> Try adding a FLIF to a website and test out if the website load anywhat
> slower with the FLIF ... at the small image sizes you get on articles,
> the performance impact is neglectable and comparable to loading a font
> file to the browser.
>> 2. Lossy file formats like JPEG should never be converted to lossless
>> formats. This will actually decrease quality (over time). The problem
>> is that the image data will still contain the exact same JPEG
>> artifacts, but the fact that it was a JPEG (and how it was encoded) is
>> lost.
> Yes, but those images should never be saved as JPG in the first place.
> Even under Android RAW-Photography is going to be a thing. FLIF just
> started to get rudimentary RAW-capabilities, which means you can just
> convert the special RAW-File to a FLIF and upload it with any loss in
> quality.
>
>> No tool specialized in squeezing the maximum quality out of
>> lossy JPEGs can work anymore. And there are a lot of super-awesome
>> tools like this. Not only tools like JPEGCROP and such that

Re: [Wikitech-l] RfC: PlatformSettings.php

2017-12-04 Thread Cyken Zeraux 
Personally, I've wanted to do away with having a settings file in PHP
entirely, and rather do something closer to how Drupal 8 handles
configurations. With the current state of configuration, its not reliably
possible to know how a wiki is configured automatically. Making any sort of
maintenance/deployment tools for a mediawiki install quite powerless if a
user edits LocalSettings.php themselves.

On Mon, Dec 4, 2017 at 1:50 PM, Kunal Mehta  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Hi,
>
> I've written an RfC to introduce a standardized and recommended way
> for re-distributors and packagers of MediaWiki to be able to tune
> DefaultSettings.php as appropriate.
>
> You can read more details on-wiki[1], and discuss it on Phabricator[2].
>
> [1]
> https://www.mediawiki.org/wiki/Requests_for_comment/PlatformSettings.php
> [2] https://phabricator.wikimedia.org/T182020
>
> Thanks!
> - -- Legoktm
> -BEGIN PGP SIGNATURE-
>
> iQJLBAEBCgA1FiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAlolpwQXHGxlZ29rdG1A
> bWVtYmVyLmZzZi5vcmcACgkQUvyOe+23/KL43RAAld/PQ+KcmBks4lkaChAVK21o
> 7yTCBogPHQpLF73a4pNDIRn+XkHoO0jRmVkoGRIEsE4wTpvyFxxJo1cTbD2QAIKl
> K0WKGBPQ8Lyyu4nFodiPwrvSGhhSi0iVvxMQPLq8K/FPF6bO8c331VAcLoJDQjyY
> rfuQMnAnvBZ7kOzQPiXYgMkSpWgzFF1hn32h7HXyNuanK5KxjeRKl3ZTjmJj2iIk
> zGoUhekJxipeIxgTb6ZT0H3YVMQdXyHuq4fvQFpOsX+KWgKspWOe4u7wbKTfCIN6
> OCtRbIypHlSG4AgeQ1Bqcq3cPyvt2YuBaEaKn1ZMgByWKy65jcWt+x0KsVSe3aYm
> jxzQI7gXBOIVvQeYHTezz91RSkL6bikI0wdBqg7Lsdr4Mb62CDQcfMmFSI1Iz8dD
> ooz7Kcc7VTfU75VokJKNCPnVqdhux7dBgw5EhxBFGFIYfwUu6j8MayIxaZT8uyKj
> OLCYOn8hslFEFeiOnSoMGtsS/6m6RR2YiTLC9r/+lGN2T/UeHm9RuCaBviZEaiQl
> S9X9CAZlkaRg74zNyPO2e/E2v68F1shbpZDutzylal9QJfA0cEhBKbfQTEPlQwgN
> qNxS7ajbb2G96HvF2Tyj8ai4YGENltmhp9M3/NzuJW6jj0fee+KNiNM7sRfMm7p4
> nW7bXwXYkK4m4SuWxDM=
> =PL+x
> -END PGP SIGNATURE-
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] FLIF for Wikimedia

2017-12-04 Thread Chad 
On Mon, Dec 4, 2017 at 10:34 AM Ruben Kelevra  wrote:

> > Sure, your suggestion avoids a lot of this. But the CPUs involved will
> > experience heavy load, both on the server as well as clients that need
> > to recode FLIF files via a JavaScript library first.
> FLIF is very fast to decode via JavaScript, and in general, as the
> example shown above, it takes just 1 second to decode and encode a
> medium size image as PNG with just one thread on a pretty outdated
> notebook with an unoptimized decoder and encoder. :)
>
> Try adding a FLIF to a website and test out if the website load anywhat
> slower with the FLIF ... at the small image sizes you get on articles,
> the performance impact is neglectable and comparable to loading a font
> file to the browser.
>
>
Requiring Javascript just to look at an image seems rather ridiculous to
me, fast or not. It's just...silly sounding. So my browser doesn't support
FLIF...you have to use JS to turn it into some format (PNG) that I can
understand...for what, minimal size savings on the order of a few KB? That
doesn't seem worth the complexity...

How does it fall back for users with Javascript disabled outright? How fast
is it on older hardware? How about mobile? Remember: not everyone has fast
desktops or laptops :)

It doesn't look like there's very much support in the authoring area
either[0]. So we'd have to encode all uploads to this format. Would we be
storing the original PNGs as well, similar to how we store video transcodes
in multiple formats? If so, there goes any space savings on WMF's end.

The fact that the Debian RFP for the encoder/decoder has stalled for almost
2 years isn't very promising... [1]

Considering there's effectively zero browser support (comments like
[2][3][4] and *especially* [5] are pretty discouraging) it just doesn't
seem worth it the technical maintenance to support it on our end.

Don't get me wrong, the format itself seems kinda cool, but I'd be very
hesitant to be a pioneer for a new graphics standard that nobody else seems
to want to pick up yet.

-Chad

[0] http://flif.info/software.html#graphics-software
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812761
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1240692#c6
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1240692#c8
[4] https://bugs.chromium.org/p/chromium/issues/detail?id=539120#c4
[5] https://bugs.chromium.org/p/chromium/issues/detail?id=539120#c11
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Help test longer edit summaries (and other comments) on testwiki

2017-12-04 Thread Brad Jorsch (Anomie) 
Unfortunately, I had to revert it. We'll try again later when the schema
change is compete.

On Mon, Dec 4, 2017 at 2:38 PM, Brad Jorsch (Anomie) 
wrote:

> I've just now enable the feature flag on the testing wikis[1][2][3] that
> allows MediaWiki to store comments longer than 255 bytes.
>
> The web UI has not been updated to allow longer comments in places where
> it enforces a limit, such as the edit summary box. But if you use the API
> to edit, or perform page moves or do other things where long comments could
> be entered and were truncated, you should now find that they're truncated
> at 1000 Unicode characters rather than 255 bytes.
>
> Please test it out! If you find errors, or places in core features (not
> comments in extensions such as SecurePoll, AbuseFilter, CheckUser, or Flow)
> where *new* comments are still being truncated to 255 bytes, or places
> where comments aren't showing up at all, please let me know. You can reply
> to this message or post a task in Phabricator and add me as a subscriber.
>
> If things go well, we'll look at rolling this out to production wikis once
> the schema changes to the production databases are complete. See
> https://phabricator.wikimedia.org/T174569 to follow progress there.
>
> If anyone is interested in submitting patches for the web UI to reflect
> the changed length limits, please do. I'll try to review them if you add me
> as a reviewer.
>
> [1]: https://test.wikipedia.org/wiki/Main_Page
> [2]: https://test2.wikipedia.org/wiki/Main_Page
> [3]: https://test.wikidata.org/wiki/Wikidata:Main_Page
>
> --
> Brad Jorsch (Anomie)
> Senior Software Engineer
> Wikimedia Foundation
>



-- 
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] RfC: PlatformSettings.php

2017-12-04 Thread Kunal Mehta 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi,

I've written an RfC to introduce a standardized and recommended way
for re-distributors and packagers of MediaWiki to be able to tune
DefaultSettings.php as appropriate.

You can read more details on-wiki[1], and discuss it on Phabricator[2].

[1]
https://www.mediawiki.org/wiki/Requests_for_comment/PlatformSettings.php
[2] https://phabricator.wikimedia.org/T182020

Thanks!
- -- Legoktm
-BEGIN PGP SIGNATURE-

iQJLBAEBCgA1FiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAlolpwQXHGxlZ29rdG1A
bWVtYmVyLmZzZi5vcmcACgkQUvyOe+23/KL43RAAld/PQ+KcmBks4lkaChAVK21o
7yTCBogPHQpLF73a4pNDIRn+XkHoO0jRmVkoGRIEsE4wTpvyFxxJo1cTbD2QAIKl
K0WKGBPQ8Lyyu4nFodiPwrvSGhhSi0iVvxMQPLq8K/FPF6bO8c331VAcLoJDQjyY
rfuQMnAnvBZ7kOzQPiXYgMkSpWgzFF1hn32h7HXyNuanK5KxjeRKl3ZTjmJj2iIk
zGoUhekJxipeIxgTb6ZT0H3YVMQdXyHuq4fvQFpOsX+KWgKspWOe4u7wbKTfCIN6
OCtRbIypHlSG4AgeQ1Bqcq3cPyvt2YuBaEaKn1ZMgByWKy65jcWt+x0KsVSe3aYm
jxzQI7gXBOIVvQeYHTezz91RSkL6bikI0wdBqg7Lsdr4Mb62CDQcfMmFSI1Iz8dD
ooz7Kcc7VTfU75VokJKNCPnVqdhux7dBgw5EhxBFGFIYfwUu6j8MayIxaZT8uyKj
OLCYOn8hslFEFeiOnSoMGtsS/6m6RR2YiTLC9r/+lGN2T/UeHm9RuCaBviZEaiQl
S9X9CAZlkaRg74zNyPO2e/E2v68F1shbpZDutzylal9QJfA0cEhBKbfQTEPlQwgN
qNxS7ajbb2G96HvF2Tyj8ai4YGENltmhp9M3/NzuJW6jj0fee+KNiNM7sRfMm7p4
nW7bXwXYkK4m4SuWxDM=
=PL+x
-END PGP SIGNATURE-

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Restricting binaries that MediaWiki shells out to

2017-12-04 Thread Kunal Mehta 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi,

For a while now, Wikimedia has been restricting the different binaries
that we shell out to (mostly image handling things) with firejail[1].
This was a manual process by writing wrapper scripts that invoked
firejail, and pointing MediaWiki to use those "binaries". It was a
pretty manual process, and other users of MediaWiki didn't benefit
from any of the work that was being done.

With [2], it's now possible to have MediaWiki invoke firejail with
restrictions specified in the code rather than configured separately.
For example, I converted the Score extension[3] to use the new shell
restrictions system.

There's more documentation available on-wiki[4]. You can test this out
yourself by installing firejail, and by setting
$wgShellRestrictionMethod = 'firejail';.

Note that firejail is a Linux-specific program, but the restriction
framework itself is abstract enough that it's likely that support for
other restriction software could be introduced.

[1] https://firejail.wordpress.com/
[2] https://gerrit.wikimedia.org/r/#/c/384930/
[3] https://gerrit.wikimedia.org/r/#/c/393830/
[4] https://www.mediawiki.org/wiki/Manual:Shell_framework#Restrictions

- -- Kunal / Legoktm
-BEGIN PGP SIGNATURE-

iQJLBAEBCgA1FiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAlolpfoXHGxlZ29rdG1A
bWVtYmVyLmZzZi5vcmcACgkQUvyOe+23/KI8CBAApq8rMUGf0SKIPYCddyBLJgMO
vL02D6xjA0j1HCmxwynolUh751ZbAi+kgNbEA8gtosxfvQls++vs+V4x1OXfxFWJ
+Fz2Q0WYLv0j0o8pQPbugDknNlMljmILDf5ISxlCyGxw+i6bGLdFOuLWVWVTBsAx
eGlGLiFT6ROZfG72K/V0hgDnT760bwLHMVhBR672zo+Sau3UdY2hGu1uG/9GJbQv
50/9z2JtZs7ZrriqX240DZNSQwi8O7L4ppJRRUo+z+Apc4+QbxUgE0hgbe6RYKCg
K0qLksh2zZwW4c/5rqwq4P85hCS56JeW+++Wq1q7iapQH+PWsCWoztyDc0yZAp1P
+UVVSe9aOEAhudoU6yvoHlxlatAJGJv6H9LcyNHTPTAyKwYJtOc5u30PHMpt65mB
oVHu7KbSljkGTpnTd01d1VY2gLjil40FwjoL76M1LBxIPe1Fx9SFJVDgwXxpK1BM
gY0RcwWSXjzB+vnUOcQL6G4FQ/BYP9+43y4G3OtMTrkK8FkANaqgdpBECM7wJltl
lP6DGluQBBVv0cgXwZ+PKQg2dJkYxbWOR0m1T219REoP6O85EEZjvhyCx3IEaHGG
NAaTc5s93XR85hP4+gjdaC8lAJRIld4y3OjQQ7w4tWVygy7Ve4J/cYiRreWyTiBf
nRgXZj47aHrM92jY2q4=
=HIvI
-END PGP SIGNATURE-

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Help test longer edit summaries (and other comments) on testwiki

2017-12-04 Thread Brad Jorsch (Anomie) 
I've just now enable the feature flag on the testing wikis[1][2][3] that
allows MediaWiki to store comments longer than 255 bytes.

The web UI has not been updated to allow longer comments in places where it
enforces a limit, such as the edit summary box. But if you use the API to
edit, or perform page moves or do other things where long comments could be
entered and were truncated, you should now find that they're truncated at
1000 Unicode characters rather than 255 bytes.

Please test it out! If you find errors, or places in core features (not
comments in extensions such as SecurePoll, AbuseFilter, CheckUser, or Flow)
where *new* comments are still being truncated to 255 bytes, or places
where comments aren't showing up at all, please let me know. You can reply
to this message or post a task in Phabricator and add me as a subscriber.

If things go well, we'll look at rolling this out to production wikis once
the schema changes to the production databases are complete. See
https://phabricator.wikimedia.org/T174569 to follow progress there.

If anyone is interested in submitting patches for the web UI to reflect the
changed length limits, please do. I'll try to review them if you add me as
a reviewer.

[1]: https://test.wikipedia.org/wiki/Main_Page
[2]: https://test2.wikipedia.org/wiki/Main_Page
[3]: https://test.wikidata.org/wiki/Wikidata:Main_Page

-- 
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Fwd: [Engineering] Wikimedia-log-errors summary - 2017-12-01

2017-12-04 Thread Greg Grossmeier 
Forwarding as engineering@ is rarely used anymore.

Thanks Timo.

- Forwarded message from Timo Tijhof  -

> Date: Fri, 1 Dec 2017 18:42:26 -0800
> From: Timo Tijhof 
> To: Engineering 
> Subject: [Engineering] Wikimedia-log-errors summary - 2017-12-01
> 
> Been a while since the last summary, so here is one.
> 
> * 168 reported unresolved issues
> * 160 issues without an assignee
> * 10 issues tagged with "Patch-For-Review"
> 
> Please check out the breakdown of open issues by project to see which
> errors in your field welcome fixing:
> 
> 
> Or get the latest straight from Logstash:
> 
> 
> Top 5 by hit count in last 24 hours:
> 
> 1. ERROR channel=PageViewInfo "Failed fetching {url}: {error}"
> 
>#PageViewInfo
>https://phabricator.wikimedia.org/T181011
> 
> 2. ERROR channel=error "Fatal error: Call to exists() on a non-object"
> 
>#ProofreadPage
>https://phabricator.wikimedia.org/T181868
> 
> 3. WARNING channel=XMP "XMPReader::parse .. "
> 
>#Multimedia #MediaWiki-File-management
>https://phabricator.wikimedia.org/T118799
> 
> 4. WARNING channel=session "Metadata merge failed: {exception}"
> 
>#MediaWiki-Authentication #CentralAuth
>https://phabricator.wikimedia.org/T158365
> 
> 5. WARNING channel=session "Metadata has an anonymous user, but a non-anon"
> 
>#MediaWiki-Authentication #CentralAuth
>https://phabricator.wikimedia.org/T181869
> 
> -- Timo

> ___
> Engineering mailing list
> engineer...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/engineering


- End forwarded message -

-- 
| Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
| Release Team ManagerA18D 1138 8E47 FAC8 1C7D |

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] FLIF for Wikimedia

2017-12-04 Thread Ruben Kelevra 
Hey Thiemo,

On 04.12.2017 10:43, Thiemo Kreuz wrote:> I consider myself an image
file format nerd, so thanks a lot for
> sharing this! FLIF was new to me.Don't mind it! :)

> I would like to share two important notes:
>
> 1. Unfortunately the flif.info website does not say a word about the
> CPU resources their current implementation burns when converting a,
> let's say, PNG to FLIF.Well, currently it's single core and not
optimized at all. You should know CABAC encoding from x264/x265 so it's
slow, but not dead slow.

The website doesn't mention it because it highly depends on the subject
as well as the setting on encoding named effort.

Currently, effort is default 60, I tried 100 a lot, but there's nearly
nothing to gain. So I assume we always want to use the good default. :)

PNG Encoding of the current featured picture of today[1] at a medium
image size for the web, 1.280×868 Pixel, opened in Gimp, converted to
sRGB and exported as maximum compression without any checkbox done in
Gimp ... takes Gimp 3 seconds to write it to the Harddisk.

Transcoding this PNG to FLIF takes on my machine (i3-2330M@2.20GHz; 12
GB RAM):
real 0m7,405s
user 0m7,320s
sys 0m0,053s

decoding the file again to PNG on the same machine (with FLIF)
real 0m1,077s
user 0m1,067s
sys 0m0,007s

For comparison, we save 708 KByte in comparison to PNG in this case, and
the PNG exported by FLIF is just 14 KByte bigger than the one created by
Gimp.

> It's pretty important to realize that CPU
> resources are even more valuable than storage space and network
> bandwidth. Sure, It's not really possible to come up with an exact
> threshold. But if, let's say, converting a PNG to FLIF saves 100 KiB,
> but takes a minute, this minute will never pay off.
So my Point was more: Get rid of this bloody Download a JPG, do some
Stuff & Upload a 3 Times locally saved JPG again, calling it an improvement.

> If you follow the discussions on Wikimedia Commons, you will find this
> argument quite often: Downloading PNGs, optimizing them, and uploading
> them again is practically never worth it. Think of all the resources
> that are burned to do this: CPU time, download and upload, database
> storage and time, disk storage for the new revision, and not to forget
> the user doing all this.
Yep, but enabling FLIF for new files would do nothing to the old Data at
all, I don't want to start a discussion about converting petabytes of
Data, but all new revisions should be done in FLIF, if you ask me.
> Sure, your suggestion avoids a lot of this. But the CPUs involved will
> experience heavy load, both on the server as well as clients that need
> to recode FLIF files via a JavaScript library first.
FLIF is very fast to decode via JavaScript, and in general, as the
example shown above, it takes just 1 second to decode and encode a
medium size image as PNG with just one thread on a pretty outdated
notebook with an unoptimized decoder and encoder. :)

Try adding a FLIF to a website and test out if the website load anywhat
slower with the FLIF ... at the small image sizes you get on articles,
the performance impact is neglectable and comparable to loading a font
file to the browser.
> 2. Lossy file formats like JPEG should never be converted to lossless
> formats. This will actually decrease quality (over time). The problem
> is that the image data will still contain the exact same JPEG
> artifacts, but the fact that it was a JPEG (and how it was encoded) is
> lost.
Yes, but those images should never be saved as JPG in the first place.
Even under Android RAW-Photography is going to be a thing. FLIF just
started to get rudimentary RAW-capabilities, which means you can just
convert the special RAW-File to a FLIF and upload it with any loss in
quality.

> No tool specialized in squeezing the maximum quality out of
> lossy JPEGs can work anymore. And there are a lot of super-awesome
> tools like this. Not only tools like JPEGCROP and such that can cut
> and even combine JPEGs without actually recoding them.
Well, if you really want to start a discussion about a lossless rotation
of JPG files, let me guess how many uploads are rotated losslessly...
0.0002%?

> There are also "JPEG repair" filters that know how to reverse the JPEG
encoding
> algorithm and can squeeze out a tiny little bit of extra information
> regular JPEG decoders can't see.
Great! Just recommend this for new uploads to be done if the source
material is JPG, let the ppl convert it with this to PNG and then to
FLIF or ask the FLIF maintainers if they want to add this as an
import-filter, for FLIF itself! :)

But the your argument was: "Think of all the resources that are burned
to do this: CPU time, download and upload, database storage and time,
disk storage for the new revision, and not to forget the user doing all
this."

Which perfectly fit's here too. On a 20 MPixel picture small JPG
Artefacts are no issue at all, but users which Download the JPG and
upload it again, after doing some needed work to

Re: [Wikitech-l] FLIF for Wikimedia

2017-12-04 Thread Thiemo Kreuz 
Hey,

I consider myself an image file format nerd, so thanks a lot for
sharing this! FLIF was new to me.

I would like to share two important notes:

1. Unfortunately the flif.info website does not say a word about the
CPU resources their current implementation burns when converting a,
let's say, PNG to FLIF. It's pretty important to realize that CPU
resources are even more valuable than storage space and network
bandwidth. Sure, It's not really possible to come up with an exact
threshold. But if, let's say, converting a PNG to FLIF saves 100 KiB,
but takes a minute, this minute will never pay off.

If you follow the discussions on Wikimedia Commons, you will find this
argument quite often: Downloading PNGs, optimizing them, and uploading
them again is practically never worth it. Think of all the resources
that are burned to do this: CPU time, download and upload, database
storage and time, disk storage for the new revision, and not to forget
the user doing all this.

Sure, your suggestion avoids a lot of this. But the CPUs involved will
experience heavy load, both on the server as well as clients that need
to recode FLIF files via a JavaScript library first.

2. Lossy file formats like JPEG should never be converted to lossless
formats. This will actually decrease quality (over time). The problem
is that the image data will still contain the exact same JPEG
artifacts, but the fact that it was a JPEG (and how it was encoded) is
lost. No tool specialized in squeezing the maximum quality out of
lossy JPEGs can work any more. And there are a lot of super-awesome
tools like this. Not only tools like JPEGCROP and such that can cut
and even combine JPEGs without actually recoding them. There are also
"JPEG repair" filters that know how to reverse the JPEG encoding
algorithm and can squeeze out a tiny little bit of extra information
regular JPEG decoders can't see.

This said, I'm all for adding FLIF to the list of allowed file formats!

Best
Thiemo

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l